Restore data from S3

[kaaskilde]

Hello,

HCL Product Version Domino 12.0.1FP1

I asked HCL Support some questions about Domino Backup and S3, they asked me to create a "New Issue" inhere instead.

We currently trying to get a backup/restore process working with S3, so far we got the backup part working towards S3.

Now we are trying to get a restore job done, but we can't get it working from S3.

We have done the configuration as follows: Restore DB command [Cmd Formula]: x:="/": q:={"};"C:\progra~1\Amazon\AWSCLIV2\aws s3 cp " + " " + q + BackupTargetDir + x + BackupNode + x + BackupMode + x + BackupDateTime + x + FileName + q + "2>&1"

But when we run the "Restore Server Cmd" it doesn't try and get the file from S3 it tries running what I pasted in down below.

Can you please point me in the correct direction?

28-02-2023 15:11:51 Restore: Translog Mode: CIRCULAR 28-02-2023 15:11:51 Restore: Domino Database Restore 28-02-2023 15:11:51 Restore: Started 28-02-2023 15:11:51 Restore: Restoring to [restore\kaatest1.nsf] [F:\HCL\Domino\Data\restore\kaatest1.nsf] 28-02-2023 15:11:51 Remote console command issued by Kristian Aaskilde/OpusNeo: load restore 28-02-2023 15:11:52 Restore: Restoring Database [F:\HCL\Domino\Data\restore\kaatest1.nsf.DAD] 28-02-2023 15:11:52 Restore: Cannot start empty apply log info for database [F:\HCL\Domino\Data\restore\kaatest1.nsf.DAD]: File does not exist 28-02-2023 15:11:52 Restore: Error opening Database [F:\HCL\Domino\Data\restore\kaatest1.nsf.DAD]: File does not exist 28-02-2023 15:11:52 Restore: Error restoring database [KAA\kaatest1.nsf] -> [restore\kaatest1.nsf]: File does not exist 28-02-2023 15:11:52 Restore: Finished

Kind regards Kristian Aaskilde

[Daniel-Nashed]

You should also get an output log file for the command invoked. It looks like the command you invoke runs into an error.

I would first replace the backslash with slashes. this is much easier to handle than the backslash, which needs to be masked.

the invocation of the AWS CLI seems to be the root cause here.

You could test the commands in the UI and will see the resulting command as an example. there is a button in the UI, which should help you to get it right.

there is one detail you should add to the S3 command to disable the percent output. this seems to be on in later versions of the CLI at least. I will update that in the DXL example.

Please add --no-progress to the command-line (this isn't your actual issue but would be the next warning you would get, because the return buffer would buffer overflow with too much log data returned from the command.

Please note S3 backup is a quite slow backup and is only intended for smaller environments. You can't usually get more than 60 MB/sec. Depending on your connection maybe even less.

Let me know if my tips help.

Just posted the update for --no-progress to the Windows example.

Can you use the new DXL file and please verify this works for you with the changed path.

Daniel

[kaaskilde]

Hi Daniel,

Thanks for the quick response that is much appreciated!

You tips worked very well thanks and I got the restore working from S3 now.

I have one question about the placement of the restored files. In the documentation it says that restore folder needs to be in the DATA directory of the Domino server, but having multiple databases in the DATA directory with the same replica ID doesn't seem all to good, so we have been trying to move the directory out by adding E:/restore as shown below in the command and the test function in the UI looks right "aws s3 cp "/local/backup/MyBackupNode/FULL/20210730124242/names.nsf" "E:/restore/names.nsf" --no-progress 2>&1"

But when the restore task runs it still tries to place it inside the DATA directory - can this be moved out or will it always place it inside the data directory?

x:= "/"; q:={"}; "aws s3 cp " + q + BackupTargetDir + x + BackupNode + x + BackupMode + x + BackupDateTime + x + FileName + q + " " + q+ "E:/restore/"+FileName+ q + " --no-progress 2>&1"

Imported the DXL file, changes the variables to our settings and did a incremental backup/restore and that worked without any problems with the --no-progress - will try and run a full backup again tomorrow.

Kristian

[Daniel-Nashed]

Great to hear my tips helped! Thanks for replying immediately. The resulting database has the .DAD extension and is not seen by the server. The restore command takes the database, disables replication (default configuration) or optionally changes the replica ID. Those are features of the restore command and they are used for all standard restore flows.

Restoring files outside the data directory is not needed and is not a good idea in general! All databases should be always seen by the server and you will need the resulting database in the data directory (default is restore, you can overwrite the is location in the global config or per restore operation).

The application takes care about the challenges you are mentioning in regard to the same replica-id. The restore operation can also add deleted documents and folders back into an original database. So the database needs to be fully accessible. In addition DAOS needs the database to be in the data directory.

Please use the standard location and open an issue, if it isn't working as intended. This would be a separate issue to keep track of the problems.

Daniel