Use goofys as backup location for SAP HANA

[dzodzo666]

Hello, is it possible to use goofys for SAP HANA backups? I have mounted a bucket under uid/gid of the SAP instance user, created necessary filestructure but when we start full backup we get several write errors from the backup scripts that probably relate to the nature of how S3 bucket works. If backup is uploading a file to bucket, it's not able to write into log on the same bucket. There may be also trouble if new redo log is created during full backup that it won't be able to write into redologs directory.

Is there some way to allow parallel writes to bucket or it's impossible because of the way how working with S3 bucket is implemented via API calls? Thanks for info.

[dzodzo666]

Just a quick workaround that comes to my mind is to create separate bucket for each part of backup objects. 1 bucket for data, 1 for redologs etc. But it's not the parallelism, thank you.

[kahing]

What tool are you using to do the backup? How many parallel writes does it do?

[dzodzo666]

Greetings, it's some proprietary and very long script by SAP. I noticed the trouble during full backup it would produce many write errors into stdout pointing to lines in script where it stores output into backup log. When opening the log we found just 1 line that denoted start of the backup. After starting to write data into S3 it would also block any attempts to list the content of directory in S3, it was hung until the backup finished then the ls command succeeded. From this behavior i suspect the FS is only able to handle 1 request at a time.

[kahing]

Without knowing what the script does it's very hard to debug this. Could you run goofys with --debug-fuse and attach the log here?

[dzodzo666]

Greetings, i'm sending in attachment /var/log/messages, the output from console and the script i'm running. Debug output starts at Jan 31 16:28:22. The script copies 3 files simultaneously to goofys mounted FS. I'm trying to run the script cps3.sh with redirecting output to the same directory where i'm copying files. If i run just the script, the copying goes fine with no errors concurrently. But if at the same time i want to redirect the output to logfile in the same location as target for files, the script starts to produce error output and in the write log there is just one line:

s3fuse-test goofys # cat write.log Wed Jan 31 16:28:22 CET 2018

If i try to ls -l at the time the copying with log output is running on another console, the ls command is stuck until copying is finished. If i try ls -l with just copying the files with no concurrent output to the log, it works ok.

cps3.sh.txt

console.txt

messages.txt

Please let me know if you want me to test some specific scenarios. Thank you for support.

[kahing]

how are you redirecting the output to write.log? The problem is it's issuing multiple flushes which goofys does not support (you cannot flush to S3 and then keep writing to it). cps3.sh itself is just doing normal cp which seems like is working fine.

[dzodzo666]

it's just the usual output redirection in bash, however i'm not redirecting also the error output (2>&1) as seen on the attached console output.

./cps3.sh > /s3mnt/goofys/write.log

how bash implements the writing/flushing is beyond my understanding sorry

[kahing]

the problem is bash uses dup2 for redirection which is not supported. You can workaround that by doing /cps3.sh | tee /s3mnt/goofys/write.log instead.

[kahing]

note that there are many problems with your backup scripts which are unrelated to goofys. For example instead of that while loop (which is buggy on its own) you should probably use wait instead.

[dzodzo666]

Greetings, this is not the actual backup script (the provided script from SAP has thousands of lines :) it's only to test the concurrent write to s3 bucket. Thank you for pointing out the problem with output redirection, that has helped.