I try to backup a 300Go directory containing lot of files on an S3 ceph radosgw. The source directory is on a lustre file system. I replace findit library by findit2 and update the endpoint of s3 Client to my Ceph S3 rados gateway. s3-cli start well and try to list all files to backup.
An option to avoid this stage and start upload directly could be good.
Ex: When an S3 upload worker is free, It ask for a next file to upload. A file system walker restart on last point and find the next file to upload.
Other solution:
A new findit3 that can work by part. Each 100 000 files events, It pause and wait for a called to its startNextPart() function. When s3 worker finished, It call the startNextPart(). When end event is get, s3 worker don't call startNextPart but stop.
Out of memory error get with the actual s3-cli:
ROOT:/s3-cli > node cli.js --config s3.conf sync /my-big-dir s3://MY_BUCKET/
1301046 files
<--- Last few GCs --->
[10845:0x21afd30] 851433 ms: Mark-sweep 1385.4 (1448.3) -> 1385.3 (1450.8) MB, 1849.4 / 3.2 ms allocation failure GC in old space requested
[10845:0x21afd30] 853877 ms: Mark-sweep 1385.3 (1450.8) -> 1385.3 (1413.3) MB, 2421.9 / 1.8 ms last resort GC in old space requested
[10845:0x21afd30] 855909 ms: Mark-sweep 1385.3 (1413.3) -> 1385.3 (1412.3) MB, 2031.5 / 2.6 ms last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0xe386a725739 <JSObject>
1: /* anonymous */ [/s3-cli/node_modules/s3/lib/index.js:~1149] [pc=0xa91f3224238](this=0x21132216e859 <EventEmitter map = 0xeee3e8b4559>,file=0x2edeaec78c99 <String[92]: /path_to_a_file_to_backup>,stat=0xdc985fbbf61 <Stats map = 0xeee3e8bf609>)
2: arguments adaptor frame: 3->2
3: emit [...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [node]
2: 0x11ef43c [node]
3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
5: v8::internal::Factory::NewUninitializedFixedArray(int) [node]
6: 0xdf0023 [node]
7: v8::internal::Runtime_GrowArrayElements(int, v8::internal::Object**, v8::internal::Isolate*) [node]
8: 0xa91f30842fd
Hi,
I try to backup a 300Go directory containing lot of files on an S3 ceph radosgw. The source directory is on a lustre file system. I replace findit library by findit2 and update the endpoint of s3 Client to my Ceph S3 rados gateway. s3-cli start well and try to list all files to backup.
An option to avoid this stage and start upload directly could be good. Ex: When an S3 upload worker is free, It ask for a next file to upload. A file system walker restart on last point and find the next file to upload. Other solution: A new findit3 that can work by part. Each 100 000 files events, It pause and wait for a called to its startNextPart() function. When s3 worker finished, It call the startNextPart(). When end event is get, s3 worker don't call startNextPart but stop.
Out of memory error get with the actual s3-cli: