Cannot upload data using openneuro/cli

[GongZhengxin]

What went wrong?

Hi, we met the problem similar to #2750.

Our initial dataset (ds004496) was uploaded in February, and we recently want to renew it for publication. After checking the remotes files and entering "Y" for uploading, no progress bar nor any errors were reported. We have tried on 4.15.1 (which worked well several weeks ago), 4.18.1 (which is recommended by warnings) and 4.19.0 (which seems the latest in npm). None of them can fix the problem. Could you please help us figure it out?

In addition, we are curious about the remote files checking procedure, it takes suspiciously a long time (around 30 minutes), is that because of the connection speed?

See the screenshot:

Expected behavior

We expect a progress bar or any errors to help us figure what to do next.

How to reproduce

No response

Desktop

• OS: ubuntu 20.04.5
• Version: node.js - v18.13.0 npm18 - 9.7.2 openneuro - 4.18.1

Phone

• Device:
• OS:
• Browser:
• Version:

Additional information

No response

[nellh]

Hi, thanks for reporting this issue. I would expect a dataset with a large number of directories like this to take some time to compare the remote files, though 30 minutes is longer than expected.

As a possible quick fix, could you try running with additional Node.js memory? You can try that like so NODE_OPTIONS=--max-old-space-size=8192 openneuro upload ...

If that does not help, you can capture a debug log by adding NODE_OPTIONS="--trace-uncaught --trace-warnings --trace-exit --trace-sigint" NODE_DEBUG=http,fs,stream that would be helpful for looking into both issues.

[Shotgunosine]

I'm having a similiar issue. I added some derivatives to https://openneuro.org/datasets/ds004787 and tried to update it through the CLI via: (opennuro) [nielsond@helix BIDS_toshare]$ openneuro upload -vvv --dataset ds004787 .

All I get is:

Adding files to "ds004787"
bids-validator@1.10.0

and it runs way to quickly to be updating anything. I added the not options you suggested redirected the output to a log file that I've attached here.

oneurolog.txt

[Shotgunosine]

In my case it turned out that I didn't have permission to read some of the files in the directory. There were also some broken symlinks. Now that I've fixed the permissions errors, the upload seems to be running fine.

EDIT: Whoops, spoke too soon, once it actually started uploading I got an invalid size error:

#
# Fatal error in , line 0
# Fatal JavaScript invalid size error 195587740
#
#
#
#FailureMessage Object: 0x7ffece815960
 1: 0x14e7c3f34297  [/gpfs/gsfs9/users/nielsond/python/envs/opennuro/bin/../lib/libnode.so.108]
 2: 0x14e7c4b126ce V8_Fatal(char const*, ...) [/gpfs/gsfs9/users/nielsond/python/envs/opennuro/bin/../lib/libnode.so.108]
 3: 0x14e7c4451acb  [/gpfs/gsfs9/users/nielsond/python/envs/opennuro/bin/../lib/libnode.so.108]
 4: 0x14e7c47fe7e1 v8::internal::Runtime_StringToArray(int, unsigned long*, v8::internal::Isolate*) [/gpfs/gsfs9/users/nielsond/python/envs/opennuro/bin/../lib/libnode.so.108]
 5: 0x14e7c4131039  [/gpfs/gsfs9/users/nielsond/python/envs/opennuro/bin/../lib/libnode.so.108]
Trace/breakpoint trap (core dumped)