Closed ncw closed 1 year ago
Hi @ncw is it possible that those tests are doing several uploads to the same location one by one? Recently we introduced a rate limit for uploads to the same path.
Yes that is very likely @mniewrzal
Should rclone retry the upload in this case? Perhaps after some time?
Any thoughts on this? This has killed the rclone + storj integration tests :-(
These are all the failures.
Could you provide logs from some of failed tests?
The overview of tests is here and you can download logs from there: https://pub.rclone.org/integration-tests/current/
However this one is representative
"go test -v -timeout 1h0m0s -remote TestStorj: -verbose -test.run '^TestIntegration$/^FsMkdir$/^FsPutFiles$'" - Starting (try 5/5)
=== RUN TestIntegration
fstests.go:423: Using remote "TestStorj:"
2023/06/20 05:13:11 DEBUG : Creating backend with remote "TestStorj:rclone-test-dupipey8yogicap7sadajux5"
2023/06/20 05:13:11 DEBUG : Using config file from "/home/rclone/.rclone.conf"
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: connecting...
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: connected: <nil>
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: mkdir -p ./
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: ls -R ./
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: OBJ ls -R ./ ("rclone-test-dupipey8yogicap7sadajux5", "")
=== RUN TestIntegration/FsMkdir
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: mkdir -p ./
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: ls -R ./
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: OBJ ls -R ./ ("rclone-test-dupipey8yogicap7sadajux5", "")
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: mkdir -p ./
=== RUN TestIntegration/FsMkdir/FsPutFiles
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: cp input ./file name.txt # [] 100
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: stat ./file name.txt
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: cp input ./hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt # [] 100
2023/06/20 05:13:12 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: stat ./hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt
2023/06/20 05:13:12 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: cp input ./file name.txt # [] 100
fstests.go:154:
Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:154
/home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:170
/home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:223
/home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:885
Error: Received unexpected error:
uplink: too many requests
storj.io/uplink.convertKnownErrors:69
storj.io/uplink.(*Upload).Commit:181
github.com/rclone/rclone/backend/storj.(*Fs).Put:582
github.com/rclone/rclone/fstest/fstests.PutTestContentsMetadata.func1:189
github.com/rclone/rclone/fstest/fstests.retry:146
github.com/rclone/rclone/fstest/fstests.PutTestContentsMetadata:170
github.com/rclone/rclone/fstest/fstests.testPutMimeType:223
github.com/rclone/rclone/fstest/fstests.Run.func13.13:885
testing.tRunner:1446
Test: TestIntegration/FsMkdir/FsPutFiles
Messages: Put
2023/06/20 05:13:12 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: ls -R ./
2023/06/20 05:13:12 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: OBJ ls -R ./ ("rclone-test-dupipey8yogicap7sadajux5", "")
--- FAIL: TestIntegration (1.17s)
--- FAIL: TestIntegration/FsMkdir (0.84s)
--- FAIL: TestIntegration/FsMkdir/FsPutFiles (0.39s)
FAIL
exit status 1
FAIL github.com/rclone/rclone/backend/storj 1.186s
"go test -v -timeout 1h0m0s -remote TestStorj: -verbose -test.run '^TestIntegration$/^FsMkdir$/^FsPutFiles$'" - Finished ERROR in 1.991328221s (try 5/5): exit status 1: Failed [TestIntegration/FsMkdir/FsPutFiles]
I think it's the same limitation we talked earlier, uploading to the same location many times. I just realized that I never replied to your comment from May and I'm deeply sorry for that.
Should rclone retry the upload in this case? Perhaps after some time?
Yes, that would be the best solution.
I ran those integration tests against local environment and I had no issues with uplink: too many requests
but we have different limit defaults for prod and test environments. On production limit is one upload to the same location per 1s so if retry will be needed it needs to be at lest 1s.
That was very helpful info thank you. I can make a workaround for that which will sleep for 1 second then retry.
I think this is probably a problem which only affects rclone's integration tests and not actual users unless they are in the habit of altering the same file very frequently.
I'll close this now.
Thank you for looking into this. Let me know if I will be able to help with anything else.
The rclone integration tests for storj ran smoothly until 2023-04-19 when they started failing with error "too many requests" quite regularly when uploading files.
This seems to be entirely on uploading files.
Has something changed in the backend?
This is the versions of modules I'm using
I note v1.10.0 is quite old (December 2022) but there isn't a newer release tagged. I can try on master if you think that is worth a go.
Thanks