Uploading large dir of images hangs

[jvasile]

Uploading a largish (500+ items) directory of largish files (images) and some html files, rclone just hung after 146 files.

The files are all images and web pages from Astronomy Photo Of The Day. I obtained that dir by cloning https://github.com/OpenTechStrategies/permanent-rclone-qa/ and running ./apod-downloader.py. Then I tried to upload them with ./upload-test.py test-tree/apod --remote-dir=apod. Things went fine, and it zipped along with 4 parallel uploads. But after 146 files, it looks like each upload hung, one by one, as it hit a file that refused to either complete or fail.

Here's the log from this run: apod.txt

If you want to reproduce this, you're welcome to use my apod-downloader and upload-test script, but I also suspect you can try to upload just about any largish set of largish files. Maybe I'll try it with a largish set of small files next.

This might be related to #99 but I got way further than just 6 jpgs before things went sideways.

[slifty]

I've made a whole slew of changes that I'm hoping will resolve this and other issues.

Two things may have been happening:

1. file handles were being closed twice, which was (sometimes) resulting in absolute chaos! This is resolved by #112
2. If anything failed on the permanent backend it would result in a failure, and rclone would then try to delete the file. Our service doesn't support deletion but that lack of support meant no response was being provided at all, and so rclone would sit there waiting for a response. This is resolved by #110

These changes will be deployed this afternoon, at which point I think it would be worth trying this again.

[jvasile]

@Fenn-CS, can you keep testing the above and then add your results here?

[Fenn-CS]

@jvasile Sure! Would follow up on it.

[slifty]

@Fenn-CS Any word on this? Would love to close it if it isn't repeating.