Failed downloads are never retried

[Jdbye]

A lot of the downloads fail repeatedly, after enough failed attempts they usually download, but I've not been able to download some of them, and had to wipe (or rename) both the db/channels folder and the download folder in order to retry because of a couple issues:

• Failed downloads are never retried, they are only tried on the initial database creation
• If db/channels is wiped, it tries to redownload every video (even ones that already exist), but they hang at 99% on the ffmpeg stage when the file already exists Even after doing this 3 times some of the videos wouldn't download. I merged the contents of the 3 different download folders, so it's mostly complete, but not quite.

Also, it seems to hang before it completes all the downloads. The interface stops updating with about 30 videos left to download, Ctrl+C doesn't close out of the program and I have to kill it and restart. Then when restarted, it doesn't resume where it left off, it thinks it completed so it just goes to "Fetched 0 videos" and starts the "Checking for new videos in 5 minutes" timer.

[Fma965]

I have the same issue

[Yannickclae]

Same Issue.

[Inrixia]

@Yannickclae @Fma965

• How many videos are you trying to download?
• Does setting downloadThreads to 1 help?
• What error is being shown when downloads are failing?

[Yannickclae]

• How many videos are you trying to download?: The whole backlog of FP exclusive. ~205
• Does setting downloadThreads to 1 help?: Haven't tried, but the problem is that it doenst even seem to want to retry the failed downloads
• What error is being shown when downloads are failing? Time out error.

[Inrixia]

@Yannickclae try using threads set to 1. If your downloading a large number of videos then things can break if you have threads set to unlimited or really high.

[Inrixia]

Also the edge servers are down atm. You can see in discord for updates on that.

[Inrixia]

Going to close this for now as https://github.com/Inrixia/Floatplane-Downloader/commit/940154d3c57c6abd7a2995c89dd9a357d35bc11e should help avoid this issue.

If this occurs again for anyone with download threads set low please re-open it.

[Jdbye]

Going to close this for now as 940154d should help avoid this issue.

If this occurs again for anyone with download threads set low please re-open it.

I had it set to 4. Does that count as low?

[Inrixia]

Yes, sounds like this may be a more complex issue. Ill re-open for now and see if there is anything I can do.

[Inrixia]

Ok so there has been a few changes.

• Default retries for floatplane have been upped to 9 & I have hopefully improved the edge round robin/selection code. This is because some floatplane download edges appear to be timing out, nothing I can really do about this but up the retries. The new download round robin code should also help avoid constantly retrying bad edges but its not a perfect solution. I was considering pinging edges before returning them for use but that's rather complex so will see if upping retries and the new fetching fixes it.

• Default downloadThreads has been set to 1, this isnt to say you cant use say 2-4 or more but enough people are using the downloader to grab the entire backlog vs just dalies that settings this to default low is much easier. If you are grabbing the entire backlog then setting this low should help avoid issues.

• Updated download/mux checks. This one isnt really all that much, just an additional check to be sure that videos not downloaded aren't marked as such. As far as I'm aware this will not actually make any difference to the behavior of the downloader.

Regarding your issue with failed downloads not even being re-attempted, this is actually likely due to the fact that by default after the first run the downloader will only search back to the last video it downloaded. This is to avoid massive api calls for people who setup the downloader with a big number and then leave it running to fetch new videos.

You can disable this functionality with the setting floatplane.forceFullSearch which is explained on the Settings Wiki

Setting this to true will "Force the downloader to search the full videosToSearch regardless of what has been downloaded. Note: Will not result in downloaded videos being redownloaded."

With this set to true you should be able to easily re-crawl through all the videos and the downloader should properly resume/retry any which are not properly downloaded or muxed.

[Inrixia]

As an added bit of clarification, once a video is found, passes channel identifier checks and then actually attempted to be downloaded there are two checks that are done, first is to see if the video is already downloaded and then if it is to see if the video is muxed (ffmpeg has added metadata).

A video is considered muxed if the size of the video file ending with .mp4 matches the expected size stored in the db. Note that videos that were downloaded but not muxed are saved as .partial, and videos which were muxing will not have their expected size match until after muxing is done.

A video is considered downloaded if either the video is already muxed or the file size of the .partial file is equal to expected size.

The expected size of the video is stored in the channel db and is set twice, firstly when the video starts downloading its set to the expected file size returned from floatplane. The secondly when the video successfully finishes muxing.

With all this in mind any video that fails to properly mux/download should be properly retried, AS LONG AS said video is actually found, passes channel identifiers and is added to the download/processing queue.

And to do that you need to use floatplane.forceFullSearch: true with an appropriate floatplane.videosToSearch value

[Inrixia]

All things considered there should probably be something to indicate that if you are using the downloader to grab the entire backlog that there are some additional settings you might want to use. Since the default behavior's/settings are designed for daily fetching.

This is something Ill think about indicating on the readme etc.

[Inrixia]

@Jdbaii With all this extra info, can you grab the latest dev build, use the above mentioned settings and let me know if you run into any further issues?

[Inrixia]

Gonna close this for now, please re-open if anyone has further issues or believe this isnt resolved :)