search

MalloyDelacroix / DownloaderForReddit

The Downloader for Reddit is a GUI application with some advanced features to extract and download submitted content from reddit.
GNU General Public License v3.0
505 stars 47 forks source link

Intermittent failures of downloads from redgifs #141

Closed FlyingSpaghett closed 3 years ago

FlyingSpaghett commented 3 years ago

Firstly, let me say a big thank you for this project and all the work you are putting in, it is an amazing tool!

The issue in question concerns webm downloads from redgifs, when downloading a users profile. A seemingly random number of webm downloads will be downloaded incorrectly when downloading a users profile. There is no error given in the error log/info screen, but looking at the files there are always a few that are corrupted. These are webm files with 0kb size, or are a smaller size than they should be (1mb for example) and thus are corrupted/incomplete and are picked up as such when playback is attempted.

It is not a content issue, as rerunning the download on a freshly created new database yields a similar number of corrupted downloads, but on different files/posts. The corrupted files correspond to different posts each time the profile download is run.

I have experimented with numerous settings, including trying to download images and videos separately, decreasing the extraction and thread count to 1 and 1 ( as well as different values), using vs not using "avoid duplicates", unchecking unneeded content websites from the list of supported sites and changing the post limit settings. Thus far none of the above changes seemed to have worked, though some of them did help a bit (lower number of corrupted downloads). DownloaderForReddit.log

MalloyDelacroix commented 3 years ago

Thanks! I'm glad you're enjoying the app!

I've been investigating this issue. I can't find anything on our end that is causing this problem. Redgifs is still a relatively new site, and doesn't have a public API yet, so the way the app connects is not as solid as some of the other sites. My best guess at this time is that this is caused by a connection issue with redgifs and is on their side of the connection.

The good news is because these downloads don't appear to be flagging out with errors when they fail to download, and since they appear to be random in which links fail; you should be able to successfully retry the downloads. To do this, go to the Download menu, and click "Run Unfinished Downloads".

I will keep an eye on this and keep searching for a solution.

FlyingSpaghett commented 3 years ago

Understood, thank you for clarifying and checking it out. Unfortunately, due to the downloads being flagged as successful by the program, the option to "Run Unfinished Downloads" does not work and only checks for new content.

I also suspected a connection issue, as without an API multiple request in quick succession definitely have a chance of ending in a corrupted download from a denied request or other connection issue. This is also why I tried setting the extraction and download thread count to 1, which does seem to help a bit, but not enough to get rid of the issue entirely unfortunately. I will also continue to monitor things from my side and try out additional settings configs to try and optimize/alleviate this issue.

Thanks again

FlyingSpaghett commented 3 years ago

Hi,

Any update on this perhaps? :)

MalloyDelacroix commented 3 years ago

None yet.

MalloyDelacroix commented 3 years ago

I think this may finally have been fixed with pull request #219. If this turns out not to be the case, please reopen this issue.