Unable to save user list

[MelanieTanaka]

I am having a lot of trouble saving the user list. If I add users then immediately save, it's all good. If I add users and then start the download, then one of 2 things happens: 1) the final download to complete is a success (an image on a host supported by the app) -> then all is good, the user list and progress can be saved.

2) if the final download to process is a failure (unsupported type) -> unable to save, any users added if not saved before starting the download are lost. The modification date on the save file is not changed.

When scenario 2 happens, the "download" button remains stuck, clicking it to stop does nothing. Attempting to save via the file menu gives a failure message.

The biggest issue with this is not only are users added to the list discarded if not saved before starting the download, but the same content is downloaded over and over again until a scenario # 1 happens and a save can be done properly.

the log shows this when attempting to save, also URL to screen capture of pop up error.

{"levelname": "ERROR", "name": "DownloaderForReddit.DownloaderForReddit.Persistence.ObjectStateHandler", "filename": "ObjectStateHandler.py", "module": "ObjectStateHandler", "funcName": "save_pickled_state", "lineno": 129, "message": "Unable to save to save_file", "asctime": "12/07/2018 04:32:02 PM", "exc_info": "Traceback (most recent call last):\n File \"DownloaderForReddit\\Persistence\\ObjectStateHandler.py\", line 121, in save_pickled_state\n File \"shelve.py\", line 124, in __setitem__\nTypeError: can't pickle Content objects", "save_file_location": "C:\\Users\\Administrator\\AppData\\Roaming\\SomeGuySoftware\\DownloaderForReddit\\save_file"}

url

[EC-O-DE]

Same. I have this app running from a removable marked drive - so it cannot save save file to c:/users/../../.. Importing doesn't work..

[MelanieTanaka]

Same. I have this app running from a removable marked drive - so it cannot save save file to c:/users/../../.. Importing doesn't work..

I don't know if that is related, I have no issue saving unless the specific scenario I describe happens. The application is running from C: drive.

[EC-O-DE]

For me its not populating user/sub lists from savefile.. :/

[MalloyDelacroix]

Would it be possible for you to attach the entire log file to this issue? If not the entire log file, at least enough that I can see what was going on with the app from the time that it was opened, through the error, and then until it is closed?

I can't reproduce the error under the circumstances that you stated, so I will need more of the log to diagnose the problem.

[EC-O-DE]

Fun fact: drives aren't removable - only marked as removable... By an app that writes stuff from memory to disk boosted by CPU. Sys drive at least for now is without encryption (not bitlocker).

So ~ drives that aren't removable are marked as removable. All other apps work fine.

I did extract your zip to c:/users/usrname/apps/DownloaderForReddit but still I didn't manage to get it import save file into the app - though it does move (copy) the actual file to it's location - but doesn't populate app screen (after restart).

So.. :) I got 2.2 and imported userlist then launched 3.0 and downloaded a lot of files again! Then used Duplicate Cleaner Pro 4.1.1 to scan and hash and deleted duplicate new content :)

[MelanieTanaka]

Would it be possible for you to attach the entire log file to this issue? If not the entire log file, at least enough that I can see what was going on with the app from the time that it was opened, through the error, and then until it is closed?

I can't reproduce the error under the circumstances that you stated, so I will need more of the log to diagnose the problem.

Here is the complete log for one session, from opening the app, adding some users, saving the list (before downloading), downloading (successfully), adding more users, saving again, starting the download again and this time stuck at the end. unable to save with error described in my first post.

https://drive.google.com/file/d/1ukC3Eb8MIfl9yB8k27J7tKFfZKQuNhax/view

Note that the users are all NSFW, so careful following any links.

[MalloyDelacroix]

It looks like this is not actually an error with the application. According to the log, you are trying to save while the downloader is still running. You will have to wait for the download to finish before the application can save.

The app operates link extraction and download in two separate threads, so the extraction process (the progress bar at the bottom right) can finish while the downloader (download count at the bottom left) is still ongoing. If the download thread appears to hang without updating the count, it most likely means that it is downloading a large video file that will likely take some time to finish. If this becomes too big of a problem, video download can be disabled through the settings menu.

Moving forward I will make the failed save dialog more clear as to indicate the reason the save failed.

[MelanieTanaka]

@MalloyDelacroix I have left it running overnight to test this, in the morning it still showed the button in the "downloading" status and unable use the save function. It says extraction complete, and no download counter visible. Using performance monitor of windows there is no network or disk activity for redditdownloder (and using 0% CPU and very little RAM), and the folder size has not increased since last night, so I do not think it is doing anything. If it were downloading a large file I would be seeing some evidence of it, but I am not.

screen capture: https://i.imgur.com/YHm7rXT.png

[MalloyDelacroix]

I see. I will reopen this issue then. I can't reproduce this behavior and I can't seem to find a reason for it. I did notice in your log file that you have a very large number of failed extractions from imgur with a strange error code. When I thought this issue was normal behavior I thought that the extraction issue was unrelated. But now I am thinking they must be related in some way.

Are you running the app from source or using a prepackaged executable?

[MelanieTanaka]

using the latest prepackaged downloaded not too long before I opened this issue.

[MalloyDelacroix]

I have patched a piece of code that will log the exception being thrown with the error that you are getting. Are you familiar with python code to be able to run the app from source? If you can do that, recreate the problem, and send me the log again it should now (hopefully) have enough information for me to see the cause. If not I can package a new executable with the patch.

[MelanieTanaka]

I haven't ever done that before so I don't think so.

[MalloyDelacroix]

Alright, there is a new release, v2.3.1, that contains this patch. If you would, please download this version, reproduce the problem, and send me the log.

[MelanieTanaka]

I have reproduced it on 2.3.1. same behavior, here is a zip with logs and screen capture of the GUI. https://drive.google.com/file/d/1bGhr8tCzH6G4hPdj0CTi_NfnlSSd8IAD/view?usp=sharing

I notice the dialogue pop-up when saving now mentions it can't save because the downloader is running, but despite waiting for a while (with no visible disk/network activity for the process) and repeatedly clicking to stop the downloading, the download status never stopped, meaning I have to close the app without saving.

I do notice the logs is much more detailed now, it seems to mention imgur rate limit exceeded, is it downloading too quickly? If the problem is with imgur downloads hanging could the app gracefully fail and allow saving progress to resume later?

edit: also checked the help -> imgur credits and it shows it being full https://i.imgur.com/3R1sVKw.png

[EC-O-DE]

No problems so far!

[MalloyDelacroix]

It looks like the errors were in fact caused by imgur's rate limit being exceeded. Imgur's api documentation doesn't really have any information about the speed at which api calls can be made. They only mention the credit limit, and you had plenty of credits. The app was supposed to handle a rate limit error gracefully by saving the post to be extracted later, but it seems it was running into other problems trying to check the credit limit which resulted in the strange error message I was seeing in the log initially.

I have implemented a rate limiter that kicks in when it receives a rate limit error and then increases the timeout each time a new rate limit error is received. This should be minimally intrusive on the download time and only limit the rate as much as it needs to not produce errors.

I think this may fix your problem, but I am unable to test your exact situation. Please try the new release of the app and see if the problem is resolved.

[MelanieTanaka]

@MalloyDelacroix I think the new rate limiter maybe works a little bit too well :P see here: https://i.imgur.com/AlA5fW5.png that's almost 50 years

Other than this pretty much the same behavior is present. I don't want to wait 50 years to give you a new log file so I try to stop the download process but it just hangs there: https://i.imgur.com/aOFbc6P.png

here is the log file: https://drive.google.com/file/d/119WfBk87PM7GdIVhPFvpDTqQbI4wPbo6/view?usp=sharing

[MalloyDelacroix]

How many downloads do you have with the app (it will tell you in the about dialog)?

[MelanieTanaka]

total downloads says 26561

[MalloyDelacroix]

I have dug through everything in the code that touches the imgur api or makes an outside connection at all, and none of it seems to have any effect on this issue. The only thing left that I think it could be is if it is a problem on the imgur end. The library the downloader uses to connect with imgur has been deprecated, but it all appears to still be working for the time being, so I'm hesitant to point to that as the cause.

I believe what might have happened is that you have hit imgur's rate limit 5 times this month and they have blocked your credentials for a one month period. This is documented in the imgur api documentation here. The way that section is worded would make me think that the block only happens when you hit the 12,500 credit limit 5 times (which obviously, with only 26,561 downloads, you haven't) but it may apply to user credits also. The documentation does not mention whether or not a credit check would indicate a full or empty allocation and I have been unable to find anything online about it.

Right now it looks like the only options are to wait one month from when you started having problems and see if the issue is resolved, find another app that you can enter your imgur credentials into to see if you have the same problem, or apply for new credentials and see if the new credentials work.

If you do try any of the above, please keep me updated.

[Crunklydunks]

I'm having the same issue with the massive timeout value. My credentials aren't banned from Imgur either. It just seems to happen if I exceed my hour's 500 requests, and starts working again after the hour is up. If I disable all but a few of the users I'm downloading from or use a smaller user list it works just fine. Problem is, it doesn't save any progress data until after it's finished running the task, and so it always restarts from the beginning when you close it.

For the time being I just try to run my download job as frequently as possible in order to avoid going over the limit, but if it's been a few days I have to disable most of the users and then painstakingly reenable a few at a time since there isn't any way to enable/disable multiple users at once as far as I can tell.

If the project was written in C#/VB.NET I could try to poke around and see what's going on, but I have only the most basic Python knowledge, unfortunately.

[MelanieTanaka]

I have dug through everything in the code that touches the imgur api or makes an outside connection at all, and none of it seems to have any effect on this issue. The only thing left that I think it could be is if it is a problem on the imgur end. The library the downloader uses to connect with imgur has been deprecated, but it all appears to still be working for the time being, so I'm hesitant to point to that as the cause.

I believe what might have happened is that you have hit imgur's rate limit 5 times this month and they have blocked your credentials for a one month period. This is documented in the imgur api documentation here. The way that section is worded would make me think that the block only happens when you hit the 12,500 credit limit 5 times (which obviously, with only 26,561 downloads, you haven't) but it may apply to user credits also. The documentation does not mention whether or not a credit check would indicate a full or empty allocation and I have been unable to find anything online about it.

Right now it looks like the only options are to wait one month from when you started having problems and see if the issue is resolved, find another app that you can enter your imgur credentials into to see if you have the same problem, or apply for new credentials and see if the new credentials work.

If you do try any of the above, please keep me updated.

I completely forgot about this until @Crunklydunks posted and it sent me an email. I did wait well over a month (from when I last posted here until around end of January), same issues. I tried making another account and it didn't help either.

I don't know of any other apps that download from imgur that require the API creds so I can't test this out, everything else I have used just notices that the URL is imgur and gets the direct links somehow, but they are not reddit specific.

[Crunklydunks]

Just noticed there's been a bunch of commits since the last release. If I can figure out how to run it from source I'll give it a try and see if it improves things.

[MalloyDelacroix]

So after re-examining everything here, I'm thinking the original problem had to have been caused by running out of user credits and that I'm just missing something in the logging.

But, the real issue here is that the application cannot save while a download is running. I will try to make an option in the settings menu that will stop extracting new content if the user credit limit is reached, or pause the download and continue after the credits reset. I will try to make this available for the next release.