Frequent errors and exit with very large collection

[alexkillby]

Firstly, thank you so much for a one-of-a-kind tool. This is a lifesaver!

I'm running into an issue with a very large collection (9786 albums, 487886 photos). It was easy to set up the job and run it, but I noticed after about 5-10 minutes I would randomly receive an error along the lines of the following... If I simply run the same command again the job restarts, scans all completed albums to find its spot again, and the error does not occur in the same spot, so it seems to not be a problem with the actual album or file from the first time around.

Traceback (most recent call last):
  File "smdl.py", line 108, in <module>
    image_req = get_json(image["Uris"][largest_media]["Uri"])
  File "smdl.py", line 45, in get_json
    return json.loads(pres[-1].text)
IndexError: list index out of range

At the moment I'm using a tool called "immortal" to run this python script supervised by another program which simply restarts it automatically if the job exits, but as I'm sure you can imagine the fact that it has to start again by scanning every single album means that this will exponentially grow in time to complete, and I have a feeling it might be impossible to complete for that reason with so many albums.

I have run apt update/upgrade. Have tried Windows and Ubuntu (seems to run longer on Ubuntu so that's what I'm using now).

If the tool simply continued through to the end, it would take a while but I know it would work perfectly. It's this constant restarting and rescanning that has ended up being a bit of a brick wall for me. Any suggestions?

[andyjsmith]

Try with the latest commit 9f4f869 and see if that addresses it. Since it works for you if you try again, I made it retry requests if they fail like that. And if they still don't work, it should now just skip them and continue instead of crashing the whole script like you showed. Let me know if this works!

[alexkillby]

Hi Andy, thank you so much! That seems to have improved things, but now I'm encountering the following error which seems to be a different issue along the same lines... Would it be possible to try to address it in the same way? Thank you so much again.

Traceback (most recent call last):
  File "smdl.py", line 96, in <module>
    images = get_json(album["Uri"] + "!images")
  File "smdl.py", line 48, in get_json
    return json.loads(pres[-1].text)
IndexError: list index out of range

[andyjsmith]

Give the latest a try, I may or may not have caught the wrong error

[alexkillby]

Looks like that solved it, thank you so much Andy, what a great tool! I really appreciate your quick response and willingness to help out!