santosderek / Imgur-To-Folder

Download Imgur albums and images to desired folder
Other
80 stars 9 forks source link

Can I use this program to backup my 80 000 images? #3

Closed Maveee closed 1 year ago

Maveee commented 6 years ago

First of all thank you for creating a free application like this one.

I have been searching for the past months for a solution to download my 80 000 images on imgur, as there is currently no method to downloading them all.. (download account images doesn't work if you have a few k)

Could it be done through this application?

Thank you!

santosderek commented 6 years ago

The application can download as many images as you give it through the command line. So if you had all 80,000 links and ran the command 'itf' with 80000 links after it, then it would download all 80000.

Recently, I found out that the official Imgur Python Module has been deprecated. I am currently implementing a way to get past the limit when requesting "favorited" items through the Imgur Rest API.

If you check back in a week or two I should have a build that will solve this issue. I'll let you know when I do as well!

Maveee commented 6 years ago

Thank you for answering, now I only need a list of all 80 000 images somehow.. that's another challenge

santosderek commented 6 years ago

Hello!

I managed to implement Imgur's Rest API now within the program, and should now theoretically be able to download all 80000 of your favorited Imgur links. I don't have 80000 images to test so if you're interested, you can give it a go.

The only downside is when looking through the documentation I found that there is a daily 12,500 request limit on Imgur's side. So Imgur might stop you from downloading all 80,000 in just one day.

If any issues with the program pop-up, please feel free to inform me!

Also, since there is now a new implementation, the way of authenticating has changed and steps can be found at the bottom of the README.md.

jherazob commented 6 years ago

Can you make it to download all of your uploaded images? I've been uploading pics to my account since accounts started and have a few, and want to back them up locally, favorites are nice to get and have, but i'd be far more interested on the things i personally uploaded to my account

santosderek commented 6 years ago

Yes I will be able to implement this. Once the feature has been added I will let you know!

Thank you!

Maveee commented 6 years ago

Yes I will be able to implement this. Once the feature has been added I will let you know!

Please let me know as well!

santosderek commented 6 years ago

Hello!

I have added the ability to download all account uploaded images within the program by using the following command after successfully installing Imgur-To-Folder:

imgurToFolder --download-account-images | OR | imgurToFolder -dai

Please see the README.md for installation instructions if needed.

Just let me know if there was anything else that I could improve!

Maveee commented 6 years ago

Thanks Derek!

Just one question: I'm getting this "NameError: name 'imgurToFolder' is not defined" when trying to download all my account images.

My json looks like this:

{ "access_token": "", "client_id": "SECRETSAKDFKLDSAJF", "client_secret": "SECRETADSLKFJKASLJ", "download_folder_path": "", "refresh_token": "", "single_images_folder": true }

jherazob commented 6 years ago

Just finished downloading my 3.6Gb of personal pics, all my thanks! Will do the favorites later.

@Maveee You probably haven't done the setup as described in the readme, i ran into the issue that once installed and executing it, after i was asked to open an url and paste the main page redirection it ran into an error because of permissions, so maybe you haven't done that, as the final json also has all the other fields filled up.

Maveee commented 6 years ago

@jherazob I get an error when trying the imgurToFolder cmd

https://i.imgur.com/iwg0CzO.png

Maveee commented 6 years ago

I'm one step further now, (installed correctly and json okay) but the command doesn't do anything (the images don't start downloading)

https://i.imgur.com/AhdiSuA.png

Any idea why this might be happening?

jherazob commented 6 years ago

Ah, Windows. I bet on something on your Python environment, but i can't say what as i haven't touched windows in a decade...

santosderek commented 6 years ago

Hey there @Maveee !

If you installed the program by using python3 setup.py install you shouldn't need the py in front of imgurtofolder. The command to run should look like

C:/...>imgurtofolder -dai

https://i.imgur.com/myyMubv.png

Looking at the image you shared, you included the py in the beginning of the line which will give an error:

...\Python\Python36\python.exe: can't open file 'imgurtofolder': [Errno 2] No such file or directory

since it is looking for a file / folder with the name imgurtofolder. The commands ImgurToFolder and itf are both entrypoints so they should be able to run in any directory within your file system, and do not need to be ran within the imgurtofolder folder.

Maveee commented 6 years ago

@santosderek thank you for replyling.

Here's what I get with and without py

https://i.imgur.com/Q0c051n.png

jherazob commented 6 years ago

Sounds to me like you need to make sure that your Python environment is fully working first, it doesn't seem like it's registering python scripts as executables. It might just be a matter of reinstalling the Python distribution or doing some file association, no idea.

Maveee commented 6 years ago

@jherazob I managed to fix it by modifying the python installation and including environment variables.

EDIT: I'm getting errors after a few minutes

https://i.imgur.com/atzNgwE.png?2

jherazob commented 6 years ago

That's the expected behavior, now you need to open that URL, then paste back the resulting one, that part is on the Readme. I think it's pretty safe that at this point it's working on your machine.

Maveee commented 6 years ago

@jherazob I edited my message, pasting the url works, but then I get the above problem.

santosderek commented 6 years ago

@Maveee Did it start downloading some images before the error or was it instant?

I know from the docs that imgur allows up to 12,500 requests per day, then will start limiting / refusing requests. I'm sure you're probably not hitting that amount, but it seems imgur might be refusing incoming connections based off of the following output:

'...max retries exceeded with url...' 

'...A connection attempt failed because the party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.'. 

It should reset after a day, but if this still becomes an issue I will implement a time delay between downloads that can be passed via command line in order to limit the amount of attempts with Imgur's server.

Also if it did start downloading some of the photos, the program will automatically skip over the already downloaded files and will not attempt any download requests.

Maveee commented 6 years ago

@santosderek Looks like it downloaded 1800 images inside the folder, if I try to re-run I get this:

https://i.imgur.com/JcpFpXs.png

I will try again in 3 hours, 24h should have passed by then.

Also good to know it won't redownload the already downloaded ones!