Added support for downlading files with various extensions

[szero]

Hello, I just got this little program and found out that it downloads all images as jpg's by default. I looked through imgur source code and found way to get extensions. Instead of getting hashes by looking through div tags, I'm fetching list of tuples (each have hash and extension) from image field in javascript function named widgetFactory.mergeConfig('gallery' ....

[szero]

I'd like also mention that this js function contains info about number of pictures in album but since we need list of all hashes either way, its redundant to get the number from it.

[jinnatar]

Does not work as advertised. Likely to be due to recent "interesting" changes on the imgur side. The master branch without this pull request correctly identifies 33 images (with pull request, 58) and downloads them all successfully, albeit without the correct extension.

Sample run that mostly fails, succeeds on a few: imguralbum.py https://imgur.com/a/aYJkp deleteme Results: Found 58 images in album Downloading Image 1 http://i.imgur.com/ >> deleteme/01- Downloading Image 2 http://i.imgur.com/ >> deleteme/02- Downloading Image 3 http://i.imgur.com/ >> deleteme/03- Downloading Image 4 http://i.imgur.com/ >> deleteme/04- Downloading Image 5 http://i.imgur.com/G2vGB.gif >> deleteme/05-G2vGB.gif Downloading Image 6 http://i.imgur.com/ >> deleteme/06- Downloading Image 7 http://i.imgur.com/ >> deleteme/07- Downloading Image 8 http://i.imgur.com/ >> deleteme/08- Downloading Image 9 http://i.imgur.com/ >> deleteme/09- Downloading Image 10 http://i.imgur.com/ >> deleteme/10- Downloading Image 11 http://i.imgur.com/ >> deleteme/11- Downloading Image 12 http://i.imgur.com/ >> deleteme/12- Downloading Image 13 http://i.imgur.com/ >> deleteme/13- Downloading Image 14
http://i.imgur.com/ >> deleteme/14- Downloading Image 15 http://i.imgur.com/ >> deleteme/15- Downloading Image 16 http://i.imgur.com/ >> deleteme/16- Downloading Image 17 http://i.imgur.com/ >> deleteme/17- Downloading Image 18 http://i.imgur.com/ >> deleteme/18- Downloading Image 19 http://i.imgur.com/ >> deleteme/19- Downloading Image 20 http://i.imgur.com/ >> deleteme/20- Downloading Image 21 http://i.imgur.com/ >> deleteme/21- Downloading Image 22 http://i.imgur.com/TtRut.gif >> deleteme/22-TtRut.gif Downloading Image 23 http://i.imgur.com/ >> deleteme/23- Downloading Image 24 http://i.imgur.com/ >> deleteme/24- Downloading Image 25 http://i.imgur.com/ >> deleteme/25- Downloading Image 26 http://i.imgur.com/ >> deleteme/26- Downloading Image 27 http://i.imgur.com/D4D0L.gif >> deleteme/27-D4D0L.gif Downloading Image 28 http://i.imgur.com/ >> deleteme/28- Downloading Image 29 http://i.imgur.com/ >> deleteme/29- Downloading Image 30 http://i.imgur.com/ >> deleteme/30- Downloading Image 31 http://i.imgur.com/ >> deleteme/31- Downloading Image 32 http://i.imgur.com/IrudDoE.gif >> deleteme/32-IrudDoE.gif Downloading Image 33 http://i.imgur.com/ >> deleteme/33- Downloading Image 34 http://i.imgur.com/ >> deleteme/34- Downloading Image 35 http://i.imgur.com/ >> deleteme/35- Downloading Image 36 http://i.imgur.com/ >> deleteme/36- Downloading Image 37 http://i.imgur.com/ >> deleteme/37- Downloading Image 38 http://i.imgur.com/UyLJJ.gif >> deleteme/38-UyLJJ.gif Downloading Image 39 http://i.imgur.com/ >> deleteme/39- Downloading Image 40 http://i.imgur.com/ >> deleteme/40- Downloading Image 41 http://i.imgur.com/ >> deleteme/41- Downloading Image 42 http://i.imgur.com/ >> deleteme/42- Downloading Image 43 http://i.imgur.com/ >> deleteme/43- Downloading Image 44 http://i.imgur.com/ >> deleteme/44- Downloading Image 45 http://i.imgur.com/ >> deleteme/45- Downloading Image 46 http://i.imgur.com/ >> deleteme/46- Downloading Image 47 http://i.imgur.com/ >> deleteme/47- Downloading Image 48 http://i.imgur.com/ >> deleteme/48- Downloading Image 49 http://i.imgur.com/ >> deleteme/49- Downloading Image 50 http://i.imgur.com/ >> deleteme/50- Downloading Image 51 http://i.imgur.com/ >> deleteme/51- Downloading Image 52 http://i.imgur.com/ >> deleteme/52- Downloading Image 53 http://i.imgur.com/ >> deleteme/53- Downloading Image 54
http://i.imgur.com/ >> deleteme/54- Downloading Image 55 http://i.imgur.com/ >> deleteme/55- Downloading Image 56 http://i.imgur.com/ >> deleteme/56- Downloading Image 57 http://i.imgur.com/ >> deleteme/57- Downloading Image 58 http://i.imgur.com/ >> deleteme/58-

Done!

Upon checking the results: %> file * 01-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 02-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 03-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 04-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 05-G2vGB.gif: GIF image data, version 89a, 312 x 212 06-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 07-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 08-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 09-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 10-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 11-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 12-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 13-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 14-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 15-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 16-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 17-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 18-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 19-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 20-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 21-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 22-TtRut.gif: GIF image data, version 89a, 480 x 270 23-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 24-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 25-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 26-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 27-D4D0L.gif: GIF image data, version 89a, 522 x 299 28-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 29-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 30-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 31-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 32-IrudDoE.gif: GIF image data, version 89a, 400 x 224 33-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 34-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 35-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 36-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 37-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 38-UyLJJ.gif: GIF image data, version 89a, 278 x 166 39-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 40-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 41-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 42-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 43-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 44-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 45-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 46-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 47-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 48-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 49-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 50-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 51-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 52-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 53-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 54-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 55-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 56-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 57-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators 58-: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators

[jinnatar]

For comparison, files as downloaded by current master; %> file deleteme/* deleteme/01-eZ6La.jpg: GIF image data, version 89a, 450 x 297 deleteme/02-9dVZG.jpg: GIF image data, version 89a, 320 x 240 deleteme/03-G2vGB.jpg: GIF image data, version 89a, 312 x 212 deleteme/04-PNlTK.jpg: GIF image data, version 89a, 300 x 192 deleteme/05-VcAnC.jpg: GIF image data, version 89a, 258 x 200 deleteme/06-CKqbO.jpg: GIF image data, version 89a, 320 x 180 deleteme/07-AuR6z.jpg: GIF image data, version 89a, 304 x 268 deleteme/08-Lf8LR.jpg: GIF image data, version 89a, 168 x 120 deleteme/09-PKwyw.jpg: GIF image data, version 89a, 200 x 113 deleteme/10-snbc6.jpg: GIF image data, version 89a, 300 x 168 deleteme/11-89CRY.jpg: GIF image data, version 89a, 500 x 248 deleteme/12-TtRut.jpg: GIF image data, version 89a, 480 x 270 deleteme/13-QH3pk.jpg: GIF image data, version 89a, 404 x 251 deleteme/14-Lb6Gh.jpg: GIF image data, version 89a, 200 x 200 deleteme/15-D4D0L.jpg: GIF image data, version 89a, 522 x 299 deleteme/16-scd62.jpg: GIF image data, version 89a, 300 x 168 deleteme/17-FWWQMyv.jpg: GIF image data, version 89a, 168 x 120 deleteme/18-IrudDoE.jpg: GIF image data, version 89a, 400 x 224 deleteme/19-M6ehHWG.jpg: GIF image data, version 89a, 305 x 170 deleteme/20-HiDqhhn.jpg: GIF image data, version 89a, 245 x 140 deleteme/21-3WkwXkA.jpg: GIF image data, version 89a, 400 x 226 deleteme/22-UyLJJ.jpg: GIF image data, version 89a, 278 x 166 deleteme/23-antqm.jpg: GIF image data, version 89a, 296 x 248 deleteme/24-ZNYtm.jpg: GIF image data, version 89a, 285 x 214 deleteme/25-Cqyno.jpg: GIF image data, version 89a, 207 x 119 deleteme/26-h78Zw.jpg: GIF image data, version 89a, 500 x 300 deleteme/27-d749N.jpg: GIF image data, version 89a, 500 x 277 deleteme/28-cQut3.jpg: GIF image data, version 89a, 400 x 225 deleteme/29-78R2VnW.jpg: GIF image data, version 89a, 600 x 338 deleteme/30-4sBFaSp.jpg: GIF image data, version 89a, 449 x 262 deleteme/31-M5JlyCP.jpg: GIF image data, version 89a, 406 x 270 deleteme/32-8ZX0miN.jpg: GIF image data, version 89a, 350 x 197 deleteme/33-RzZgMf6.jpg: GIF image data, version 89a, 500 x 235

[szero]

@Artanicus Thanks for noticing that! I looked through imgur source again and it seemed like the code didn't change and yet the regex stopped matching properly. Anyway I simplified it and now it works like it should. I also added method that counts up occurrences of extensions in album.

[jinnatar]

Works well as of 82515d5, solves my use case beautifully.

[scresante]

Tested out @Szero 's version against the latest from @alexgisby , and it solves the file extension problem. This was a pretty serious bug for me and now it's fixed. Thanks Szero!

[alexgisby]

Guys this is totally fantastic! Apologies for not getting to it sooner, top work! <3