RipMeApp / ripme

Downloads albums in bulk
MIT License
3.67k stars 624 forks source link

Deviantart Eclipse not working!! (Please Fix Deviantart!) #1717

Open Zelsione opened 3 years ago

Zelsione commented 3 years ago

Deviantart Eclipse not working!!!!

https://www.deviantart.com/ed-benes-studio https://www.deviantart.com/ed-benes-studio/gallery

Rip complete, saved to C:\Users\xxx\Downloads\DeviantArt\Rips\deviantart_ed_gallery_featured HTTP status code 403 for URL https://www.deviantart.com/users/login

Deviantart Watch please rip!!!!!

https://www.deviantart.com/notifications/watch/deviations/Ed-Benes-Studio

Null (Other message)

rautamiekka commented 3 years ago

'Last Version' is an absolutely invalid version number. NEVER use it.

This is an old and known problem: ever since DA rolled Eclipse the support was broken in every app.

After a while of Eclipse rollup I researched the gallery/favs and came up with a pretty simple (somewhat simpler than before Eclipse) method of ripping the gallery and subsequently the favs since those systems were merged as of Eclipse, in Python 3, outside RipMe.

But it soon broke and I fixed it, but later found a better (read: foolproof) but very User-uninformative method: since the site makes a massive use of JSON databases, fetching the 1st JSON, reading the JSON to determine the pic addresses and whether there's an additional page, download the pics gathered from the JSON (cuz DA could block the access before you're done with all the JSON's instead of gathering the addresses and downloading them in a separate run. Not too likely, but possible, and such chances can't be taken), fetching the 2nd JSON, and so on. This means the User won't have a clue how long the ripping is gonna take since no ending numbers are known and shown. The JSON databases are the way to rip Eclipse, you can no longer simply process the HTML cuz the site no longer provides ways to show a set number of things at once, instead EVERYTHING is dynamically loaded, which makes my earlier Bash script that used predictable patterns useless.

However, for many years I've been suffering from a persistent (hard to recover from) lack of general motivation to do much anything beside gaming/eating/sleeping/showering, let alone rewrite the method (it's not just a matter of rewriting a few lines: the whole logic has to be rewritten, and Eclipse had introduced a complex dynamically appearing set of unwritten rules to determining the best-quality pic and the address to that pic, which I've only partially deciphered and written the logic for. Best would be using a headless Selenium instance to support login and perform all the ripping, which means a complete rewrite of the entire app), and with the problem of the app not understanding the concept of galleries/favs having >999 items cuz Eclipse uses shortened numbers (1000 = 1k, 1200 = 1.2k; no galleries having a million items has been found) to represent those (and the fact making it understand them is proving to be hard), AFAIK there's no app that can rip an Eclipse gallery/favs.

No, no attacks used, the JSON method was found purely by reading the resulting HTML and the connection headers which screamed the exact, unprotected (if DA was to block access to those JSON's the site would break like a twig, and they obviously can't have that), addresses.

Wiiplay123 commented 3 years ago

The way I've been managing logins on my custom build of RipMe 1.5.2 that I used all the way up until Eclipse broke it was just manually putting in a cookie EACH TIME you rip a gallery. If you don't manually put the cookie in each time you open a gallery page the download buttons started 403ing sooner, or just 403ed instantly.

I'm still waiting for someone to at least get basic DeviantArt ripping working, so I can look at the code and hopefully reimplement description ripping.

Zelsione commented 3 years ago

thanks information @rautamiekka (Edit java version)