RipMeApp / ripme

Downloads albums in bulk
MIT License
3.72k stars 631 forks source link

deviantart failed to rip #1356

Open tayser1 opened 5 years ago

tayser1 commented 5 years ago
Ripme version: 1.7.85
Java version: 10.0.2 64 bit
Operating system: Windows 10
Exact URL you were trying to rip when the problem occurred: https://www.deviantart.com/mewitti/gallery/?catpath=/
Please include any additional information about how to reproduce the problem:

insert own deviantart cookie in rip.properties located at local appdata "auth=cookie;auth_secure=cookie;userinfo=cookie" try rip deviantart user gallery

Expected Behavior

Rip deviantart gallery

Actual Behavior

Not downloading -> Error: null

MrPlaygon commented 5 years ago

I am currently experiencing the same problem and working on a solution. This null error comes from DA thinking you're not a human (which the program is not) and you would have to solve a captcha. That however is impossible when you are not a human (at least for me).

@cyian-1756 I want to know your opinion what to do with the deviantart ripper. I see two options atm (correct me if I'm wrong pls):

  1. Change the DA Ripper back to use the Deviantart API -> Makes it stable again
  2. Change the current DA Ripper to not log in. This might (did not try in depth yet) solve some problems regarding downloads but will result in sometimes incomplete galleries because you have to be logged in to view certain art pieces.

I personally would recommend the first for now because I get more time to build a new DA Ripper and to test some things on DA Website. Especially the weird 401 Errors I encoutered recently.

Cheers and a great day MrPlaygon

cyian-1756 commented 5 years ago

@MrPlaygon

Change the DA Ripper back to use the Deviantart API -> Makes it stable again

I think this would be the best option

rautamiekka commented 5 years ago

@MrPlaygon

Change the DA Ripper back to use the Deviantart API -> Makes it stable again

I think this would be the best option

That was already discussed, concluding that using the API makes zero sense, and gives very limited amount of data.

MrPlaygon commented 5 years ago

I know we already discussed that but I still think it is worth considering to go back because the HTML website has so many weird things going on. For example the login problem. With that many users who use ripme there is no way to prevent DeviantArt from noticing that the user is a bot.

However I will still try to make an html ripper work without login and see how efficient it is.

MrPlaygon commented 5 years ago

I found the root of the problem. Deviantart has some upcoming design changes in their website (new HTML and CSS). Sometimes the new and sometimes the old design loads. Therefore it sometimes works sometimes not. I will pause development of an HTML Ripper until DA fully switches to the new design.