Ackater / writing.com-archival

Utility for downloading Interactives from Writing.com
https://ackater.github.io/writing.com-archival
22 stars 3 forks source link

reverted xpath changes #46

Closed GunnerGuyven closed 2 years ago

GunnerGuyven commented 2 years ago

These DOM changes were caused by the auto-scroller feature being enabled it should be disabled when scraping.

My apologies.

Ackater commented 2 years ago

Thanks!

Ackater commented 2 years ago

Auto-scroller, the premium account feature? I'd be careful using this tool with a paid account, as I mention in #40 if you hit the websites too much, they've started scrambling the contents per account.

But I was also scraping every interactive, so I'm not sure what their threshold is.

GunnerGuyven commented 2 years ago

It was indeed the premium feature. I activated an account to do this testing because I was getting tired of waiting for the lockouts to lift.

The series of events were:

That last assumption was wrong, it was a consequence of going premium to do testing. Oh well.

Thanks for the warning. I'll make a throwaway to do scraping :)