Ornias1993
commented
4 years ago It's always mentioned that this script scraping users causes members to be banned
The scraping is undetectable by Fetlife. The OLD version (not taken over by me) had a feature for legacy search which was detectable.
With this script I always went slow with scraping and never had any problems This script doesn't include a feature to decide how fast or what you are scraping. I think you are confusing a different script. fetlife-aslsearch-reborn doensn't include legacy search
Waiting for a database to fill with users is pretty useless to me as I'm probably the only person in my area to use this script, except maybe for one or two others
Next version includes about 4 miljoen extra users, prescraped. Which has been mentioned a few times already. Besides, It doesn't just scrape users you visit, but all users listed on a page.
Would it be possible to scrape users in a certain area and do it at a really slow rate so as to be undetected? For example I could set it going overnight and check it in the morning? That would be really useful.
Prescraping is already listed as requested feature in #67 . However, prescraping will not be available to regular users, because it would open up the server for DDOS attacks. Prescraping will only be available for the host and just run all fetlife users 1 by 1. Closing this due to duplicate.
Would it also be possible to only scrape the same users once or at least indicate a particular user has already been seen by you? Perhaps adding a check box next to each profile that remains after each scrape? In the same way streaming platforms leave a check next to films or episodes already watched. This would make it much more efficient and prevent opening profiles only to discover they haven't been active for 8 years!
That would mean I would need to support every design change by fetlife. That is not going to happen. Scraping is done automatically when ever you "see" a user (not just opening a profile). It's not meant to be used in a way where you open up profiles specifically to be scraped., Besides that would require all sorts of local data to be stored, which needs a complete redesign AGAIN. Fetlife ASL Reborn is Not meant as a "has seen" indicator on profiles, you ofcourse are free to create a plugin for that and (if opensource in a compatible licence) might or might not get implemented in fetlife ASL. But currently it's not even considered. Closing because "not going to happen in at least 2-4 years"
Description
Category
Feature
Detailed Bug Report
Steps to Reproduce
Please enter the steps to reproduce the bug or behaviour: 1. 2. 3.
I dunno if this is OK but I've used this script off and on for a while now and there's a couple of things I'd like to see in it.
It's always mentioned that this script scraping users causes members to be banned. I personally never experienced this. I certainly tripped fetlife's systems where they locked me out for a while but that was only for a few hours and that was on a different asl scraper script. With this script I always went slow with scraping and never had any problems.
Waiting for a database to fill with users is pretty useless to me as I'm probably the only person in my area to use this script, except maybe for one or two others. So I have a couple of suggestions:
Would it be possible to scrape users in a certain area and do it at a really slow rate so as to be undetected? For example I could set it going overnight and check it in the morning? That would be really useful.
Would it also be possible to only scrape the same users once or at least indicate a particular user has already been seen by you? Perhaps adding a check box next to each profile that remains after each scrape? In the same way streaming platforms leave a check next to films or episodes already watched. This would make it much more efficient and prevent opening profiles only to discover they haven't been active for 8 years!
I don't know if any of these are possible but I think they would be extremely useful :)