search

montroly / fansyloader

Easy to use onlyfans and fansly scraper
46 stars 3 forks source link

Posts could not be saved to database: bun: Insert(empty *reflect.rtype) #4

Open abesnowman opened 3 years ago

abesnowman commented 3 years ago

Describe the bug

After auto-updating to 0.6.0, recieved the following and the program halts.

fansyloader / v0.6.0

SUCCESS Configuration file loaded: C:\Apps\fansyloader\config\config.toml DEBUG Application start SUCCESS Webserver listening on 127.0.0.1:8080 • Step 1: Open the Chrome or Firefox browser and do NOT use private mode • Step 2: Log in to REDACTED with the same browser • Step 3: Open the url http://127.0.0.1:8080 with the same browser • Step 4: Stay logged in

SUCCESS Request received SUCCESS User-agent detected SUCCESS Browser database found SUCCESS Authentication successful. User: Abe Snowman - Yeti Vigilante ☃️ () INFO Download will be started

INFO Load purchased posts (offset=0) DEBUG Purchased posts loaded DEBUG Save posts in db WARNING Posts could not be saved to database: bun: Insert(empty reflect.rtype) ERROR Error while scraping the files: bun: Insert(empty reflect.rtype) ERROR Links could not be downloaded: bun: Insert(empty *reflect.rtype)

Desktop (please complete the following information):

Logs emailed.

abesnowman commented 3 years ago

FYI, within seconds of posting this issue (before redacting the URL of the content host and my twitter username, for OAUTH), my OF account was banned/deleted.

montroly commented 3 years ago

Hi, first of all thanks for the issue. But WTF? How does the ban show?

Does anyone else have a problem with this? I run the scraper several times a day and haven't had a ban yet.

montroly commented 3 years ago

SUCCESS Authentication successful. User: Abe Snowman - Yeti Vigilante snowman_with_snow ()

Is that your real username at the top of the console pulldown? :see_no_evil:

abesnowman commented 3 years ago

It was, before I removed it - doesn't matter now, I reckon. (username) was what within the parenthesis.

Edited by montroly: username replaced

abesnowman commented 3 years ago

I suspect they have a scraper (ironically) trolling github for possible usernames of OF users, and drop the banhammer when there's a match. I ranted about it a bit, but also noticed that their TOS effectively prohibit anyone from accessing the site, so .... anyways, not sure how much I can help debug this, now.

I did go in and flip the last 3 entries in the sqlite db to "downloaded == 0" and it made no noticeable difference. I also flipped the log bit and the debug bit in the config file.

montroly commented 3 years ago

I have created a release which should solve the problem, but unfortunately this does not bring you anything anymore. I don't think that the ban has anything to do with the error, because it only refers to the local installation.

It was, before I removed it - doesn't matter now, I reckon. (username) was what within the parenthesis.

So you replaced the name in the issue opening? (Username and display name)

I suspect they have a scraper (ironically) trolling github for possible usernames of OF users, and drop the banhammer when there's a match. I ranted about it a bit, but also noticed that their TOS effectively prohibit anyone from accessing the site, so .... anyways, not sure how much I can help debug this, now.

I find it strange how quickly the ban was applied. Maybe they really use a scraper but it's crazy to just ban usernames from Github. I also saw an issue about a ban on another scraper, but didn't expect to read about it here so quickly. :-/

abesnowman commented 3 years ago

Correct - originally, the line was: SUCCESS Authentication successful. User: Abe Snowman - Yeti Vigilante ☃️ (abesnowman)

montroly commented 3 years ago

I'm just a little confused because you removed the username but not the display name? Best I remove the username output from the program.

If you do not know how to get a new account, you may find help in this forum: https://forum.sexy-egirls.com/forums/helping-the-community.11/

abesnowman commented 3 years ago

The display name is a variable that can be easily changed by the user on Twitter, and often is - and is not required to be unique. The username is much more static and does not change frequently because it is unique across the service, and thus would be the ideal key for a user on OF created by Twitter OAUTH, and potentially the most useful data to pull out of a search for potential violators of the AUP based on github projects.

Does that make sense?

montroly commented 3 years ago

Does that make sense?

Yes, that makes sense. But the display name is also in the OF database and can be used to identify users if it is unique. The username would be better of course but I think the display name should be sufficient for OF to identify a user in most cases. Thus I think I should not output both of them.

abesnowman commented 3 years ago

Yes, I would remove any identifying data, and also the name of the service being scraped - in case it's also in the search criteria.

aukeba commented 2 years ago

I'm having this problem as well but without the ban as far as I can see. I've tried everything but the manual mode as quite frankly I can't figure that one out. Is there any fixes for this?