search

datawhores / OF-Scraper

A completely revamped and redesigned fork, reimagined from scratch based on the original onlyfans-scraper
MIT License
691 stars 59 forks source link

Scraper gets stuck, causing BSOD #231

Closed SpoogLord closed 1 year ago

SpoogLord commented 1 year ago

This is technically a follow up to a different thread, but that thread had completely derailed into a new issue That thread (has a little more info)

For some reason, around 1 month ago the scraper started getting stuck on the message "Setting date for modified directories" after completely scraping a models page, upon getting stuck the RAM usage of the program would slowly climb until my computer would bluescreen with an unknown error. It's mildly inconsistent and seems to only be happening to me? (Judging by the lack of other issue threads mentioning this)

No matter who I scrape, there's around a 1/4 chance of it getting stuck, which would probably be fine if I weren't subbed to 22 accounts (only 2 I actually paid for), but I am.

The full message it gets stuck on: [download.setDirectoriesDate:872] PID: 13240 Setting Date for modified directories download.py:872

Things I've tried:

The only potential solution I can come up with now is reinstalling Windows, as I currently use a custom debloated installation of Windows 10 Pro, I plan to try this within the next month, and when I do I will update this thread. Otherwise I will be waiting for a response or an update. I am open to any questions or suggestions.

Video Config (Very slightly modified from last I shared it) Log (Fresh as of making this post) Batch file I use

Edit: Forgot to include the vid with --output debug

SpoogLord commented 1 year ago

I'm also just noticing that no matter what I do, when I start the scraper it says I'm using version 3.0.12. I've updated and downgraded several times and it still says 3.0.12. is that normal?

SpoogLord commented 1 year ago

So, apparently when Puk0 told me to update python, I installed a new instance of python on the later version, kept the old version of python with the old ofscraper installation, and didn't add the new python installation to my PATH, so this entire time I've been using an old version. I have just fixed this, and am now on ofscraper 3.3.10 because pip claims 3.3.11 does not exist.

As for the important part: my issue is still occurring. The scraper gets stuck, outputs the same error as before, but it doesn't seem to be increasing in RAM usage anymore, so likely no more possibility of bsod

SpoogLord commented 1 year ago

Just fresh installed windows, and it still gets stuck. I'm now on a stock windows 10 pro installation, I installed all the requirements, and the only thing I transferred were the .bat file and the config. I have no idea what's wrong at this point. The log still looks the same, the config and bat files are the same as before, the window is basically identical to the one seen in the videos I've uploaded, the errors are the same.. Idk. Nothing about this makes sense to me anymore. I guess I'm open to suggestions, but I feel like I've tried nearly everything now. Will update this thread if/when an update comes out or something new happens.

SpoogLord commented 1 year ago

So I went with a super jank workaround. I made a bat file for everyone I'm subbed to, then made a bat file that runs all the other bat files at the same time. It uhh.. it works? but individually they all run slow and it sorta maxes my CPU usage until some of the models finish, that and some of the windows never close bc of the 'setting dates' issue, so I have to close them manually. but I mean, if it works it works? if the setting dates issue can be solved I would love to not have to do this, but in my current situation this seems to be the best I can do, to scrape everyone I'm subbed to

If anyone has like, any ideas please let me know, bc this kinda sucks lol

I'm also open to switching scrapers if someone knows of one that might work better. I've tried switching to UltimaScraper, but installing that seems like a pain, I cant find any official guide and the few guides I can find seem to be outdated or they just don't work. I also used to use Taux1c(?)'s scraper, but it stopped working and he stopped updating it.

cjb900 commented 1 year ago

Since you are the only one I've seen having this issue this has to be something on your end somewhere or possibly something with the models you are subed to or may be something with your OF account hard to say. Even if this is somehow a bug in the scraper you are the only one encountering it which would make this hard to reproduce. The only thing i can suggest is if you have another computer if you run the scraper there do you have the same issue? If you don't have another computer you could try setting up a VM (with VirtualBox or VMware) and using a linux distro like Ubuntu and install the scraper there and see if it runs any better there.

SpoogLord commented 1 year ago

I don't have another computer available to me, but I did completely wipe my PC, system files and all, so unless its something wrong with my hardware (which I doubt, everything else works fine) I guess it would have to be something about my OF account? I have not tried linux, but I have no confidence in my ability to do so lol, I may try messing with that at some point though.

cjb900 commented 1 year ago

If you installed anything else after you reinstalled windows other then this scraper its possible that could be causing issues with this scraper somehow and why no one else is having this issue (at least no one else that I've seen is reporting this). If you didn't then to test if its an issue with your account you could try creating a new test OF account and see if you have the same issue there. If you have the same issue then it would most likely have to be something on your end since if this is somehow a bug in the scraper no one else is having this issue.

SpoogLord commented 1 year ago

So its not my account, which I guess is good. I opened a VM on Windows 10 home, installed nothing but the scraper, and now it works. So.. Either the issue is Windows 10 pro, which I doubt, or its something I installed after, but that list is short, Steam, Valorant, Discord, Spotify, OBS, and Logitech GHub. I don't see why any of these things would interfere, but I guess something is? Idk. Issue kinda solved, just needed a VM.

SpoogLord commented 1 year ago

May have spoke too soon, glad I didn't close thread. On my first scrape it made it through all models and all posts, on the second run it got stuck again. same config, same .bat file, same account, but in a vm with nothing else installed. I don't even know anymore. I guess it has to be something with my account? or some super niche jank with the scraper, but I doubt.

SpoogLord commented 1 year ago

not really sure if this is relevant, I cant tell what it means. The scraper seems to still be running after outputting this but it might be tied to my issue? image

(this was during scanning with -uf) (-uf did not fix my issue as it does 'setting dates' after downloading from each model anyway)

SpoogLord commented 1 year ago

another error, again, scraper still seems to be running, but this is my first time noticing this error image

SpoogLord commented 1 year ago

Some more errors, these and the previous one all happened while scraping the same model, yes it took 2+ hours to scrape one model, no I don't know why. I had logging off bc the log files take space and are moderately large, but I'll be turning it back on from now

image image image image

SpoogLord commented 1 year ago

After toying with it for a bit I noticed that when running it as admin it doesn't get stuck at "Setting Date for modified directories", but instead fails all downloads with the error "The system cannot find the path specified: [dir]" and "Error creating directory, check the directory and make sure correct permissions have been issued". I switched my directory off my nas just to test and.. suddenly it gets stuck at 'setting dates' again. I don't think this helps at all, but like, info.

cjb900 commented 1 year ago

can you share your full config.json file? Because no one is currently having these issues (if they are they aren't coming here or to discord mentioning it).

SpoogLord commented 1 year ago

can you share your full config.json file? Because no one is currently having these issues (if they are they aren't coming here or to discord mentioning it).

I already did in the initial post

BUT Massive update, I got it working.

This was my downfall:

Changing the save location to " " seems to be the same as having the dir be " " and the save location C:\Users\Sage\Data\ofscraper (?)

Simply put: it is not the same.

More complicatedly put (and possibly inaccurate, I'm guessing): The "save_location" is the path you want the directory to be in, the "dir_format" is the format of that directory. As for why the scraper had an issue with what I was trying, I'm not really sure, but if I had to guess, I had the save location as " " which made the scraper default to saving to C:/, but then in the dir format I was telling it to output to Z:/ etc, so it was looking for a separate mounted drive within my c drive..?

Anyway. It works now, everything I tried for the last like 2 months didn't work bc of my own stupidity, and I swear at some point I moved my config and it still had issues but I guess it didn't and I'm dumb, or I somehow failed to move a fucking file or smth, idk.

SpoogLord commented 1 year ago

btw this is my new, working config, not that it really matters. CDM is not set up bc i don't care enough atm, and i turned off multithreading just bc it seemed like it was causing other people issues and I was paranoid. may change it later but like, whatever Config.json