search

datawhores / OF-Scraper

A completely revamped and redesigned fork, reimagined from scratch based on the original onlyfans-scraper
MIT License
610 stars 52 forks source link

Error "Broken pipe" while scraping #337

Open dia-val opened 6 months ago

dia-val commented 6 months ago

Describe the bug After scraping a profile, the error "Broken pipe" appears and subsequent profiles do not get scraped as the error gets thrown after every action. Files seem to be downloaded correctly, but something after the scraping fails and the script will not continue correctly. This started 1-2 weeks ago. Weirdly, this is not reflected in the logs.

To Reproduce Scrape any profile or run the scraper with "--username ALL" with multiple subscriptions.

Expected behavior Normal processing of profiles.

Screenshots/Logs Traceback (most recent call last): File "/usr/local/lib/python3.11/logging/handlers.py", line 1496, in emit self.enqueue(self.prepare(record)) File "/usr/local/lib/python3.11/logging/handlers.py", line 1454, in enqueue self.queue.put_nowait(record) File "", line 2, in put_nowait File "/venv/lib/python3.11/site-packages/multiprocess/managers.py", line 821, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/venv/lib/python3.11/site-packages/multiprocess/connection.py", line 209, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/venv/lib/python3.11/site-packages/multiprocess/connection.py", line 430, in _send_bytes self._send(header + buf) File "/venv/lib/python3.11/site-packages/multiprocess/connection.py", line 387, in _send n = write(self._handle, buf) ^^^^^^^^^^^^^^^^^^^^^^^^ BrokenPipeError: [Errno 32] Broken pipe Call stack: File "/venv/bin/ofscraper", line 8, in sys.exit(main()) File "/venv/lib/python3.11/site-packages/ofscraper/main.py", line 10, in main load.main() File "/venv/lib/python3.11/site-packages/ofscraper/runner/load.py", line 23, in main run.main() File "/venv/lib/python3.11/site-packages/ofscraper/runner/run.py", line 17, in main main_helper() File "/venv/lib/python3.11/site-packages/ofscraper/runner/run.py", line 47, in main_helper picker.pick() File "/venv/lib/python3.11/site-packages/ofscraper/commands/picker.py", line 20, in pick scraper.main() File "/venv/lib/python3.11/site-packages/ofscraper/commands/scraper.py", line 88, in main scrapper() File "/venv/lib/python3.11/site-packages/ofscraper/commands/scraper.py", line 123, in scrapper process_selected_areas() File "/venv/lib/python3.11/site-packages/ofscraper/commands/scraper.py", line 41, in process_selected_areas run.run_helper(functs) File "/venv/lib/python3.11/site-packages/ofscraper/utils/run.py", line 109, in run_helper job_func() File "/venv/lib/python3.11/site-packages/ofscraper/utils/context/exit.py", line 85, in inner func(args, kwargs) File "/venv/lib/python3.11/site-packages/ofscraper/actions/process.py", line 71, in process_post normal_post_process() File "/venv/lib/python3.11/site-packages/ofscraper/utils/context/exit.py", line 85, in inner func(*args, *kwargs) File "/venv/lib/python3.11/site-packages/ofscraper/actions/process.py", line 164, in normal_post_process combined_urls = OF.process_areas(ele, model_id) File "/venv/lib/python3.11/site-packages/ofscraper/actions/scraper.py", line 476, in process_areas profile_dicts = process_profile(username) File "/venv/lib/python3.11/site-packages/ofscraper/utils/system/free.py", line 16, in inner return func(args, kwargs) File "/venv/lib/python3.11/site-packages/ofscraper/actions/scraper.py", line 323, in process_profile user_profile = profile.scrape_profile(username) File "/venv/lib/python3.11/site-packages/ofscraper/api/profile.py", line 43, in scrape_profile return scrape_profile_helper(c, username) File "/venv/lib/python3.11/site-packages/ofscraper/api/profile.py", line 64, in scrape_profile_helper log.info(

Hint: Please make sure that the log formatting is readable, use a paste site like https://paste.passtheheadphones.me/

Desktop (please complete the following information):

exe22 commented 4 months ago

I'm running the latest stable version ofscraper 3.8.16 and facing the same issue, unfortunately:

you can find the terminal response here: https://paste.passtheheadphones.me/?8f9cbff8bf891fce#2N3VnvKUuJ5q8kz1oFNvvBfUoaNXK7rYbhC9TCkWiq6q

datawhores commented 3 months ago

does this still happen on 3.10dev

dia-val commented 2 months ago

I upgraded to 3.10.2 and it still happens:

Traceback (most recent call last): File "/venv/lib/python3.11/site-packages/ofscraper/utils/logs/stdout.py", line 37, in logger_process messages = funct(timeout=constants.getattr("LOGGER_TIMEOUT")) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 2, in get File "/venv/lib/python3.11/site-packages/multiprocess/managers.py", line 821, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/venv/lib/python3.11/site-packages/multiprocess/connection.py", line 209, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/venv/lib/python3.11/site-packages/multiprocess/connection.py", line 430, in _send_bytes self._send(header + buf) File "/venv/lib/python3.11/site-packages/multiprocess/connection.py", line 387, in _send n = write(self._handle, buf) ^^^^^^^^^^^^^^^^^^^^^^^^ BrokenPipeError: [Errno 32] Broken pipe

Any idea why? Environment is still the same as mentioned above.

datawhores commented 4 weeks ago

Are you using the official docker images?

dia-val commented 3 weeks ago

Yes, just upgraded to 3.11 with this image: datawhores/of-scraper:3.11 and the error persists, although now only seems to happen after downloading content from several models in succession. Doesn't seem to happen when downloading from a single model. Maybe that helps with finding the error.