search

datawhores / OF-Scraper

A completely revamped and redesigned fork, reimagined from scratch based on the original onlyfans-scraper
MIT License
701 stars 59 forks source link

Models do not download, give precheck issues on 3.10.20, 3,11B1 #440

Closed ExplorerNL closed 4 months ago

ExplorerNL commented 4 months ago

Describe the bug

Some model their pictures do not download, give precheck issues on 3.10.20, 3,11B1

To Reproduce

ofscraper --action download --posts all -ts --username ALL @pause

Expected behavior

It to download the models like previous versions

Screenshots/Logs

I see this in console: [after._return_val:23] precheck failed for timeline using db [after._return_val:23] precheck failed for archived using db

I see this in logging: 2024-07-20 07:30:49:[sessionmanager._wait_picker:157] sleeping for 3.83721415230145 seconds before retry 2024-07-20 07:30:49:[main_download.main_download_downloader:116] Number of Open Files -> 7 2024-07-20 07:30:49:[main_download.main_download_downloader:119] Open Files -> [('C:\Users\your_username\.config\ofscraper\main_profile\cache_sql\cache.db-shm', -1), ('C:\Users\your_username\AppData\Local\Programs\Python\Python311\Scripts\ofscraper.exe', -1), ('C:\Users\your_username\.config\ofscraper\main_profile\cache_sql\cache.db', -1), ('C:\Users\your_username\.config\ofscraper\logging\ofscraper_main_profile_2024-07-20.log', -1), ('C:\your_username\System32\en-US\kernel32.dll.mui', -1), ('C:\your_username\System32\en-US\KernelBase.dll.mui', -1), ('C:\Users\your_username\.config\ofscraper\main_profile\cache_sql\cache.db-wal', -1)] 2024-07-20 07:30:55:[data.fresh_data_handler_main:16] Media:35889814 Post:4745746 fresh download for media https://uk.upload.onlyfans.com/files/BF67D67E-FB8B-4771-9CD5-E9C65296BEF2.MP4 2024-07-20 07:30:55:[data.fresh_data_handler_main:20] Media:35889814 Post:4745746 resume_size: 0 2024-07-20 07:30:55:[main_download.main_download_sendreq:135] Media:35889814 Post:4745746 download temp path \192.168.50.2\Adult\Web\OnlyFans\ava-grace\Posts\Videos\BF67D67E_FB8B_4771_9CD5_E9C65296BEF2_source_35889814.part 2024-07-20 07:30:55:[main_download.send_req_inner:151] Media:35889814 Post:4745746 Downloading media with url https://uk.upload.onlyfans.com/files/BF67D67E-FB8B-4771-9CD5-E9C65296BEF2.MP4 2024-07-20 07:30:56:[helpers.inner:11] Cannot connect to host uk.upload.onlyfans.com:443 ssl:<ssl.SSLContext object at 0x000002B1EAF47BF0> 2024-07-20 07:30:56:[helpers.inner:11] Traceback (most recent call last): File "C:\Users\your_username\AppData\Local\Programs\Python\Python311\Lib\site-packages\aiohttp\connector.py", line 1203, in _create_direct_connection hosts = await self._resolve_host(host, port, traces=traces) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\your_username\AppData\Local\Programs\Python\Python311\Lib\site-packages\aiohttp\connector.py", line 880, in _resolve_host return await asyncio.shield(resolved_host_task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\your_username\AppData\Local\Programs\Python\Python311\Lib\site-packages\aiohttp\connector.py", line 917, in _resolve_host_with_throttle addrs = await self._resolver.resolve(host, port, family=self._family) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\your_username\AppData\Local\Programs\Python\Python311\Lib\site-packages\aiohttp\resolver.py", line 33, in resolve infos = await self._loop.getaddrinfo( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\your_username\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 868, in getaddrinfo return await self.run_in_executor( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\your_username\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures\thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\your_username\AppData\Local\Programs\Python\Python311\Lib\socket.py", line 962, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ socket.gaierror: [Errno 11001] getaddrinfo failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "C:\Users\your_username\AppData\Local\Programs\Python\Python311\Lib\site-packages\ofscraper\classes\sessionmanager\sessionmanager.py", line 495, in requests_async r = await self._aio_funct( ^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\your_username\AppData\Local\Programs\Python\Python311\Lib\site-packages\ofscraper\classes\sessionmanager\sessionmanager.py", line 588, in _aio_funct r = await self._session._request(method, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\your_username\AppData\Local\Programs\Python\Python311\Lib\site-packages\aiohttp\client.py", line 578, in _request conn = await self._connector.connect( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\your_username\AppData\Local\Programs\Python\Python311\Lib\site-packages\aiohttp\connector.py", line 544, in connect proto = await self._create_connection(req, traces, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\your_username\AppData\Local\Programs\Python\Python311\Lib\site-packages\aiohttp\connector.py", line 944, in _createconnection , proto = await self._create_direct_connection(req, traces, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\your_username\AppData\Local\Programs\Python\Python311\Lib\site-packages\aiohttp\connector.py", line 1209, in _create_direct_connection raise ClientConnectorError(req.connection_key, exc) from exc aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host uk.upload.onlyfans.com:443 ssl:<ssl.SSLContext object at 0x000002B1EAF47BF0>

Config

{ "main_profile": "main_profile", "metadata": "{configpath}/{profile}/.data/{modelusername}{model_id}", "discord": "", "file_options": { "save_location": "//192.168.50.2/Adult/Web/OnlyFans", "dir_format": "{model_username}/{responsetype}/{mediatype}/", "fileformat": "{date}{filename}.{ext}", "textlength": 0, "space_replacer": " ", "date": "YYYY-MM-DD", "text_type_default": "letter", "truncation_default": true }, "download_options": { "filter": [ "Images", "Audios", "Videos" ], "auto_resume": true, "system_free_min": 0, "max_post_count": 0 }, "binary_options": { "ffmpeg": "C:\Users\Windows\.config\ofscraper\bin\ffmpeg.exe" }, "cdm_options": { "private-key": "C:\Users\Windows\.config\ofscraper\private_key.pem", "client-id": "C:\Users\Windows\.config\ofscraper\client_id.bin", "key-mode-default": "manual", "keydb_api": "" }, "performance_options": { "download_sems": 6, "thread_count": 2 }, "content_filter_options": { "block_ads": true, "file_size_max": 0, "file_size_min": 0, "length_max": null, "length_min": null }, "advanced_options": { "code-execution": false, "dynamic-mode-default": "datawhores", "backend": "aio", "downloadbars": false, "cache-mode": "sqlite", "appendlog": true, "custom_values": null, "sanitize_text": false, "temp_dir": null, "remove_hash_match": false, "infinite_loop_action_mode": false, "post_download_script": null, "enable_auto_after": true, "default_user_list": "main", "default_black_list": "" }, "responsetype": { "timeline": "Posts", "message": "Messages", "archived": "Archived", "paid": "Messages", "stories": "Stories", "highlights": "Stories", "profile": "Profile", "pinned": "Posts", "streams": "Streams" }, "overwrites": { "audios": {}, "videos": {}, "images": {}, "text": {} } }

System Info

datawhores commented 4 months ago

This may be a bad link or download uk.onlyfans.com

is not the typical download link anymore

Update: Yes giving that the aiohttp error is basically saying that the link is not connectable, and the fact that the link is very close to the us.upload.onlyfans.com, which was also found to be unconnectable and marked as a url to skip. The uk.upload.onlyfans.com has also been marked as a url to skip

ExplorerNL commented 4 months ago

This may be a bad link or download uk.onlyfans.com

is not the typical download link anymore

Update: Yes giving that the aiohttp error is basically saying that the link is not connectable, and the fact that the link is very close to the us.upload.onlyfans.com, which was also found to be unconnectable and marked as a url to skip. The uk.upload.onlyfans.com has also been marked as a url to skip

What about the prechecks errors? I also got this after a while In the logging

Screenshot_2024-07-20_at_08 05 29
datawhores commented 4 months ago

Well I've had way more aggressive download settings and I've been fine with that. So chances are it is something else causing the error

In my experience the rate limiting comes from using the api to scrape

Are you able to show more of the error, I just want to know where in the script the error is coming from

Also a few questions

ExplorerNL commented 4 months ago

No like action No VPN No I use it for months already Some creators are ‘larger’ but not too big

Will upload the logs

ExplorerNL commented 4 months ago

Well I've had way more aggressive download settings and I've been fine with that. So chances are it is something else causing the error

In my experience the rate limiting comes from using the api to scrape

Are you able to show more of the error, I just want to know where in the script the error is coming from

Also a few questions

  • Did you use the like action?
  • Are you on a vpn?
  • Was this after the first run
  • is this a large creator?

ofscraper_main_profile_2024-07-20.log Logs uploaded

datawhores commented 4 months ago

Are you able to update to the latest commit on 3.10b branch. If you can I would install with the --suffix arg with pipx, to install into a new virtualenv.

There are over 100 instances of cloudflare in the log However most of these seem to be coming from uk.upload.onlyfans.com, which is not accessible any more , and removing that url should get rid of a bunch of the errors

Another thing is that you can change some values to make the script less aggressive in scraping posts. Personally I leave the values as is, and allow the script to retry when a it encounters an error.

I would update first and see what the logs look after

datawhores commented 4 months ago

Please open new issue if needed