search

datawhores / OF-Scraper

A completely revamped and redesigned fork, reimagined from scratch based on the original onlyfans-scraper
MIT License
559 stars 50 forks source link

Script crashes with TypeError trying to scrape paid model page (3.10) #398

Closed guyhurst90 closed 3 weeks ago

guyhurst90 commented 4 weeks ago

Describe the bug

Script crashes trying to scrape paid model page

To Reproduce

Using release 3.10, try to scrape paid model page.

Expected behavior

Script not to raise TypeError and crash

Screenshots/Logs

 2024-06-05 01:54:39:[main.filtermediaAreas:53]  filter 1-> initial media no filter count: 26
 2024-06-05 01:54:39:[main.filtermediaAreas:62]  filter 3-> media datesort count: 26
 2024-06-05 01:54:39:[helpers.mediatype_type_filter:72]  filtering Media to images,audios,videos
 2024-06-05 01:54:39:[main.filtermediaAreas:67]  filter 4-> media post media type filter count: 26
 2024-06-05 01:54:39:[main.filtermediaAreas:72]  filter 5-> media post date filter: 26
 2024-06-05 01:54:39:[main.filtermediaAreas:77]  filter 6->  media post timed post filter count: 26
 2024-06-05 01:54:39:[main.filtermediaAreas:82]  filter 7->  media text filter count: 26
 2024-06-05 01:54:39:[main.filtermediaAreas:88]  filter 8->  media excluded text filter count: 26
 2024-06-05 01:54:39:[main.filtermediaAreas:93]  filter 9->  media download type filter count: 26
 2024-06-05 01:54:39:[main.filtermediaAreas:98]  filter 10->  media mass message filter count: 26
 2024-06-05 01:54:39:[helpers.final_media_sort:131]  Using download sort None
 2024-06-05 01:54:39:[main.filtermediaAreas:107]  filter 12->  media final sort filter count: 26
 2024-06-05 01:54:39:[main.filterPostFinal:113]  filter 1-> initial posts no filter count: 12
 2024-06-05 01:54:39:[main.filterPostFinal:118]  filter 2->  post date sort filter count: 12
 2024-06-05 01:54:39:[main.filterPostFinal:123]  filter 3-> post dupe filter count: 12
 2024-06-05 01:54:39:[main.filterPostFinal:127]  filter 4-> timed posts filter count: 12
 2024-06-05 01:54:39:[main.filterPostFinal:132]  filter 5->  post text filter count: 12
 2024-06-05 01:54:39:[main.filterPostFinal:137]  filter 6->  post excluded text filter count 12
 2024-06-05 01:54:39:[main.filterPostFinal:142]  filter 7->  mass msg filter count 12
 2024-06-05 01:54:39:[helpers.final_post_sort:265]  Using post sort None
 2024-06-05 01:54:39:[main.filtermediaFinal:14]  filter 1-> initial media no filter count: 26
 2024-06-05 01:54:39:[main.filtermediaFinal:19]  filter 2-> sorted media count: 26
 2024-06-05 01:54:39:[main.filtermediaFinal:25]  filter 3-> viewable media filter count: 26
 2024-06-05 01:54:39:[main.filtermediaFinal:32]  filter 4->  media dupe media_id filter count: 26
 2024-06-05 01:54:39:[main.filtermediaFinal:36]  filter 5->  media unviewable filter count: 26
 2024-06-05 01:54:39:[helpers.previous_download_filter:153]  reading database to retrive previous downloads
 2024-06-05 01:54:39:[request.get_request_auth:49]  getting new signature
 2024-06-05 01:54:41:[helpers.previous_download_filter:171]  Downloading unique media across all models
 2024-06-05 01:54:41:[helpers.inner:11]  'NoneType' object is not subscriptable
 2024-06-05 01:54:41:[helpers.inner:11]  Traceback (most recent call last):
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/commands/scraper/scraper.py", line 77, in main
    scrapper()
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/commands/scraper/scraper.py", line 112, in scrapper
    process_prompts()
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/utils/context/exit.py", line 92, in inner
    raise E
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/utils/context/exit.py", line 85, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/commands/scraper/scraper.py", line 58, in process_prompts
    if menu.main_menu_action():
       ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/utils/menu.py", line 41, in main_menu_action
    runner()
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/utils/context/exit.py", line 92, in inner
    raise E
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/utils/context/exit.py", line 85, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/commands/scraper/runner.py", line 48, in runner
    scrape_paid_data=download_action.scrape_paid_all()
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/utils/context/run_async.py", line 29, in inner
    raise E
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/utils/context/run_async.py", line 15, in inner
    tasks = loop.run_until_complete(coro(*args, **kwargs))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/commands/scraper/actions/download.py", line 50, in scrape_paid_all
    async for count,value,length in process_scrape_paid():
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/commands/helpers/scrape_paid.py", line 21, in process_scrape_paid
    async for ele in process_paid_dict():
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/commands/helpers/scrape_paid.py", line 26, in process_paid_dict
    user_dict = await OF.process_all_paid()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/commands/scraper/post.py", line 351, in process_all_paid
    final_medias=filters.filtermediaFinal(insert_media)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/filters/media/main.py", line 42, in filtermediaFinal
    return helpers.previous_download_filter(media, username=username, model_id=model_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/filters/media/helpers/helpers.py", line 172, in previous_download_filter
    media_ids = set(get_media_ids_downloaded(model_id=model_id, username=username))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/db/operations_/wrapper.py", line 122, in inner
    raise E
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/db/operations_/wrapper.py", line 98, in inner
    or placeholder.databasePlaceholder().databasePathHelper(
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/classes/placeholder.py", line 72, in inner
    return f(args[0], *args[1:], **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/classes/placeholder.py", line 123, in databasePathHelper
    self._variables.update({"first_letter": model_username[0].capitalize()})
                                            ~~~~~~~~~~~~~~^^^
TypeError: 'NoneType' object is not subscriptable
 2024-06-05 01:54:41:[run.main:28]  'NoneType' object is not subscriptable
 2024-06-05 01:54:41:[close.sendCloseMessage:41]  Finished Script

System Info

datawhores commented 3 weeks ago

fixed please update via the 3.10b branch

guyhurst90 commented 3 weeks ago

I tried the latest commit on 3.10b (035f5b43018ef8ef5c918c0b61b47c6317f9c698) but it gives a 400 Bad Request error trying to download (with or without scrape).

2024-06-05 20:28:00:[helpers.inner:11]  Client error '400 Bad Request' for url 'https://onlyfans.com/api2/v2/users/me'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400
 2024-06-05 20:28:00:[helpers.inner:11]  Traceback (most recent call last):
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/api/init.py", line 34, in getstatus
    me.scrape_user()
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/api/me.py", line 33, in scrape_user
    return _scraper_user_helper(c)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/api/me.py", line 53, in _scraper_user_helper
    raise E
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/api/me.py", line 38, in _scraper_user_helper
    with c.requests(constants.getattr("meEP")) as r:
  File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/classes/sessionmanager.py", line 313, in requests
    for _ in Retrying(
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/tenacity/__init__.py", line 435, in __iter__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/tenacity/__init__.py", line 368, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/tenacity/__init__.py", line 410, in exc_check
    raise retry_exc.reraise()
          ^^^^^^^^^^^^^^^^^^^
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/tenacity/__init__.py", line 183, in reraise
    raise self.last_attempt.result()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/classes/sessionmanager.py", line 367, in requests
    raise E
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/ofscraper/classes/sessionmanager.py", line 360, in requests
    r.raise_for_status()
  File "/home/luser/.local/pipx/venvs/ofscraper/lib/python3.12/site-packages/httpx/_models.py", line 761, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://onlyfans.com/api2/v2/users/me'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400

 2024-06-05 20:28:00:[checkers.check_auth:20]  Auth Failed
datawhores commented 3 weeks ago

Please read instructions https://gist.github.com/datawhores/8e8be131fb1b190b50c5324ab2ca5f89

guyhurst90 commented 3 weeks ago

Yeah, looks like changing rules to growik seems to have fixed that 400 error. With that out of the way it does look to successfully scrape the page, so closing this issue.