DIGITALCRIMINAL / ArchivedUltimaScraper

Scrape content from OnlyFans and Fansly
GNU General Public License v3.0
948 stars 39 forks source link

Fansly downloader fails on downloading all models when hitting a model that has deactivated their account #751

Closed ghost closed 1 year ago

ghost commented 1 year ago

When trying the scraper with Fansly on all models, it downloads from everyone I follow, however I've followed some models in the past that have now deactivated their accounts. Fansly does not afaict allow me to unfollow them, so when the scraper inevitably hits a model that no longer exists (showing their previous username), it fails with the following:

  File "/mnt/xxx/personal-hosting/scraper/start_us.py", line 83, in <module>
    asyncio.run(main())
  File "/usr/lib64/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib64/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/mnt/xxx/personal-hosting/scraper/start_us.py", line 60, in main
    api = await main_datascraper.start_datascraper(
  File "/mnt/xxx/personal-hosting/scraper/ultima_scraper/datascraper/main_datascraper.py", line 126, in start_datascraper
    await default(datascraper)
  File "/mnt/xxx/personal-hosting/scraper/ultima_scraper/datascraper/main_datascraper.py", line 106, in default
    await datascraper.process_jobs(datascraper, subscription_list, site_settings)
  File "/mnt/xxx/personal-hosting/scraper/ultima_scraper/modules/module_streamliner.py", line 531, in process_jobs
    await datascraper.start_datascraper(authed, subscription.username)
  File "/mnt/xxx/personal-hosting/scraper/ultima_scraper/modules/module_streamliner.py", line 101, in start_datascraper
    await self.profile_scraper(subscription)
  File "/mnt/xxx/personal-hosting/scraper/ultima_scraper/modules/module_streamliner.py", line 161, in profile_scraper
    header = await subscription.get_header()
  File "/mnt/xxx/personal-hosting/scraper/venv/lib64/python3.10/site-packages/ultima_scraper_api/apis/fansly/classes/user_model.py", line 627, in get_header
    return self.header["locations"][0]["location"]
TypeError: 'NoneType' object is not subscriptable

As far as I see, the only work around at the moment is to either individually scrape each creator or manually list in an array in the config the dozens of creators I follow (and maintaining that list up to date) so it never tries to download from the deactivated accounts.

ghost commented 1 year ago

I mistook an active creator for an inactive one. Fansly search didn't bring up the creator despite me following them, so I assumed it was one of the "Unknown User"s. Resolved by unfollowing that creator.