search

prof79 / fansly-downloader-ng

Easy-to-use fansly.com content downloading tool. Written in Python and available as a standalone Windows Executable. Enjoy your Fansly content offline anytime, anywhere in the highest possible content resolution! Fully customizable to download in bulk or single: photos, videos & audio from timeline, messages, collection & single posts.
GNU General Public License v3.0
292 stars 16 forks source link

Fix API size limit #5

Closed icewinterberry12 closed 10 months ago

icewinterberry12 commented 10 months ago

When https://apiv3.fansly.com/api/v1/account/media/orders/ API return a long list of accountMediaId ids, the second API call to https://apiv3.fansly.com/api/v1/account/media?ids={accountMediaIds} fails because it receive an HTTP status code 413 - Request Entity Too Large and the scraper exit with the following traceback:

 ERROR | 19:16 || An unexpected error occurred: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/app/./fansly_downloader_ng.py", line 169, in <module>
    exit_code = main(config)
                ^^^^^^^^^^^^
  File "/usr/src/app/./fansly_downloader_ng.py", line 133, in main
    download_collections(config, state)
  File "/usr/src/app/download/collections.py", line 42, in download_collections
    post_object = post_object.json()
                  ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Exiting in 15 seconds ...

This PR introduce batched requests

prof79 commented 10 months ago

@icewinterberry12 amazing, thanks a lot for your troubleshooting and efforts! I'll merge and do the rest like version numbering and building later. Hopefully, this also fixes issues other people are having.