[Bug]: Downloading all Fantia Posts from one creator results in 403

Aaeeschylus commented 1 year ago

Which program are you seeing this problem on?

Cultured Downloader (exe), Cultured Downloader (py)

Program Version

4.2.2

Steps to reproduce the bug

Go to "Download all Fantia posts from creator(s)" section
Put in link for creator and select multiple pages (ie. 1-12)
Enter Y for Download Images, thumbnail and attachments
Start downloading

Expected behaviour

All images/attachments/thumbnails from all pages entered should be downloaded.

Relevant log output (If any)

----------------------------------------------------------------------------------------------------
2023-03-12 10:05:46,513 [Cultured Downloader V4.2.2] [ERROR]: 
Uncaught TypeError
----------------------------------------------------------------------------------------------------
Traceback (most recent call last):
  File "utils\download.py", line 95, in async_download_file
  File "httpx\_models.py", line 749, in raise_for_status
httpx.HTTPStatusError: Client error '403 Forbidden' for url 'https://cc.fantia.jp/uploads/post_content_photo/file/8723128/9c63c89f-9c09-41de-b86f-c02ca249c65c.jpg?Key-Pair-Id=APKAIOCKYZS7WKBB6G7A&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jYy5mYW50aWEuanAvdXBsb2Fkcy9wb3N0X2NvbnRlbnRfcGhvdG8vZmlsZS84NzIzMTI4LzljNjNjODlmLTljMDktNDFkZS1iODZmLWMwMmNhMjQ5YzY1Yy5qcGciLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE2Nzg1NzUyNDV9fX1dfQ__&Signature=dxXDCPVukZMbq3YerrxpGYmreBPu1lmSlEDgtyqIg8NI0JOtKdApmiABLvtqVDFL54TKsMmNoe8k1th-cMvlXV0RHn9mdDwUSSrQXw4fnUdHb243XUuJwBNKrNVab4WQFFKO0L7PT7mu0lEKC9U6sUljO63jcUiY3pCvkuQV2i1FbgMvp5ZKHPoQqgCpBn7JQ5kUYsZ1FfE~Qx0QYMfE0vDLZvkIFFn~toykcExGykrIhqixnroNjPGNlaTcNNCrALpyQWSUttgcUPw7EN1V802JQOSmpb1kO2UJ7japjbBN-7h8v-3hbLCccN8tQqY10IXpDVK8kA6U1Kh9oc~sgw__'
For more information check: https://httpstatuses.com/403

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "cultured_downloader.py", line 540, in <module>
  File "cultured_downloader.py", line 468, in main
  File "cultured_downloader.py", line 463, in initialise
  File "cultured_downloader.py", line 215, in main_program
  File "cultured_downloader.py", line 195, in nested_menu
  File "cultured_downloader.py", line 164, in download_process
  File "asyncio\runners.py", line 190, in run
  File "asyncio\runners.py", line 118, in run
  File "asyncio\base_events.py", line 653, in run_until_complete
  File "utils\web_driver.py", line 687, in execute_download_process
  File "utils\functional.py", line 1011, in check_download_tasks
  File "utils\functional.py", line 983, in call_async_result
  File "utils\download.py", line 107, in async_download_file
  File "concurrent\futures\thread.py", line 58, in run
  File "<frozen genericpath>", line 19, in exists
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

It downloads the first page or so correctly, but once it hits the 403, it stops downloading anything else and closes.

Screenshots (Optional)

No response

KJHJason commented 1 year ago

how long did it take for it to download the first page?

Aaeeschylus commented 1 year ago

how long did it take for it to download the first page?

Not very long at all. about 5-10 minutes? I didn't time it.

KJHJason commented 1 year ago

Thanks for raising this issue. This is an oversight from my end as it looks like Fantia has a short expiry duration of their signed AWS S3 URLs.

I can reduce the chance of the signed URL from expiring but not eliminating the problem entirely. However, the fix will be in the upcoming v5.0.0 of this Python program where it uses the upcoming v1.1.1 of the CLI tool for its download logic. Please wait till then!

KJHJason commented 5 months ago

Fixed in https://github.com/KJHJason/Cultured-Downloader/releases/tag/5.0.0

KJHJason / Cultured-Downloader