search

prof79 / fansly-downloader-ng

Easy-to-use fansly.com content downloading tool. Written in Python and available as a standalone Windows Executable. Enjoy your Fansly content offline anytime, anywhere in the highest possible content resolution! Fully customizable to download in bulk or single: photos, videos & audio from timeline, messages, collection & single posts.
GNU General Public License v3.0
267 stars 15 forks source link

Timeline cursor times out #4

Closed spiderzero3 closed 8 months ago

spiderzero3 commented 9 months ago

It seems like the script times out after a certain point. I'm subscribed to two users and I'm missing quite a lot from their profiles after several attempts. Nothing gets downloaded any more. I get the same message for both users with one missing 224 items and the other missing 711.

[ Info ] [2023-12-21 | 15:04]: Deduplication [Media ID]: image '2021-08-26_at_13-08_id_287369269656690688.png' → skipped
[ Info ] [2023-12-21 | 15:04]: Inspecting Timeline cursor: 287369558304497664 [CID: 285372811931824128]
[ Info ] [2023-12-21 | 15:04]: Slowing down for 60 s ...
[ Info ] [2023-12-21 | 15:05]: Inspecting Timeline cursor: 287369558304497664 [CID: 285372811931824128]
[ Info ] [2023-12-21 | 15:05]: 
╔═
  Finished Normal type download for @xxxxxxx!
  Total timeline media: 529 pictures & 183 videos (= 712 items)
  Total message media: 0
  Total media (timeline & messages): 712
  Downloaded media: 0 pictures & 0 videos (= 0 items)
  Duplicates skipped: 488
  Missing items: 224 (this may indicate a problem)
                                                                          ═╝

EDIT: Just ran the same profile through the Avnsx's downloader and got the same result. No errors or anything, but 0 downloads and 488 duplicates. Wouldn't have known anything was missing if your script didn't list missing items, so great feature!

Animalm4st3r commented 9 months ago

Could be the same issue that other are having, were it only downloads the first 5 items in posts that have more then 5 items in it

prof79 commented 9 months ago

Hi! I'm glad you like the summary feature!

This is difficult to diagnose from where I stand. What I see in the log you posted is like normal behavior: At some time in the collection scraping process, an empty timeline cursor will be returned to indicate the end of all posts. However, there was some drama where Fansly introduced strong rate-limiting measures to battle such scraper programs. So the scraper may have gotten an empty timeline response even though it wasn't the end. That's why I introduced that 60 sec delay so if you actually were rate-limited by Fansly, after that period a proper response would be guaranteed. If still empty you're actually at the end of the creator's content. So this is how all normal downloads end, too. I have no knowledge how to technically tell a rate-limited response from the last/empty timeline.

You could for example try to bump up the tries and delay using arguments like these an the command-line:

-tr 3 -td 120

This will slow things down considerably but will tell with almost 100 % certainty whether 287369558304497664 is indeed the last cursor or not. Then the issues lies elsewhere.

I presume you're following and have non-expired subscription to the creator?

You may try v0.7.6 but I fear it won't do you any good since you do not get an error/exception.

I'll take a look maybe there is also something wrong with the total items/scrapable/non-scrapable calculations.

spiderzero3 commented 9 months ago

Ran those args into v0.7.6 and yeah, it stops at the exact same spot (287369558304497664). I'm still subscribed and all of the timeline media should be available to me at the tier I have.

I also double checked the summary and found a discrepancy. The summary states there are 223 missing items now, but when I check the number of files I actually have downloaded, I'm missing 234 files. According to the model's profile, there are 531 photos & 187 videos available where I have 303 photos & 181 videos downloaded. The 531/187 timeline media is also accurately counted in the summary, so I don't know where the difference is coming from.

prof79 commented 9 months ago

Hi @spiderzero3! Can you check what we discussed under #3 and test if the pre-release fixes your issues? Thanks!

https://github.com/prof79/fansly-downloader-ng/releases/tag/ondemand

prof79 commented 9 months ago

You may now take this one:

https://github.com/prof79/fansly-downloader-ng/releases/tag/v0.7.10

spiderzero3 commented 8 months ago

It's definitely an improvement! As of right now, I have 515/184 photos & videos downloaded out of 531/188 total (which is accurate to Fansly's count) with no missing items reported.

prof79 commented 8 months ago

If there are no missing items reported I don't know what more I could do. The creator may have some items that are either in a higher-tier subscription or pay-per-view regardless of subscription (blurry preview in the timeline). So the downloader seems to get all which is "scrapable". Also note that the amount of items on disk may diverge because of duplicate items/re-posts on-line and items a creator may have deleted meanwhile.

prof79 commented 8 months ago

Duplicate of #3