Not downloading all content

[deadbeatdandylyon]

I ran the Fansly Downloader .exe for a creator and it did successfully go through. however I noticed as I was going through comparing what was downloaded, not all images were downloaded that is on her current page and also available in the tier I am subscribed to.

So I decided to run the script again and it is still missing a section of photos. I also ran into the following error message which I don't know if this has something to do with it or not.

[Avnsx]

Creator name?

[deadbeatdandylyon]

According to the creators bio there should be - 603 Photos and 663 Videos

But in the folder that downloaded for Timeline - 572 photos were downloaded and 610 videos were downloaded

I also tried running Fansly Downloader.py to see if that provides any other information or if it was able to download the missing content but I run into the same issue as well

[Avnsx]

The error on your screenshot is such a easy fix it's literally just adding Image.MAX_IMAGE_PIXELS = None, which I would've had to put into the python source, if I knew about this limit PIL safety limit upfront ..

You can press enter to continue after it errors like that, is it continuing to download other content afterwards?

If it continues to download afterwards, at most you might be missing like around 30 media items within the timeline page that errored out.

If not too bad, I'll add the support for that in the next version 0.4.1 which I'll release in about half a year or 700 stars on this repository.

[deadbeatdandylyon]

Yes I am able to press enter and it will continue to download the rest of the content.

[Avnsx]

Yes I am able to press enter and it will continue to download the rest of the content.

If you care so much about that couple content missing, that is attached within that bugged out timeline page, you can work your way around the bug by setting download_mode to Single and downloading the missing content manually, obviously except the insanely high resolution photo that is the root of this Error

[deadbeatdandylyon]

Well although I am a bit worried about the photos, i'm more concerned about the videos as to also be missing about 50 videos that's a bit on the wild side. I'll just look through one by one until the bug gets fixed, so thank you for your info on this

[ForcedToRock]

I'm having a similar issue of the downloader not downloading everything. I didn't get any error message though. I didn't note the exact number of files for comparison, but as an estimate with 1 model, but old videos folder had over 200 files, while the new videos folder had 180-something, and I know I have Media Preview downloads enabled

[Avnsx]

Even if you were subscribed, you're not going to be able to download 100% of all the content available on every creator. As some of them just restrict specific content to literally be purchaseable only etc.

I do think it's very unlikely that content is missing in Timeline. As additionally before releasing I had someone test the old 0.3.5 version and the new 0.4 version on 27 different creators and it would download about the same amount of content.

Otherwise if you think it would be missing content, find the specific missing content and give me the exact post ids where that missing media is sourced from.

[deadbeatdandylyon]

Well running the older version of Fansly Scraper I had no issues with content being download. I do agree with you that some content may be PPV only and may require a Purchase however for the content I am referencing too I am on an NO PPV tier meaning that everything is unlocked. so being purchaseable only is not the issue with this particular case

Well as I did state I did check the photos on the creators page it does tell how many photos are there and after the download was completed I checked the photos in comparison to what the site had and there was about 30 missing, where as the older Fansly scraper 0.3.5 did download those particular photos that were missing, so something changed in an update from 0.3.5 to the new version of 0.4

As far as the videos go on this creator account at the top it states 663 Videos, What downloaded to the computer after the scrape is 611

as far as the Post Id's I don't know how to find that information, but if that will help, then i'll try to find it

[Avnsx]

This whole conversation is pointless.

To help me verify that this is a genuine bug:

1. Visit the creator page.
2. Identify posts that appear to have download issues.
3. If you suspect a post is not being downloaded properly, change the download_mode to Single in the configuration file and attempt to download the content from that post. If the message states "1. duplicate declined," it means you have already downloaded the post previously. If it successfully downloads the content, it indicates that the downloaders functionality is working correctly. The only situation where a bug would be confirmed is if the message states "no scrapable media found" for the post you attempted to download from (and you can visibly see, that there's media attached to that post).
4. Inform me of the post IDs (click on a post, to see its post id in the url bar) for such posts. If these posts are not restricted to subscribers only, I can verify that there is indeed a bug. If the posts are accessible only to subscribers, I will require access to an account with the affected content in order to confirm that it is not being downloaded and for debugging purposes, so that I can ensure compatibility with the Fansly downloader. Send me a e-mail with account credentials to AvnDev@protonmail.com

[deadbeatdandylyon]

Okay so I changed the mode to Single as you suggested

I copied the Post ID "328258312162258944"

The result is below

I also tried on another Post ID: "329933941077192704" and same result

I also as a test I tried using the python script and in a completely different directory for download and same results.

I will send an email

[ForcedToRock]

So I just freshly scraped a profile, and for me, it might be based on the fact the post is accessible for everyone and not subscribers only. I downloaded a post of a free-for-all video that the Normal mode option missed.

Post ID in question: 295387599030198272

[Avnsx]

Post ID in question: 295387599030198272

Can't replicate the issue on that post ID, for me it downloads just fine.

Verify that the authentication token you entered into the configuration file, actually has access to that post.

[ForcedToRock]

It downloads when I specify the post ID and set it to Single mode, what I'm trying to say is that in Normal mode, that post was skipped. It makes me think Normal mode doesn't account for free-for-everyone posts, only subscriber-only posts.

[deadbeatdandylyon]

Thanks for the verification, how does one verify if the authentication token has access to the post, All I know is that I went to developer tools, / network / refeshed the page / type in apiv at the top and copied the "authorization" token under "account?username but I don't know how to verify if the authentation token has access to it?

[Avnsx]

how does one verify if the authentication token has access to the post

Set the Authorization_token variable to ReplaceMe in the config.ini file save the changes and re-launch version 0.4. That should put it into a tutorial kind of mode, which will make it interactively help you find the authentication token corresponding to the account you want to connect it to.

[deadbeatdandylyon]

Awesome did as you suggested, I changed everything to default to "ReplaceMe" for the .exe version, I re-ran it it walked me through as you stated. for some reason it was registering firefox browser to get the configuration from so switched it to opera gx which it walked me through. after doing so I changed the config.ini download mode to "single" and still couldn't download the file.

[Avnsx]

I copied the Post ID "328258312162258944"

I can confirm the authenticity of this bug and will provide a fix soon.

[deadbeatdandylyon]

Well I am happy that at the end of the day it did not come to User Error as sometimes I know that could be the issue. So I really do appreciate you taking the time to look into this for me and identify that it is an issue. So thank you very much for the help

[Avnsx]

@ForcedToRock @deadbeatdandylyon The latest commit https://github.com/Avnsx/fansly-downloader/commit/115d549204268db8a5bf6d7bd5b425c8add1d71c should've fixed all issues mentioned within this issue ticket, please verify that it downloads the missing media in the highest possible resolution.

You can run the latest commit by installing the python version of fansly downloader: https://github.com/Avnsx/fansly-downloader#python-version-requirements

[ForcedToRock]

I can't start the python version, it throws up an error that the config.ini file can't be read or found, even though it's in the same directory as the downloader.

[Avnsx]

@ForcedToRock If you're using a IDE, don't open it with the "open Folder" button as that will break the pathing. Open only fansly_downloader.py by double clicking on it, which should open it in the python launcher, which comes with any python version from python.org

[ForcedToRock]

I tried the latest commit and noticed no change, it still doesn't download everything, seemingly skipping posts that aren't locked to subscriber-only.

[Avnsx]

@ForcedToRock

seemingly skipping posts that aren't locked to subscriber-only

Link me exact post ids, of posts which are being skipped.

Post ID in question: 295387599030198272

Also the last post id which you linked is not true, that video from Ah,ptik is being downloaded regardless with the media id 295387293710032896 in Timeline > Videos. And yes I downloaded it with the setting download_mode on Normal (which is Timeline + Messages), but what I said before about verifying that posts can not be downloaded with download_mode on Single is also 100% correct information, because timeline, messages, single, collections all use the exact same function, to download stuff, it's all tunneled through the same thing in the codebase. So it's not really possible for Single mode to download less or more content then any other mode would be able to download.

[ForcedToRock]

I can guarantee that the post ID I linked never got downloaded from the app in Normal Mode, I needed to download it via Single mode to get it.

[deadbeatdandylyon]

please verify that it downloads the missing media in the highest possible resolution.

The most recent Python version includes updates that ensure successful video downloads from Timeline for me, although obviously the executable version has not implemented these changes yet.

(shortened for clarity)

[Avnsx]

so I think at this point the python version is working correctly but not the .exe

That might've not been obvious to you @deadbeatdandylyon, but there's no way that the .exe works properly already. I only updated the python source code. The way updates are published, is that every change gets tested on the python source first and once I am about 90% sure what I changed there is good, I compile it and publish it as a .exe file (which I obviously didn't do yet).

So yeah, as I guessed my latest commits fixed the python source and it downloads the videos properly for you now, which is good.

I just have no clue how @ForcedToRock is still having issues, with media from Timeline.

[deadbeatdandylyon]

I didn't realize that how things work is first it's the python and then it's .exe i just wanted to provide the full update just to be safe but next time I will no for sure.

As a helping test for you @Avnsx I went and ran a scrape using the fansly_downloader.py on previously mentioned creator that @ForcedToRock is having an issue with and I set my download mode to "normal" and it was able to download the video without issue.

I also switched it to single mode and downloaded the same post into another directory and it downloaded also just fine for me.

[ForcedToRock]

Thanks, but was the duplicate threshold option enabled though?

[Avnsx]

@ForcedToRock Please read more about utilize_duplicate_threshold here: https://github.com/Avnsx/fansly-downloader/wiki/Explanation-of-provided-programs-&-their-functionality#explanation-of-configini

[deadbeatdandylyon]

Yes I did have utilise_duplicate_threshold set to True

[ForcedToRock]

It's true for me as well, I was just trying to figure out if I did anything wrong with the config.

show_downloads = True download_media_previews = True open_folder_when_finished = True download_mode = Normal download_directory = Local_directory separate_messages = True separate_previews = False separate_timeline = True utilise_duplicate_threshold = True

[Avnsx]

@ForcedToRock Just try setting the threshold option to False and see if that helps you, it's not like it matters at this point.

[deadbeatdandylyon]

I don't know if this is normal but with the utilise_duplicate_treshold set to False should I be seeing this in the scrape process

I'm just trying to help if I can and if I set mine to False should I be seeing this happen?

[Avnsx]

@deadbeatdandylyon Yes you should see that happen, because now the downloader iterates to the absolute end again, regardless if you already have above 20% of the content downloaded and stored somewhere already or not.

So now it iterates to the end > recognizes it already downloaded the content before > declines it, because we don't want it twice.

[ForcedToRock]

Crazy that I'm downloading over 40GB of content over and over to try and figure out a problem. Right now I'm trying to download with the duplicate threshold thing set to "disabled." Until that, I read this comment and thought it might not be an issue for others, but issue for me, who doesn't have the 24-hour format on my PC.

https://github.com/Avnsx/fansly-downloader/issues/104#issuecomment-1595178750

Edit: Ok, disabling the duplicate threshold for some reason worked, everything from the profile where that post ID shared was downloaded. Seemingly everything. Is there a program or something where I can compare directories for like duplicates? I want to compare the rip from v0.4 to 0.3.5 so I can be sure it's safe to delete the 0.3.5 rip.

[Avnsx]

@ForcedToRock Btw since you're not using the 24 hour format on your PC, are the filenames actually working correctly? Like it should say something with AM or PM in there

[ForcedToRock]

No AM or PM, the filenames indicate that the 24-hour format's used, just not my clock. For example. 2023-05-14_at_20-10 - That's the file name of a video with no AM or PM.

[Avnsx]

with no AM or PM.

Yeah well I hoped that I could detect if the system is using am or pm and adjust timestamps given by fansly downloader accordingly, but I guess it's bugged out. Whatever then, don't care too much to fix it atm https://github.com/Avnsx/fansly-downloader/blob/2e85993d9ae02c09f1d66f6388b801f296f6b1e0/fansly_downloader.py#L418-L419

[Avnsx]

The Executable file that finally fixes this bug, is now also officially released, if you're on the previous 0.4 executable version, start it and it'll automatically update itself.

Else you can download it from releases.