Bug: downloaded video is 720p while higher res is available

[RalkeyOfficial]

The Fansly scrapper downloaded some videos at 720p resolution, While 1080p and 2160p were available (and not behind a paywall).

It also does this for multiple content pieces from other content creators.

I noticed this happens with newer content content posted in the last 2 weeks

[Avnsx]

What download mode were you using? (Normal, Single or Collections)

or are you on the older version v0.3.5 ?

[RalkeyOfficial]

Normal and I'm using 0.3.6

[Avnsx]

Hey so I've looked into this and figured out that this is caused by the video file being a .m3u8 type media & the fansly API serving these a little bit different then usual.

The post ID you linked is also the only one I could find, that uses this new .m3u8 format.

I'm in the process of adapting fansly scraper to support this new format and simultaneously stay compatible with the old format & structure, for media that was posted before may 2023.

Could you please link me more similar post IDs, that also utilise this new m3u8 format, but variate in types?

More specifically I mean for example m3u8 format, previews which subdivide into pictures or videos. And a couple bundles that contain m3u8 format files.

Also do you think that there's anything wrong with just leaving the m3u8 files as such and not trying to convert them to mp4 files (after being downloaded)? As such conversion would be easiest, with the usage of ffmpeg. Unfortunately there's also a thousand reasons to not want to introduce ffmpeg into this project.

[RalkeyOfficial]

Also do you think that there's anything wrong with just leaving the m3u8 files as such and not trying to convert them to mp4 files (after being downloaded)?

You need to convert .m3u8 into .mp4, because it downloads the actual video file in chunks ending with .ts

for example: downloads 4 files:

1. 512758848092643328_1080.m3u8

this file contains the following content:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:2
#EXT-X-MEDIA-SEQUENCE:1
#EXT-X-PLAYLIST-TYPE:VOD
#EXTINF:2,
512758848092643328_1080_00001.ts   <-- important part
#EXTINF:2,
512758848092643328_1080_00002.ts   <-- important part
#EXTINF:1,
512758848092643328_1080_00003.ts   <-- important part
#EXT-X-ENDLIST

This tells whatever software they're using how to run the video. Then it downloads the actual video files in chunks (audio included).

2. 512758848092643328_1080_00001.ts
3. 512758848092643328_1080_00002.ts
4. 512758848092643328_1080_00003.ts

these chunks are very logical, as in if you append them counting up you will get the full video.

Which is what is being displayed to the user.

Keep in mind that this is only for this post ID, depending on the length of the video, the amount of chunks can increase or decrease.

And for obvious reasons it would be bad not to merge them into a .mp4

[RalkeyOfficial]

@Avnsx I don't know if you still need this, but this post post/513826839995625472 has two videos embedded

[Avnsx]

@Avnsx I don't know if you still need this

Nah I've already locally fixed fansly scraper about 90% to be compatible with .m3u8 and that without directly introducing the requirement to install ffmpeg to the users system, which is huge.

The reason it's taking me a bit longer is, because after I've adjusted the most newer functions to work with .m3u8's I decided to do some re-factoring and adjust the old code to the new code I had pushed with that one latest commit https://github.com/Avnsx/fansly/commit/b30a65df220f13b95cba5fa4e5b032baa82a1d75.

So in the next commit you will see that everything will actually be tunneled through those new functions and this also allows me to way easier maintain this code in the future, because everything is handled in their own sections now, but still utilising those couple main functions.

Which means you can also do download_mode = Timeline or Messages separately now and it'll separately download them then. Or you can enter "Normal" and it'll work combined.

I hope this stupidly long comment has been helpful

Regarding this the 2nd part where you linked me posts that utilised the .m3u8 format was actually very usefull. But the first part of the message, where you explained me how .m3u8's work was kinda funny to me, because I never intentionally wanted that explanation, I just failed to properly express myself in my initial message, which led to you writing that.

What I actually need right now is to find a creator with an unholy amount of videos and posts, because I just noticed that fansly has removed the maximum integer for limit https://github.com/Avnsx/fansly/blob/b30a65df220f13b95cba5fa4e5b032baa82a1d75/fansly_scraper.py#LL642C7-L642C7

Which means fansly scraper doesn't need to iterate in steps of 50, like it was before. Instead I am now able to do limit: 9999 and will actually get a response with as many posts as the creator has uploaded.

I did set it to 9999 right now to test in my local version, but I never found a creator that even has remotely close to 9999 posts so I wonder how far that even goes up now. Because maybe there's a limit at idk 500, 900 or anywhere below 9999 and I wouldn't notice. So it would be cool if you found someone with thousands of posts and let me know their username so I can test it properly.

Also I am planning to remove hashing & naming_convention (filenames will now always just be Date_posted and media_id) in the next commit and replace hashing with just reading media_id from downloaded filenames. I also thought about adding the media_id as metadata to each media; but then there's the problem that this current commit allows for any type of file extensions and every file extension / type requires its own way of adding metadata, like pythons PIL libary has to handle each file format differently. So to make it compatible like that with metadata would be too much work. I could ofc also save all images in .png and all videos in .mp4, but do we really want that trade off?

Then again I'm not sure if reading media_id out of filenames is smart either, because I'm sure there's people that re-name files. Which means as soon as they re-name a single file in a download directory created by fansly scraper, that file would be re-downloaded again in the next scraping process, while the old file still exists with the modified name. What do you think would be the smartest logic here?

[RalkeyOfficial]

I personally think that adding the ID to the metadata is the best option, but if that is too difficult, I would opt to keep the hashing system. While that may be resource intensive and slower, the user would at least not re-download content if they choose to change the name of the downloaded file. Also, it would allow me to just restart the scraping process to get the 1080p versions instead of the 720p which i currently have. ALSO, hashing can be used to avoid duplicates. And I personally don't see the issue with saving all content as .png and .mp4, but that's your call to make.

[RalkeyOfficial]

I am also really curious as to how you handled FFmpeg

You said “without directly introducing the requirement to install ffmpeg to the users system”, did you put the FFmpeg files in the Project DIR and refer to that in your code? Or have you found another method that kind of acts like FFmpeg?

[Avnsx]

@RalkeyOfficial Could you help me again?

Can you find & link someone that has the same .m3u8 video file uploaded multiple times?

[RalkeyOfficial]

Sorry, but I can't help you with this one. None of the creators I follow re-upload the same video multiple times.

You could try to make your own account and do it yourself.

[Avnsx]

None of the creators I follow re-upload the same video multiple times.

How about duplicate pictures?

Additionally, I will be releasing a new commit that entails a substantial rewrite of approximately 70% of Fansly scraper. Only a few remaining tasks need to be addressed.

Please note that I may choose to delete this issue ticket completely in order to prevent any unwanted attention from DMCA agents or Google indexing. I do not want this issue ticket, along with the entire repository, to be associated with the usernames of Fansly creators. It is important to avoid any unnecessary exposure or attention.

[RalkeyOfficial]

Same story, no duplicate images. The creators on this platform tend to be very organized (unlike twitter)

I do not want this issue ticket, along with the entire repository, to be associated with the usernames of Fansly creators. It is important to avoid any unnecessary exposure or attention.

I don't know if it will help, but I'll be editing my previous comments to remove the platform name and creator names Edit: seems like you already did most of the work for me

[Avnsx]

Hello, @RalkeyOfficial! I wanted to inform you that I have just published the new commit https://github.com/Avnsx/Fansly-Downloader-App/commit/8c591cb2d1d1ee35a1243f788d1cb83c99e65209. This release corresponds to version 0.4, which was previously named 0.3.6.

I would greatly appreciate it if you could test this new version and provide feedback. Please let me know your thoughts on it and inform me if you encounter any bugs or issues. Thank you!

[Amoa123]

I happened to be checking to see if there were any updates right as you released the newest version. I can confirm that it is now downloading both 1080p and 2160p versions of videos that it was only downloading 720p versions of the same videos just 2 days ago. Files are being converted to .mp4 after the download is finished.

[Loxasluscious]

New to github so sorry if this is a silly question, but how do I update to version 0.4? I'm using the windows desktop launcher. I used the updater located among the files but that only brought it up to 3.5

again, sorry for an amateur question

[RalkeyOfficial]

New to github so sorry if this is a silly question, but how do I update to version 0.4? I'm using the windows desktop launcher. I used the updater located among the files but that only brought it up to 3.5

Version 0.4 has not been compiled and published yet. As of now, you can get it by downloading it as zip.

you do need to have Python installed to use it like this

[Avnsx]

Closing, as the 0.4 version is finally released and does target & solve the topics mentioned within this issue.

If you're on the previous 0.3.5 version you can comfortably update using the updater.exe supplied in the version, or just manually download the 0.4 version from the repositories Releases.