Not finding all files.

[supernova4420]

I am using the latest version, and when starting a scrape where a folder is already created, only files up to the most recent download (that are already there) are detected. All later posts are not found. (Website reports 480 posts, scrape only finds 440) . This is the same for multiple accounts. However I am able to fully download from new accounts for some reason.

I have tried deleting users.db and account specific database files and makes no change.

Incremental downloads is false.

[melithine]

Paste or attach your whole config.json please.

[supernova4420]

{ "DownloadAvatarHeaderPhoto": true, "DownloadPaidPosts": true, "DownloadPosts": true, "DownloadArchived": true, "DownloadStreams": true, "DownloadStories": true, "DownloadHighlights": true, "DownloadMessages": true, "DownloadPaidMessages": true, "DownloadImages": true, "DownloadVideos": true, "DownloadAudios": true, "IncludeExpiredSubscriptions": false, "SkipAds": true, "DownloadPath": "", "PaidPostFileNameFormat": "", "PostFileNameFormat": "", "PaidMessageFileNameFormat": "", "MessageFileNameFormat": "", "RenameExistingFilesWhenCustomFormatIsSelected": false, "Timeout": null, "FolderPerPaidPost": false, "FolderPerPost": false, "FolderPerPaidMessage": false, "FolderPerMessage": false, "LimitDownloadRate": false, "DownloadLimitInMbPerSec": 1000, "DownloadOnlySpecificDates": false, "DownloadDateSelection": "after", "CustomDate": "", "ShowScrapeSize": false, "DownloadPostsIncrementally": false }

[melithine]

I don't see any issues with the config. However, it's common for there to be a discrepancy between the creator media count and the number of files downloaded by ofdl, as ofdl will skip previously downloaded content. So if the creator posts the same image/video twice (or more), ofdl will only download one copy, not 2+. Creators posting duplicates is a frequent occurrence, particularly on larger accounts.

[supernova4420]

I have checked manually and ofdl is not grabbing all posts. It doesn't see any past a previous scrape done a few weeks ago.

[melithine]

Any errors in the log file?

[supernova4420]

No log file in the /logs folder were generated

[melithine]

How are you running it? Windows, Linux, Docker, etc.?

[supernova4420]

Docker in a Ubuntu VM

[thatcoderguy]

I know this would be a pain... but if you could find a piece of content on the site which you know hasnt been downloaded, I can look into that.

[melithine]

In docker, the logs will be under the config directory, or was that where you were already looking?

[supernova4420]

Yeah that is where I was looking. I bound the directory using docker-compose.

[melithine]

Not really sure how to help if there are no errors and it's just silently ignoring things. Maybe paste your compose file?

[supernova4420]

`version: '3'

services: of-dl: container_name: of-dl image: ghcr.io/sim0n00ps/of-dl:latest volumes:

• /media/Encrypted:/data
• /home/sz/OF-DL/config:/config stdin_open: true tty: true`

[melithine]

I'm officially stumped, then. @sim0n00ps do you have any thoughts?

[sim0n00ps]

I'm officially stumped, then. @sim0n00ps do you have any thoughts?

If the OP is willing to provide an account that is preferably a free subscription that it is occurring on then I can try to replicate?

[supernova4420]

I have done a simple test that has resulted in the same failure:

• I have installed a fresh OF.DL.V1.7.60.zip file
• I have copied over my auth.json and this config.json

{ "DownloadAvatarHeaderPhoto": true, "DownloadPaidPosts": true, "DownloadPosts": true, "DownloadArchived": true, "DownloadStreams": true, "DownloadStories": true, "DownloadHighlights": true, "DownloadMessages": true, "DownloadPaidMessages": true, "DownloadImages": true, "DownloadVideos": false, "DownloadAudios": true, "IncludeExpiredSubscriptions": false, "SkipAds": false, "DownloadPath": "", "PaidPostFileNameFormat": "", "PostFileNameFormat": "", "PaidMessageFileNameFormat": "", "MessageFileNameFormat": "", "RenameExistingFilesWhenCustomFormatIsSelected": false, "Timeout": null, "FolderPerPaidPost": false, "FolderPerPost": false, "FolderPerPaidMessage": false, "FolderPerMessage": false, "LimitDownloadRate": false, "DownloadLimitInMbPerSec": 1000, "DownloadOnlySpecificDates": true, "DownloadDateSelection": "after", "CustomDate": "2024-04-20", "ShowScrapeSize": true }

• I have chosen Custom>onlyfans (account: @onlyfans)
• This results in zero anything getting detected - nothing downloads
• no log files generated

[melithine]

You have videos disabled in that config.

[supernova4420]

Apologies, I didn't check if they had any photos. I have repeated the test using the account @oftv, as they have photos. I have adjusted the date as seen in the config.json file:

{ "DownloadAvatarHeaderPhoto": true, "DownloadPaidPosts": true, "DownloadPosts": true, "DownloadArchived": true, "DownloadStreams": true, "DownloadStories": true, "DownloadHighlights": true, "DownloadMessages": true, "DownloadPaidMessages": true, "DownloadImages": true, "DownloadVideos": false, "DownloadAudios": true, "IncludeExpiredSubscriptions": false, "SkipAds": false, "DownloadPath": "", "PaidPostFileNameFormat": "", "PostFileNameFormat": "", "PaidMessageFileNameFormat": "", "MessageFileNameFormat": "", "RenameExistingFilesWhenCustomFormatIsSelected": false, "Timeout": null, "FolderPerPaidPost": false, "FolderPerPost": false, "FolderPerPaidMessage": false, "FolderPerMessage": false, "LimitDownloadRate": false, "DownloadLimitInMbPerSec": 1000, "DownloadOnlySpecificDates": true, "DownloadDateSelection": "after", "CustomDate": "2022-07-22", "ShowScrapeSize": true }

Still no photos downloaded

[btiplitz]

I have perhaps a similar issues in that I am not downloading messages I pad for in the past week. I did get errors during the download. Without deleting the metadata file I wonder if the db can be adjusted. There are null entries in the file. What happens if I delete the null entries (and do you expect them? I did get some errors and the null entries might be about the same number of errors I got).

[melithine]

I have perhaps a similar issues in that I am not downloading messages I pad for in the past week. I did get errors during the download. Without deleting the metadata file I wonder if the db can be adjusted. There are null entries in the file. What happens if I delete the null entries (and do you expect them? I did get some errors and the null entries might be about the same number of errors I got).

I would try scraping the purchased tab, as that works a bit differently.

I don't know how safe it is to edit those DBs. I wouldn't revise them to do your bidding...

[btiplitz]

I have perhaps a similar issues in that I am not downloading messages I pad for in the past week. I did get errors during the download. Without deleting the metadata file I wonder if the db can be adjusted. There are null entries in the file. What happens if I delete the null entries (and do you expect them? I did get some errors and the null entries might be about the same number of errors I got).

Now I think I might be getting all my posts. I noticed the date of the download is actually not the date it was posted. Perhaps it's the data of the 1st time the creator pushed the material. Not sure. I did see a post not get downloaded that was new, but then it pulled the message the next day