search

ALonelyJuicebox / OFMetadataToStash

This script imports OnlyFans metadata into your Stash Database
BSD 2-Clause "Simplified" License
73 stars 10 forks source link

File Path Mapping is inconsistent / incorrecr #30

Open twist3dimages opened 10 months ago

twist3dimages commented 10 months ago

When I run the script I get a lot of these messages (running on Windows now)

Info: There's a file in this OnlyFans metadata database that we couldn't find in your Stash database but the file IS on your filesystem. Try running a Scan Task in Stash then re-running this script or changing your Search Specificity mode to Low.

Info: There's a file in this OnlyFans metadata database that we couldn't find in your Stash database but the file IS on your filesystem. Try running a Scan Task in Stash then re-running this script or changing your Search Specificity mode to Low.

image

I have confirrmed these files exist

ALonelyJuicebox commented 10 months ago

Oh that's interesting. Did you scrape OF on a *nix based operating system and then run this script while on Windows? Under the hood, the metadata database stores the directory and the filename separately, and this script needs the full path to the media. To handle that, this script makes an educated guess as to what directory delineator is appropriate (/ or ) based on the operating system that you're currently running the script on, which partially explains what you're seeing.

It's strange though that there seem to be folders inside your filename or something...

We can double check my theory though-- are you willing to help troubleshoot a bit? (otherwise I'll need access to that metadata database of yours)

twist3dimages commented 10 months ago

Oh that's interesting. Did you scrape OF on a *nix based operating system and then run this script while on Windows? Under the hood, the metadata database stores the directory and the filename separately, and this script needs the full path to the media. To handle that, this script makes an educated guess as to what directory delineator is appropriate (/ or ) based on the operating system that you're currently running the script on, which partially explains what you're seeing.

It's strange though that there seem to be folders inside your filename or something...

We can double check my theory though-- are you willing to help troubleshoot a bit? (otherwise I'll need access to that metadata database of yours)

  • Download DB Browser for SQLite https://sqlitebrowser.org/
  • Click on the Browse Data tab
  • Under the dropdown for Table, click on Medias
  • Under the column for filename there's an empty row that will say "Filter" in it. Enter "0hakgfjsx4unbkgs2v2dg_source.mp4"
  • The program should now just show you a single row of results.
  • Send over the directory and the filename from that row. (you can obfuscate as much as you need)

Haha yes, downloaded on Linux and ran on Windows until DigitalCriminal stopped updating it, so using OF-DL on my windows machine. Here's the screenshot you requested. I am not sure why the format is like that

image

Notice the \ and / where I marked over

I am not sure if this is already implemented, but i had to end up creating a bunch of directories like

user_data/sites5 user_data/sites

etc. Maybe a feature request to see if the file exists and match off the filename, for example i may have moved files from user_data/sites to user_data/sites5

ALonelyJuicebox commented 10 months ago

Yeah I can't quite explain why your filesystem looks the way that it does unless the OF scraper just...doesn't handle linux filepaths correctly. There's no reason I can think of that the OF scraper would put the wrong folder delineation character after the username of a performer but then put normal delineation characters for everything after that. Very strange indeed. 👀 Anyway, that's an oddity outside of this script, and it shouldn't technically matter tbh.

Two more quick questions-- 1 - What did you set your Scan Specificity mode to? If you're unsure, just open the OFMetadataToStash_Config file in a text editor

2 - While you confirmed that the files exist on the filesystem, are you sure they actually exist in Stash?

The reason why I'm asking is that if you have your scan mode set to normal, it matches based on ensuring the performer name is somewhere in the path and that the filesize matches between what's in Stash and what's in the metadata database. Based on what you've shown me so far, I don't see any reason why the performer name wouldn't be in the path somewhere, and there shouldn't be an issue matching the filesize.

Alternatively you could set the scan mode to "low" and see how that goes.

twist3dimages commented 10 months ago

Yeah I can't quite explain why your filesystem looks the way that it does unless the OF scraper just...doesn't handle linux filepaths correctly. There's no reason I can think of that the OF scraper would put the wrong folder delineation character after the username of a performer but then put normal delineation characters for everything after that. Very strange indeed. 👀 Anyway, that's an oddity outside of this script, and it shouldn't technically matter tbh.

Two more quick questions-- 1 - What did you set your Scan Specificity mode to? If you're unsure, just open the OFMetadataToStash_Config file in a text editor

2 - While you confirmed that the files exist on the filesystem, are you sure they actually exist in Stash?

The reason why I'm asking is that if you have your scan mode set to normal, it matches based on ensuring the performer name is somewhere in the path and that the filesize matches between what's in Stash and what's in the metadata database. Based on what you've shown me so far, I don't see any reason why the performer name wouldn't be in the path somewhere, and there shouldn't be an issue matching the filesize.

Alternatively you could set the scan mode to "low" and see how that goes.

  1. It's set to low

  2. they do

I know it's fucked haha, is there any way you can add something to check the file system and if windows replace every / with \ in the directory and vice versa on Linux?

ALonelyJuicebox commented 10 months ago

This wouldn't address the problem though-- on "low" this script just asks the Stash db if there's a file in the Stash database that matches the filesize of the file in your OF DB. No filepaths are involved (outside of the error message doing its own file system check to try and help the user sort out what the deal is)

Let me do some digging and see if I can replicate this in some form while having this script set to low.

twist3dimages commented 10 months ago

The new update correctly identifies the images (even with the \ and / being wonky) but not videos anymore. I just scanned. Doing it on my linux server and windows

Info: There's a file in this OnlyFans metadata database that we couldn't find in your Stash database but the file IS on your filesystem. Try running a Scan Task in Stash then re-running this script or changing your Search Specificity mode to Low.

Info: There's a file in this OnlyFans metadata database that we couldn't find in your Stash database but the file IS on your filesystem. Try running a Scan Task in Stash then re-running this script or changing your Search Specificity mode to Low.

Info: There's a file in this OnlyFans metadata database that we couldn't find in your Stash database but the file IS on your filesystem. Try running a Scan Task in Stash then re-running this script or changing your Search Specificity mode to Low.

ALonelyJuicebox commented 10 months ago

Hmm...the plot thickens. Can you go back to normal specificity mode one more time?