search

ThePornDatabase / namer

Renames adult video files so that the plex/jellyfin plugins and stash script will match without user input. Runs server or command mode.
https://theporndb.net
184 stars 29 forks source link

Arm64 vs aarch64 in pull request 123 #124

Closed gebau00a closed 1 year ago

gebau00a commented 1 year ago

Pull request 123 added an arch check for videophash.py On arm64, it expects the arch type arm64, but distributions know arm64 as aarch64. Running namer directly results in a Unsupport architecture error aarch64

Adding the dict link 'aarch64': 'arm64' runs, but videohashes-arm64-linux fails to produce output, thus an error is raised

Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 995, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
> File "/home/andy/.local/lib/python3.11/site-packages/namer/watchdog.py", line 137, in __processing_thread
    handle(command)
  File "/home/andy/.local/lib/python3.11/site-packages/namer/watchdog.py", line 54, in handle
    process_file(command)
  File "/home/andy/.local/lib/python3.11/site-packages/namer/namer.py", line 176, in process_file
    phash = VideoPerceptualHash().get_stash_phash(command.target_movie_file) if command.config.search_phash else None
  File "/home/andy/.local/lib/python3.11/site-packages/namer/videophash.py", line 91, in get_stash_phash
   return self._get_stash_phash(file, stat.st_size, stat.st_mtime)
  File "/home/andy/.local/lib/python3.11/site-packages/namer/videophash.py", line 96, in _get_stash_phash
    return self.__execute_stash_phash(file)
  File "/home/andy/.local/lib/python3.11/site-packages/namer/videophash.py", line 134, in __execute_stash_phash
    data = json.loads(stdout, object_hook=lambda d: SimpleNamespace(**d))
  File "/usr/lib/python3.11/json/__init__.py", line 359, in loads
    return cls(**kw).decode(s)
  File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
 File "/usr/lib/python3.11/json/decoder.py", line 355, in raw_decode
   raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Running videohashes-arm64-linux manually with a link to a video file generates the error ffmpeg/ffprobe executables not found

Scanning through the compiled code for videohashes-arm64-linux shows several matches for arm64, but none for aarch64. I assume it's looking for an arm64 ffmpeg, and cannot find an aarch64 ffmpeg.

Not sure if this is an upstream error from https://github.com/peolic/videohashes or where the videohashes executable comes from

4c0d3r commented 1 year ago

Video hashes come from that repo, that they run locally is great (you do need to install ffmpeg though for it function). Adding aarch64 to this map: https://github.com/ThePornDatabase/namer/blob/main/namer/videophash.py#L38-L42. Adding it now.

gebau00a commented 1 year ago

I have ffmpeg installed, namer was working flawlessly before the videophash function was updated. Nor sure why it cannot find ffmpeg.

4c0d3r commented 1 year ago

try latest release (just releaseed), should allow namer to call videohashes-arm64-linux. Verify ffmpeg is on the path, as videohashes-arm64-linux calls ffmpeg as well.

4c0d3r commented 1 year ago

Just verified, nothing special in the ffmpeg/fprobe look up, just need to be on path: https://github.com/stashapp/stash/blob/develop/pkg/ffmpeg/downloader.go#L21-L39

4c0d3r commented 1 year ago

@gebau00a please try with latest release and update here, I suspect it should work. Tempted to close this, but don't want to be perceived as rude, so please let me know what happens with the latest namer and ffmpeg/ffprobe verified on the path.

gebau00a commented 1 year ago

Journal log, namer run as systems service:

Dec 24 00:55:08 odroid python[113640]: 2022-12-24 00:55:08 | ℹ️ INFO     | ffmpeg version 4.4.3 found
Dec 24 00:55:08 odroid python[113640]: 2022-12-24 00:55:08 | ℹ️ INFO     | ffprobe version 4.4.3 found
Dec 24 00:55:08 odroid python[113640]: 2022-12-24 00:55:08 | ℹ️ INFO     | Start porndb scene watcher.... watching: /mnt/downloads/toTag/newFiles
Dec 24 00:55:08 odroid python[113640]: 2022-12-24 00:55:08 | ℹ️ INFO     | Starting server: http://0.0.0.0:6980/
Dec 24 00:55:25 odroid python[113640]: 2022-12-24 00:55:25 | ℹ️ INFO     | watchdog process called for xxx
Dec 24 00:55:25 odroid python[113640]: 2022-12-24 00:55:25 | ℹ️ INFO     | Moving /mnt/downloads/toTag/newFiles/xxx to /mnt/downloads/toTag/tag_work/xxx for processing
Dec 24 00:55:25 odroid python[113640]: 2022-12-24 00:55:25 | ℹ️ INFO     | Processing: xxx
Dec 24 00:55:25 odroid python[113640]: 2022-12-24 00:55:25 | ℹ️ INFO     | Calculating phash for file "xxx"
Dec 24 00:55:25 odroid python[113640]: 2022-12-24 00:55:25 | ❌ ERROR    | An error has been caught in function '__processing_thread', process 'MainProcess' (113640), thread 'Thread-2 (__processing_thread)' (281472612823456):
Dec 24 00:55:25 odroid python[113640]: Traceback (most recent call last):
Dec 24 00:55:25 odroid python[113640]:   File "/usr/lib/python3.11/threading.py", line 995, in _bootstrap
Dec 24 00:55:25 odroid python[113640]:     self._bootstrap_inner()
Dec 24 00:55:25 odroid python[113640]:   File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
Dec 24 00:55:25 odroid python[113640]:     self.run()
Dec 24 00:55:25 odroid python[113640]:   File "/usr/lib/python3.11/threading.py", line 975, in run
Dec 24 00:55:25 odroid python[113640]:     self._target(*self._args, **self._kwargs)
Dec 24 00:55:25 odroid python[113640]: > File "/home/andy/.local/lib/python3.11/site-packages/namer/watchdog.py", line 137, in __processing_thread
Dec 24 00:55:25 odroid python[113640]:     handle(command)
Dec 24 00:55:25 odroid python[113640]:   File "/home/andy/.local/lib/python3.11/site-packages/namer/watchdog.py", line 54, in handle
Dec 24 00:55:25 odroid python[113640]:     process_file(command)
Dec 24 00:55:25 odroid python[113640]:   File "/home/andy/.local/lib/python3.11/site-packages/namer/namer.py", line 176, in process_file
Dec 24 00:55:25 odroid python[113640]:     phash = VideoPerceptualHash().get_stash_phash(command.target_movie_file) if command.config.search_phash else None
Dec 24 00:55:25 odroid python[113640]:   File "/home/andy/.local/lib/python3.11/site-packages/namer/videophash.py", line 91, in get_stash_phash
Dec 24 00:55:25 odroid python[113640]:     return self._get_stash_phash(file, stat.st_size, stat.st_mtime)
Dec 24 00:55:25 odroid python[113640]:   File "/home/andy/.local/lib/python3.11/site-packages/namer/videophash.py", line 96, in _get_stash_phash
Dec 24 00:55:25 odroid python[113640]:     return self.__execute_stash_phash(file)
Dec 24 00:55:25 odroid python[113640]:   File "/home/andy/.local/lib/python3.11/site-packages/namer/videophash.py", line 134, in __execute_stash_phash
Dec 24 00:55:25 odroid python[113640]:     data = json.loads(stdout, object_hook=lambda d: SimpleNamespace(**d))
Dec 24 00:55:25 odroid python[113640]:   File "/usr/lib/python3.11/json/__init__.py", line 359, in loads
Dec 24 00:55:25 odroid python[113640]:     return cls(**kw).decode(s)
Dec 24 00:55:25 odroid python[113640]:   File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
Dec 24 00:55:25 odroid python[113640]:     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Dec 24 00:55:25 odroid python[113640]:   File "/usr/lib/python3.11/json/decoder.py", line 355, in raw_decode
Dec 24 00:55:25 odroid python[113640]:     raise JSONDecodeError("Expecting value", s, err.value) from None
Dec 24 00:55:25 odroid python[113640]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
gebau00a commented 1 year ago

I think the problem is not in Namer itself, but in the videohash executable

4c0d3r commented 1 year ago

Is xxx a real video file?

gebau00a commented 1 year ago

Yes it is. I only replaced the name as the name has no influence on the bug. None of the files is working

4c0d3r commented 1 year ago

well crap. You can turn off phashes in the config as a work around. see the https://github.com/ThePornDatabase/namer/blob/main/namer/namer.cfg.default for the layout.

I have no way to debug this, so unless you have some spare time and or know a bit of go the above might be your only option . Hopefully someone with the hardware has time to look at the arm64 video hasher.

DirtyRacer1337 commented 1 year ago

Just need to print output and see that it says, errors not in json format even with -json flag

4c0d3r commented 1 year ago

According to gebau00a:

Running videohashes-arm64-linux manually with a link to a video file generates the error ffmpeg/ffprobe executables not found

which are on the path according to namer's log

Improving error handling is a good idea, but it'll lead us back here.

DirtyRacer1337 commented 1 year ago

Added error printing, #125

4c0d3r commented 1 year ago

Releasing that as 1.10.5, maybe it'll provide more info.

DirtyRacer1337 commented 1 year ago

Well if videohashes can't find ffmpeg it's problem with stash code, https://github.com/stashapp/stash

4c0d3r commented 1 year ago

Yeah only down side is that seemed solid to me: https://github.com/stashapp/stash/blob/develop/pkg/ffmpeg/downloader.go#L21-L39

4c0d3r commented 1 year ago

@gebau00a give 1.10.5 a shot when you have a chance -- we can verify the error at least

gebau00a commented 1 year ago

Yeah, cause is the videohash executable

Dec 24 08:36:19 odroid python[121854]: 2022-12-24 08:36:19 | ℹ️ INFO     | ffmpeg version 4.4.3 found
Dec 24 08:36:19 odroid python[121854]: 2022-12-24 08:36:19 | ℹ️ INFO     | ffprobe version 4.4.3 found
Dec 24 08:36:19 odroid python[121854]: 2022-12-24 08:36:19 | ℹ️ INFO     | Start porndb scene watcher.... watching: /mnt/downloads/toTag/newFiles
Dec 24 08:36:20 odroid python[121854]: 2022-12-24 08:36:20 | ℹ️ INFO     | Starting server: http://0.0.0.0:6980/
Dec 24 08:36:31 odroid python[121854]: 2022-12-24 08:36:31 | ℹ️ INFO     | watchdog process called for xxx.mp4
Dec 24 08:36:31 odroid python[121854]: 2022-12-24 08:36:31 | ℹ️ INFO     | Moving /mnt/downloads/toTag/newFiles/xxx.mp4 to /mnt/downloads/toTag/tag_work/xxx.mp4 for processing
Dec 24 08:36:31 odroid python[121854]: 2022-12-24 08:36:31 | ℹ️ INFO     | Processing: /mnt/downloads/toTag/tag_work/xxx.mp4
Dec 24 08:36:31 odroid python[121854]: 2022-12-24 08:36:31 | ℹ️ INFO     | Calculating phash for file "/mnt/downloads/toTag/tag_work/xxx.mp4"
Dec 24 08:36:31 odroid python[121854]: 2022-12-24 08:36:31 | ❌ ERROR    | ffmpeg/ffprobe executables not found
Dec 24 08:36:31 odroid python[121854]: 2022-12-24 08:36:31 | ℹ️ INFO     | Requesting GET "https://api.metadataapi.net/scenes?parse=xxx&limit=25"
Dec 24 08:36:32 odroid python[121854]: 2022-12-24 08:36:32 | ℹ️ INFO     | Requesting GET "https://api.metadataapi.net/scenes?parse=xxx&limit=25"
4c0d3r commented 1 year ago

Opened issue: https://github.com/peolic/videohashes/issues/5 to track this.

gebau00a commented 1 year ago

I put up some comments in the bug report in videohashes Seems like the videohashes executable is only searching in the active working dir for the executable, which is ok for Windows but not standard practice for Unix systems. I posted also a snippet how to correct this, using a different function. Let's hope we see a result soon.

With your correction on the other hand to error handle the phash function, Namer is running already like before, despite now only doing name checking, but good enough until a fix is up by upstream. Thanks for thisA

A symlink of ffmpeg and ffprobe in the tools subdirectory works when running the videohashes program directly, but when namer is started as service, it's not finding them still, as videohashes is searching in the active directory during start of the service.

gebau00a commented 1 year ago

I found a temporary solution for my problem, shifting the working dir of the namer service to /usr/bin

[Unit]
Description=Namer service
Wants=network-online.target
After=network-online.target nss-lookup.target

[Service]
WorkingDirectory=/usr/bin
Type=exec
Restart=always
RestartSec=1
User=andy
UMask=0000
ExecStart=python -m namer watchdog
#StandardOutput=null

[Install]
WantedBy=multi-user.target

For those interested in the watchdog service file. The magic here is the WorkingDirectory shift, which is done before namer starts, so all current directory requests from Videohashes are pointing to /usr/bin/ before searching for ffmpeg and ffprobe executables. It's working for now but I think in the end it's still an upstream bug as it will only work for watchdog now.

4c0d3r commented 1 year ago

OMG.... you really trust this code.... I would copy ffmpeg to the work dir......

gebau00a commented 1 year ago

This is my service script I wrote myself, so this one I trust. Linking in the tools subdirectory only works when executing the file itself, else it runs with the dir from where namer was called, in / by default. Better to let namer run in /usr/bin where the user has no write access, else I have to most likely create symlinks for ffmpeg in /, which I see as way worse.

gebau00a commented 1 year ago

Poelic found the issue. When ffmpeg is in the normal PATH, the configuration options are checked upfront. webp option was missing from my ffmpeg. Compiling it fresh with webp solved the issue.

I only suggested to change the error message as it's not consistent with what happens.

But I propose to close this bug as it's settled and only an upstream improvement left.

Thanks for all your help.

4c0d3r commented 1 year ago

Perhaps Namer should follow stash's led here and download ffmpeg:

https://github.com/stashapp/stash/blob/develop/pkg/ffmpeg/downloader.go#L170-L190

4c0d3r commented 1 year ago

videohasher will get the ability to download ffmpeg: https://github.com/peolic/videohashes/pull/6 which namer will leverage once it's merged.

4c0d3r commented 1 year ago

Fixed in namer in https://github.com/ThePornDatabase/namer/pull/130, now ffmpeg can be download as well. Though the running user needs to have permission to write to the pip install dir.