search

clinton-hall / nzbToMedia

Provides NZB and Torrent postprocessing To CouchPotatoServer, SickBeard/SickRage, HeadPhones, Mylar and Gamez
GNU General Public License v3.0
674 stars 176 forks source link

GIT Update Error #1888

Closed phybersplice closed 1 year ago

phybersplice commented 2 years ago

Describe the bug When the script tries to update, it fails with an error - Update failed, not restarting

Technical Specs

  1. Running on Synology 1821+ with Docker'
  2. Python version 3.9
  3. Download Client: NZBget
  4. Intended Media Management: Radarr, Sonarr

Expected behavior Git Update to happen.

Log

[INFO] Executing post-process-script nzbToMedia/nzbToMedia.py for Scarborough.2021.1080p.WEB.H264-KBOX
[INFO] nzbToMedia: warning: unable to access '/root/.config/git/ignore': Permission denied
[INFO] nzbToMedia: [20:16:10] [INFO]::MAIN: Python v3.9 will reach end of life in 1216 days.
[INFO] nzbToMedia: [20:16:10] [INFO]::MAIN: Checking database structure...
[INFO] nzbToMedia: [20:16:10] [INFO]::MAIN: Checking if git needs an update
[INFO] nzbToMedia: [20:16:10] [INFO]::MAIN: There is a newer version available (you're 1 commit behind)
[INFO] nzbToMedia: [20:16:10] [INFO]::MAIN: Auto-Updating nzbToMedia, Please wait ...
[INFO] nzbToMedia: warning: unable to access '/root/.config/git/ignore': Permission denied
[INFO] nzbToMedia: [20:16:11] [ERROR]::MAIN: Update failed, not restarting. Check your log for more information.
[INFO] nzbToMedia: [20:16:11] [INFO]::MAIN: nzbToMedia Version:27669389216902d1085660167e7bda0bd8527ecf Branch:master (Linux 4.4.180+)
[INFO] nzbToMedia: [20:16:11] [INFO]::MAIN: #########################################################
[INFO] nzbToMedia: [20:16:11] [INFO]::MAIN: ## ..::[nzbToMedia.py]::.. ##
[INFO] nzbToMedia: [20:16:11] [INFO]::MAIN: #########################################################
[INFO] nzbToMedia: [20:16:11] [INFO]::MAIN: Script triggered from NZBGet Version 21.1.
[INFO] nzbToMedia: [20:16:11] [INFO]::MAIN: Auto-detected SECTION:Radarr
[INFO] nzbToMedia: [20:16:11] [INFO]::MAIN: Calling Radarr:movies to post-process:Scarborough.2021.1080p.WEB.H264-KBOX
clinton-hall commented 2 years ago

warning: unable to access '/root/.config/git/ignore': Permission denied

I'm guessing that the script was cloned by root user, but NZBGet is running as a different user? Perhaps try cloning the script using the same user that NZBGet runs as.

phybersplice commented 2 years ago

Might you be able to give me a hand in fixing the dockerfile? https://github.com/phybersplice/docker-nzbget/blob/master/Dockerfile

I run the user as "dockerlimited" on the NAS.
I'm sure I could manually update using the git command ... but that kind of defeats the purpose, right?

clinton-hall commented 2 years ago

I don't know docker but from what I can see https://github.com/phybersplice/docker-nzbget/blob/master/Dockerfile#L131 COPY root/ / is copying the root/ dir into / So I think perhaps add --from=buildstage COPY /root/ /root/

that way the docker image would be able to access /root/ with the git config that was used to clone?

phybersplice commented 2 years ago

I will try that tomorrow.

Thanks for the tip.

On Jun 9, 2022, at 10:54 PM, Clinton Hall @.***> wrote:

I don't know docker but from what I can see https://github.com/phybersplice/docker-nzbget/blob/master/Dockerfile#L131 https://github.com/phybersplice/docker-nzbget/blob/master/Dockerfile#L131 COPY root/ / is copying the root/ dir into / So I think perhaps add --from=buildstage COPY /root/ /root/

that way the docker image would be able to access /root/ with the git config that was used to clone?

— Reply to this email directly, view it on GitHub https://github.com/clinton-hall/nzbToMedia/issues/1888#issuecomment-1151870145, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZ75QXT76K4LOLNZVMGUEDVOKU7TANCNFSM5YEXLZLA. You are receiving this because you authored the thread.

phybersplice commented 2 years ago

so much for trying it tomorrow ... just tried it now.
It didn't work. When nzbget doesn't start up properly even though it passes a docker build/tag/push.

clinton-hall commented 2 years ago

damn... I might have put an extra / in there. perhaps COPY root/ /root/ I also note that the linked file has changed a bit so not sure if the order has changed etc... looking at this again, it looks like the git clone happens after the copy anyway, so probably best not to mess with that.

From what I can see, using git in the dockerfile is for cloning repositories to build, and it is not intended to be updateable within the docker.

so I think the best way here is to remove the nzbToMedia clone from docker file and put the clone into the runtime

https://github.com/phybersplice/docker-nzbget/blob/master/root/etc/services.d/nzbget/run#L11

add something like

[ -d /app/nzbget/share/nzbget/scripts/nzbToMedia/.git ] || git clone https://github.com/clinton-hall/nzbToMedia.git /app/nzbget/share/nzbget/scripts/nzbToMedia

phybersplice commented 2 years ago

Yeah - you did :) I've updated the GIT and the Docker Hub with the current config as you suggested, commenting out the git clone line in the Dockerfile.
I've updated the file in the /root/etc as you also suggested.
I now get these errors when doing a post-process.

Mon Sep 26 20:16:26 2022    INFO    Post-processing again Bullet.Train.2022.1080p.AMZN.WEB-DL.DDP5.1.H.264.DUAL-alfaHD
Mon Sep 26 20:16:26 2022    INFO    Bullet.Train.2022.1080p.AMZN.WEB-DL.DDP5.1.H.264.DUAL-alfaHD returned from history back to download queue
Mon Sep 26 20:16:26 2022    INFO    Queueing Bullet.Train.2022.1080p.AMZN.WEB-DL.DDP5.1.H.264.DUAL-alfaHD for post-processing
Mon Sep 26 20:16:26 2022    INFO    Cleaning up Bullet.Train.2022.1080p.AMZN.WEB-DL.DDP5.1.H.264.DUAL-alfaHD
Mon Sep 26 20:16:26 2022    INFO    Nothing to cleanup for Bullet.Train.2022.1080p.AMZN.WEB-DL.DDP5.1.H.264.DUAL-alfaHD
Mon Sep 26 20:16:26 2022    INFO    Executing post-process-script nzbToMedia/nzbToMedia.py for Bullet.Train.2022.1080p.AMZN.WEB-DL.DDP5.1.H.264.DUAL-alfaHD
Mon Sep 26 20:16:26 2022    INFO    nzbToMedia: fatal: detected dubious ownership in repository at '/app/nzbget/share/nzbget/scripts/nzbToMedia'
Mon Sep 26 20:16:26 2022    INFO    nzbToMedia: To add an exception for this directory, call:
Mon Sep 26 20:16:26 2022    INFO    nzbToMedia:     git config --global --add safe.directory /app/nzbget/share/nzbget/scripts/nzbToMedia
Mon Sep 26 20:16:26 2022    INFO    nzbToMedia: fatal: detected dubious ownership in repository at '/app/nzbget/share/nzbget/scripts/nzbToMedia'
Mon Sep 26 20:16:26 2022    INFO    nzbToMedia: To add an exception for this directory, call:
Mon Sep 26 20:16:26 2022    INFO    nzbToMedia:     git config --global --add safe.directory /app/nzbget/share/nzbget/scripts/nzbToMedia
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia: Changing to directory: /app/nzbget/share/nzbget/scripts/nzbToMedia
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia: -- Cleaning bytecode --
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia: WARNING: Automatic cleanup could not be executed.
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia:          If errors occur, manual cleanup may be required.
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia: REASON : Error Code: 128
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia: -- Cleaning folders: ['libs', 'core'] --
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia: WARNING: Automatic cleanup could not be executed.
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia:          If errors occur, manual cleanup may be required.
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia: REASON : Error Code: 128
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia: Returning to directory:  /run/s6/legacy-services/nzbget
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia: -- Cleanup finished --
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia: Traceback (most recent call last):
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia:   File "/app/nzbget/share/nzbget/scripts/nzbToMedia/nzbToMedia.py", line 735, in <module>
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia:     import core
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia:   File "/app/nzbget/share/nzbget/scripts/nzbToMedia/core/__init__.py", line 52, in <module>
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia:     from core import logger, main_db, version_check, databases, transcoder
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia:   File "/app/nzbget/share/nzbget/scripts/nzbToMedia/core/databases.py", line 11, in <module>
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia:     from core.utils import backup_versioned_file
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia:   File "/app/nzbget/share/nzbget/scripts/nzbToMedia/core/utils/__init__.py", line 14, in <module>
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia:     from core.utils.download_info import get_download_info, update_download_info_status
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia:   File "/app/nzbget/share/nzbget/scripts/nzbToMedia/core/utils/download_info.py", line 14, in <module>
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia:     database = main_db.DBConnection()
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia:   File "/app/nzbget/share/nzbget/scripts/nzbToMedia/core/main_db.py", line 65, in __init__
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia:     self.connection = sqlite3.connect(db_filename(filename), 20)
Mon Sep 26 20:16:28 2022    INFO    nzbToMedia: sqlite3.OperationalError: unable to open database file
Mon Sep 26 20:16:28 2022    ERROR   Post-process-script nzbToMedia/nzbToMedia.py for Bullet.Train.2022.1080p.AMZN.WEB-DL.DDP5.1.H.264.DUAL-alfaHD failed (terminated with unknown status)
Mon Sep 26 20:16:28 2022    INFO    Collection Bullet.Train.2022.1080p.AMZN.WEB-DL.DDP5.1.H.264.DUAL-alfaHD added to history
clinton-hall commented 2 years ago

When it is cloned this way, who is the owner and what permissions are set for /app/nzbget/share/nzbget/scripts/nzbToMedia

phybersplice commented 2 years ago

Looks like it's root.

root@b2ec63e641d6:/app/nzbget/share/nzbget/scripts# ls -ltr
total 16
-rwxr-xr-x 1 abc  abc   3370 Sep 26 13:31 Logger.py
-rwxr-xr-x 1 abc  abc  10475 Sep 26 13:31 EMail.py
drwxr-xr-x 1 root root   786 Sep 26 20:15 nzbToMedia

root@b2ec63e641d6:/app/nzbget/share/nzbget/scripts/nzbToMedia# ls -ltr
total 228
-rw-r--r-- 1 root root    26 Sep 26 20:15 _config.yml
-rwxr-xr-x 1 root root 16974 Sep 26 20:15 TorrentToMedia.py
-rw-r--r-- 1 root root  6369 Sep 26 20:15 cleanup.py
-rw-r--r-- 1 root root  1693 Sep 26 20:15 azure-pipelines.yml
-rw-r--r-- 1 root root 25595 Sep 26 20:15 autoProcessMedia.cfg.spec
-rwxr-xr-x 1 root root   517 Sep 26 20:15 getffmpeg.sh
-rw-r--r-- 1 root root  5475 Sep 26 20:15 eol.py
drwxr-xr-x 1 root root   334 Sep 26 20:15 core
drwxr-xr-x 1 root root   320 Sep 26 20:15 libs
-rw-r--r-- 1 root root 35127 Sep 26 20:15 license.txt
-rwxr-xr-x 1 root root  3378 Sep 26 20:15 nzbToMylar.py
-rwxr-xr-x 1 root root 33003 Sep 26 20:15 nzbToMedia.py
-rwxr-xr-x 1 root root  6979 Sep 26 20:15 nzbToLidarr.py
-rwxr-xr-x 1 root root  3413 Sep 26 20:15 nzbToLazyLibrarian.py
-rwxr-xr-x 1 root root  3824 Sep 26 20:15 nzbToHeadPhones.py
-rwxr-xr-x 1 root root  2993 Sep 26 20:15 nzbToGamez.py
-rwxr-xr-x 1 root root  7699 Sep 26 20:15 nzbToCouchPotato.py
-rw-r--r-- 1 root root  3738 Sep 26 20:15 setup.py
-rwxr-xr-x 1 root root  7462 Sep 26 20:15 nzbToWatcher3.py
-rwxr-xr-x 1 root root  7580 Sep 26 20:15 nzbToSickBeard.py
-rwxr-xr-x 1 root root  7581 Sep 26 20:15 nzbToSiCKRAGE.py
-rwxr-xr-x 1 root root  7469 Sep 26 20:15 nzbToRadarr.py
-rwxr-xr-x 1 root root  7369 Sep 26 20:15 nzbToNzbDrone.py
-rw-r--r-- 1 root root  3849 Sep 26 20:15 tox.ini
drwxr-xr-x 1 root root   110 Sep 26 20:15 tests
clinton-hall commented 2 years ago

ok... so the run-time script still runs as root...

what happens if you add s6-setuidgid abc to the start of that line

[ -d /app/nzbget/share/nzbget/scripts/nzbToMedia/.git ] || s6-setuidgid abc git clone https://github.com/clinton-hall/nzbToMedia.git /app/nzbget/share/nzbget/scripts/nzbToMedia

this should run the git clone as the abc account? the same as for NZBGet

phybersplice commented 2 years ago

I will set that up and push to Git/Docker momentarily and will test and report back in a few minutes. For real :)

phybersplice commented 2 years ago

So after all this, I still get the error "nzbToMedia: warning: unable to access '/root/.config/git/ignore': Permission Denied :). The other errors have gone away and it's cleaned up nicely.

Mon Sep 26 20:48:49 2022    INFO    Post-processing again Bullet.Train.2022.1080p.AMZN.WEB-DL.DDP5.1.H.264.DUAL-alfaHD
Mon Sep 26 20:48:49 2022    INFO    Bullet.Train.2022.1080p.AMZN.WEB-DL.DDP5.1.H.264.DUAL-alfaHD returned from history back to download queue
Mon Sep 26 20:48:49 2022    INFO    Queueing Bullet.Train.2022.1080p.AMZN.WEB-DL.DDP5.1.H.264.DUAL-alfaHD for post-processing
Mon Sep 26 20:48:49 2022    INFO    Cleaning up Bullet.Train.2022.1080p.AMZN.WEB-DL.DDP5.1.H.264.DUAL-alfaHD
Mon Sep 26 20:48:49 2022    INFO    Nothing to cleanup for Bullet.Train.2022.1080p.AMZN.WEB-DL.DDP5.1.H.264.DUAL-alfaHD
Mon Sep 26 20:48:49 2022    INFO    Executing post-process-script nzbToMedia/nzbToMedia.py for Bullet.Train.2022.1080p.AMZN.WEB-DL.DDP5.1.H.264.DUAL-alfaHD
Mon Sep 26 20:48:49 2022    INFO    nzbToMedia: warning: unable to access '/root/.config/git/ignore': Permission denied
Mon Sep 26 20:48:52 2022    INFO    nzbToMedia: [20:48:52] [WARNING]::MAIN:  category is set for CouchPotato and Watcher3. Please check your config in NZBGet
Mon Sep 26 20:48:52 2022    INFO    nzbToMedia: [20:48:52] [INFO]::MAIN: Python v3.9 will reach end of life in 1105 days.
Mon Sep 26 20:48:52 2022    INFO    nzbToMedia: [20:48:52] [INFO]::MAIN: Checking database structure...
Mon Sep 26 20:48:52 2022    INFO    nzbToMedia: [20:48:52] [INFO]::MAIN: Database upgrade required: Initial Schema
Mon Sep 26 20:48:52 2022    INFO    nzbToMedia: [20:48:52] [INFO]::MAIN: Checking if git needs an update
Mon Sep 26 20:48:53 2022    INFO    nzbToMedia: [20:48:53] [INFO]::MAIN: No update needed
Mon Sep 26 20:48:53 2022    INFO    nzbToMedia: [20:48:53] [INFO]::MAIN: nzbToMedia Version:7436ba77161af412c00e745e13149576da583721 Branch:master (Linux 4.4.180+)
Mon Sep 26 20:48:53 2022    INFO    nzbToMedia: [20:48:53] [INFO]::MAIN: #########################################################
Mon Sep 26 20:48:53 2022    INFO    nzbToMedia: [20:48:53] [INFO]::MAIN: ## ..::[nzbToMedia.py]::.. ##
Mon Sep 26 20:48:53 2022    INFO    nzbToMedia: [20:48:53] [INFO]::MAIN: #########################################################
Mon Sep 26 20:48:53 2022    INFO    nzbToMedia: [20:48:53] [INFO]::MAIN: Script triggered from NZBGet Version 21.1.
Mon Sep 26 20:48:53 2022    INFO    nzbToMedia: [20:48:53] [INFO]::MAIN: Auto-detected SECTION:Radarr
Mon Sep 26 20:48:53 2022    INFO    nzbToMedia: [20:48:53] [INFO]::MAIN: Calling Radarr:movies to post-process:Bullet.Train.2022.1080p.AMZN.WEB-DL.DDP5.1.H.264.DUAL-alfaHD
Mon Sep 26 20:48:53 2022    INFO    nzbToMedia: [20:48:53] [INFO]::MAIN: Attemping imdbID lookup for Bullet.Train.2022.1080p.AMZN.WEB-DL.DDP5.1.H.264.DUAL-alfaHD
Mon Sep 26 20:48:53 2022    INFO    nzbToMedia: [20:48:53] [INFO]::MAIN: Searching folder and file names for imdbID ...
Mon Sep 26 20:48:53 2022    INFO    nzbToMedia: [20:48:53] [INFO]::MAIN: Found imdbID [tt12593682] via file name
Mon Sep 26 20:48:53 2022    INFO    nzbToMedia: [20:48:53] [INFO]::TRANSCODER: Checking [Bullet.Train.2022.1080p.AMZN.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.cp(tt12593682).mkv] for corruption, please stand by ...
Mon Sep 26 20:48:53 2022    INFO    nzbToMedia: [20:48:53] [INFO]::TRANSCODER: SUCCESS: [Bullet.Train.2022.1080p.AMZN.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.cp(tt12593682).mkv] has no corruption.
clinton-hall commented 2 years ago

ok.... weird...

perhaps [ -d /app/nzbget/share/nzbget/scripts/nzbToMedia/.git ] || s6-setuidgid abc export HOME=/home/abc git clone https://github.com/clinton-hall/nzbToMedia.git /app/nzbget/share/nzbget/scripts/nzbToMedia

this should in theory ensure that the "correct" (read as non-root) home directory is set and git should write its files there

phybersplice commented 2 years ago

this didn't run git clone so the scripts aren't present.

clinton-hall commented 2 years ago

ah... without the export?

[ -d /app/nzbget/share/nzbget/scripts/nzbToMedia/.git ] || s6-setuidgid abc HOME=/home/abc git clone https://github.com/clinton-hall/nzbToMedia.git /app/nzbget/share/nzbget/scripts/nzbToMedia

phybersplice commented 2 years ago

this runs, grabs the nzbToMedia scripts: [ -d /app/nzbget/share/nzbget/scripts/nzbToMedia/.git ] || s6-setuidgid abc git clone https://github.com/clinton-hall/nzbToMedia.git /app/nzbget/share/nzbget/scripts/nzbToMedia

This doesn't: [ -d /app/nzbget/share/nzbget/scripts/nzbToMedia/.git ] || s6-setuidgid abc export HOME=/home/abc git clone https://github.com/clinton-hall/nzbToMedia.git /app/nzbget/share/nzbget/scripts/nzbToMedia

I think we're on different TZ's. I'm Eastern TZ. I'll be back on at 9am Eastern (my time). Hoping you can figure this one out. I'm lost.

Thanks :)

clinton-hall commented 2 years ago

ok, thr [ -d /app/nzbget/share/nzbget/scripts/nzbToMedia/.git ] || s6-setuidgid abc HOME=/home/abc git clone https://github.com/clinton-hall/nzbToMedia.git /app/nzbget/share/nzbget/scripts/nzbToMedia

or perhaps

if [[ !-d /app/nzbget/share/nzbget/scripts/nzbToMedia/.git ]]; then
  export HOME=/home/abc 
  s6-setuidgid abc git clone https://github.com/clinton-hall/nzbToMedia.git /app/nzbget/share/nzbget/scripts/nzbToMedia
fi

I'm in New Zealand, current NZDT = UTC + 13 hours. Once you have tried these let me know. The problem is I don't have a docker setup here to test.

phybersplice commented 2 years ago

Tried both suggestions. Neither pull your GIT into the directory. In fact, the nzbget daemon doesn't run.

clinton-hall commented 2 years ago

I have added a possible fix via git pull.

phybersplice commented 1 year ago

Closing off this issue