search

I-am-PUID-0 / DMB

Debrid Media Bridge
MIT License
125 stars 9 forks source link

Riven v0.12+ breaking changes required deleting settings and database (Previously: Torrentio is broken #55

Closed profucius closed 2 months ago

profucius commented 2 months ago

New bug after closing #54 . I opened a ticket on the Riven repo directly for this one, and they referred me back to you/DMB again.

https://github.com/rivenmedia/riven/issues/722

Essentially the issue is that when I enable Torrentio in Riven settings, it crashes the Riven backend. Then when I restart the stack, it cannot load at all, and shuts down. When I asked for help, they said something about needing a host/port for it? I'm a bit confused.

I can modify and save other settings successfully, but when I enable Torrentio and save, this crashes.

I-am-PUID-0 commented 2 months ago

You know what Profucius says?

"You fix a man's bugs, he's good for a day. You teach a man to fix bugs, he's good for a lifetime"

Ok, bad jokes aside. Can you send me a list of the DMB and Riven settings you had enabled; I'll see if I can replicate the bug.

profucius commented 2 months ago

You know what Profucius says?

You got me there haha! Although normally I am adept at handling bugs, but with the combination of being newish with docker, and Riven making deep changes, I'm a bit turned around here.

I've attached the config files via a zip file in the link below, figured that is easier. Thanks for taking a look. btw if you need to swap in your tokens for testing, I replaced them all with "redacted".

profucius DMB configs.zip

I-am-PUID-0 commented 2 months ago

Thanks for the additional info. I've tried replicating the bug, but I have been unsuccessful. Is the bug consistently occurring?

NoSync commented 2 months ago

Same issue for me. In fact, even after disabling torrentio manually from riven’s json settings, I get the same issue on startup and everything will eventually shut down - I’m not quite sure how to restore functionality.

I also tried a hard reset of Riven’s DB, but no luck.

profucius commented 2 months ago

Just updated to DMB v5.1.10, issue is still happening unfortunately. Not sure what else to try, other than totally scrapping my entire setup and starting over again, which would suck as I spent hours customizing it.

This is what happens in the logs immediately after clicking Save after enabling Torrentio in Riven WebUI:

Sep 25, 2024 08:46:06 - ERROR - PostgreSQL subprocess: type "states" does not exist
Sep 25, 2024 08:46:06 - INFO - PostgreSQL subprocess: 2024-09-25 13:46:06.908 UTC [223] STATEMENT:  ALTER TABLE "MediaItem" ALTER COLUMN last_state TYPE states
Sep 25, 2024 08:46:06 - WARNING - riven_backend subprocess: | db.run_migrations - Error running migrations: (psycopg2.errors.UndefinedObject) type "states" does not exist
Sep 25, 2024 08:46:06 - INFO - riven_backend subprocess: [SQL: ALTER TABLE "MediaItem" ALTER COLUMN last_state TYPE states ]
Sep 25, 2024 08:46:06 - INFO - riven_backend subprocess: (Background on this error at: https://sqlalche.me/e/20/f405)
Sep 25, 2024 08:46:07 - INFO - riven_backend subprocess: Generating /riven/backend/data/alembic/2024_09_25_338ab47698b5_auto_upg_20240925084606.py ...  done
Sep 25, 2024 08:46:07 - ERROR - PostgreSQL subprocess: type "states" does not exist
Sep 25, 2024 08:46:07 - INFO - PostgreSQL subprocess: 2024-09-25 13:46:07.194 UTC [223] STATEMENT:  ALTER TABLE "MediaItem" ALTER COLUMN last_state TYPE states
Sep 25, 2024 08:46:07 - WARNING - riven_backend subprocess: | db.run_migrations - Error running migrations: (psycopg2.errors.UndefinedObject) type "states" does not exist
Sep 25, 2024 08:46:07 - INFO - riven_backend subprocess: [SQL: ALTER TABLE "MediaItem" ALTER COLUMN last_state TYPE states ]
Sep 25, 2024 08:46:07 - INFO - riven_backend subprocess: (Background on this error at: https://sqlalche.me/e/20/f405)
Sep 25, 2024 08:46:07 - INFO - riven_backend subprocess: Generating /riven/backend/data/alembic/2024_09_25_5df5b5413669_auto_upg_20240925084607.py ...  done
Sep 25, 2024 08:46:07 - CRITICAL - riven_backend subprocess: | main.<module> - Server has been stopped
Sep 25, 2024 08:46:08 - INFO - riven_backend subprocess: | 👾 API       | main.dispatch - GET /settings/load - 200 - 2.51s
Sep 25, 2024 08:46:08 - INFO - riven_frontend subprocess: TypeError: fetch failed
Sep 25, 2024 08:46:08 - INFO - riven_frontend subprocess: at node:internal/deps/undici/undici:12500:13
Sep 25, 2024 08:46:08 - INFO - riven_frontend subprocess: at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
Sep 25, 2024 08:46:08 - INFO - riven_frontend subprocess: [cause]: Error: connect ECONNREFUSED 127.0.0.1:8080
Sep 25, 2024 08:46:08 - INFO - riven_frontend subprocess: at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1605:16) {
Sep 25, 2024 08:46:08 - INFO - riven_frontend subprocess: errno: -111,
Sep 25, 2024 08:46:08 - INFO - riven_frontend subprocess: code: 'ECONNREFUSED',
Sep 25, 2024 08:46:08 - INFO - riven_frontend subprocess: syscall: 'connect',
Sep 25, 2024 08:46:08 - INFO - riven_frontend subprocess: address: '127.0.0.1',
Sep 25, 2024 08:46:08 - INFO - riven_frontend subprocess: port: 8080
Sep 25, 2024 08:46:08 - INFO - riven_frontend subprocess: }
Sep 25, 2024 08:46:08 - INFO - riven_frontend subprocess: }

And then in the Riven WebUI this shows: image

And then I refresh the page and this shows: image

With only this in the log:

Sep 25, 2024 08:47:37 - INFO - riven_frontend subprocess: }
Sep 25, 2024 08:48:08 - ERROR - The Riven backend process is not running.

If I can find the time, I'll try starting a separate brand new DMB stack and see if I can possibly reproduce it clean.

profucius commented 2 months ago

Same issue for me. In fact, even after disabling torrentio manually from riven’s json settings, I get the same issue on startup and everything will eventually shut down - I’m not quite sure how to restore functionality.

I also tried a hard reset of Riven’s DB, but no luck.

This shouldn't be happening to you. I'm able to reverse it and get back into Riven again by simply changing this line to false in the Riven settings.json file, then restarting the stack (it takes about 3-5 minutes to fully boot)

        "torrentio": {
            "enabled": false,
I-am-PUID-0 commented 2 months ago

Try deleting the Riven/data/alembic directory and restarting.

I-am-PUID-0 commented 2 months ago

As a last option, try deleting the PostgreSQL data directory and restarting.

profucius commented 2 months ago

Try deleting the Riven/data/alembic directory and restarting.

I tried this and it did not work.

As a last option, try deleting the PostgreSQL data directory and restarting.

Then I tried this, and it worked! Torrentio enabled successfully and Riven continued running. The downside is that I lost everything in the Library tab that Riven was tracking, but honestly that is a sacrifice I'm willing to make to get it running again.

Any idea how this might've happened? I'm currently assuming something went wrong with the recent Riven major changes.

Also is this technically a DMB issue or is it a Riven issue? I'm considering updating my original ticket on the Riven repo.

I-am-PUID-0 commented 2 months ago

It was likely due to the Riven update that required the database to be reset; new states were added. From the logs riven_backend subprocess: | db.run_migrations - Error running migrations: (psycopg2.errors.UndefinedObject) type "states" does not exist it was trying to migrate the database, but failing.

NoSync commented 2 months ago

Something's even more odd on my side.

  1. I made sure that torrentio is disabled in settings.json, and it is. Restarted, didn't help.
  2. I zapped the alembic directory. Restarted, the directory is not recreated, didn't help.
  3. I zapped the pgsql data directory. Restarted, the data directory is recreated, didn't help.

Relevant part of the logs, for reference:

Sep 25, 2024 16:12:34 - INFO - Attempt 10/10 to fetch settings from http://127.0.0.1:8080/settings/get/all
Sep 25, 2024 16:12:34 - ERROR - Error fetching settings: HTTPConnectionPool(host='127.0.0.1', port=8080): Max retries exceeded with url: /settings/get/all (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f29379c3f10>: Failed to establish a new connection: [Errno 111] Connection refused'))
Sep 25, 2024 16:12:34 - ERROR - Max retries reached. Failed to fetch settings from http://127.0.0.1:8080/settings/get/all
Sep 25, 2024 16:12:34 - ERROR - Error loading Riven settings: No active exception to reraise
Sep 25, 2024 16:12:34 - ERROR - An error occurred in the Riven backend setup: No active exception to reraise
Sep 25, 2024 16:12:34 - INFO - Shutdown signal received. Cleaning up...
Sep 25, 2024 16:12:34 - INFO - Stopping riven_frontend
Sep 25, 2024 16:12:34 - WARNING - riven_frontend process not found or already stopped.
Sep 25, 2024 16:12:34 - INFO - Stopping riven_backend
Sep 25, 2024 16:12:34 - INFO - riven_backend process terminated gracefully.

And this is the relevant part of my docker compose:

services:
  dmb:
    image: iampuid0/dmb:latest
    container_name: dmb
    restart: unless-stopped
    stop_grace_period: 60s
    stdin_open: true
    tty: true
    volumes:
      - /home/nosync/media/dmb/config:/config:rw
      - /home/nosync/media/dmb/log:/log:rw
      - /home/nosync/media/dmb/zurg/RD:/zurg/RD:rw 
      - /home/nosync/media/dmb/zurg/mnt:/data:rshared
      - /home/nosync/media/dmb/riven/data:/riven/backend/data:rw
      - /home/nosync/media/dmb/riven/mnt:/mnt:rw
      - /home/nosync/media/dmb/pgsql/data:/postgres_data:rw
      ## Location for rclone cache if enabled
      - /data/cache/rclone:/cache:rw
      - /home/nosync/media/dmb/zilean/data:/zilean/app/data:rw
    environment:
      - TZ=Europe/Zurich
      - PUID=1000
      - PGID=1000
      - ZURG_ENABLED=true
      - ZURG_UPDATE=true
      - ZURG_LOG_LEVEL=info
      - RD_API_KEY=redacted
      - RCLONE_MOUNT_NAME=dmb
      - RCLONE_DIR_CACHE_TIME=10s
      - RCLONE_CACHE_DIR=/cache
      - RCLONE_VFS_READ_CHUNK_SIZE=1M
      - RCLONE_VFS_READ_CHUNK_SIZE_LIMIT=32M
      - RIVEN_ENABLED=true
      - RIVEN_BACKEND_UPDATE=true
      - RIVEN_FRONTEND_UPDATE=true
      - RIVEN_LOG_LEVEL=info
      - FRONTEND_LOGS=off
      - ORIGIN=https://redacted.proxy.com      
      - PLEX_TOKEN=redacted
      - PLEX_ADDRESS=https://redacted2.proxy.com
      - DUPLICATE_CLEANUP=true
      - CLEANUP_INTERVAL=24
      - SEERR_API_KEY=redacted
      - SEERR_ADDRESS=https://redacted3.proxy.com
      - ZILEAN_ENABLED=true
      - ZILEAN_UPDATE=true
      - DMB_LOG_LEVEL=info
      - COLOR_LOG_ENABLED=true
    devices:
      - /dev/fuse:/dev/fuse:rwm
    cap_add:
      - SYS_ADMIN     
    security_opt:
      - apparmor:unconfined
      - no-new-privileges
    healthcheck:
      start_period: 1m
      interval: 10s
      retries: 3
    networks:
      macvlan-docker:
        ipv4_address: 192.168.50.245
profucius commented 2 months ago

It was likely due to the Riven update that required the database to be reset; new states were added.

Interesting, I have painfully come to realize how huge of a change this Riven update was after all this. They said they'd announced it on their Discord, but I wasn't a member of it yet, but even if I had been I didn't see where they had used @everyone so I don't know if I'd have even been notified. I requested that they make it more clear on the github releases page when a breaking change occurs.

Would it be possible for you to reflect such major changes onto your github releases page too? The extra visibility would help prevent some potential catastrophes in the future, especially to those who have installed DMB but aren't following the original devs/discords.

One last question, is it possible to disable DMB from auto-updating all the contained softwares on every release? I'm currently assuming I just need to hardcode the version in the docker compose file for any software I don't want to update. I might do that in Riven for the time being so that I can have a break from this.

profucius commented 2 months ago

Sep 25, 2024 16:12:34 - INFO - Attempt 10/10 to fetch settings from http://127.0.0.1:8080/settings/get/all Sep 25, 2024 16:12:34 - ERROR - Error fetching settings: HTTPConnectionPool(host='127.0.0.1', port=8080): Max retries exceeded with url: /settings/get/all (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f29379c3f10>: Failed to establish a new connection: [Errno 111] Connection refused')) Sep 25, 2024 16:12:34 - ERROR - Max retries reached. Failed to fetch settings from http://127.0.0.1:8080/settings/get/all Sep 25, 2024 16:12:34 - ERROR - Error loading Riven settings: No active exception to reraise

I seem to remember seeing this during my testing. Try renaming your Riven settings.json file and restart the stack. It'll recreate a new one. If that works, then it's something in your Riven settings, it might be the "debug" flag set to enabled, that kept failing for me on the newest version. If that doesn't work, then you might try grabbing a clean docker compose yaml from the DMB repo and boot the stack with that. If that works, then deductively re-add your compose settings back in until you find the culprit. If none of that works, PUID might know what to do next.

@NoSync I believe you should open a separate ticket for this, it appears to be a different issue at this point.

I-am-PUID-0 commented 2 months ago

It was likely due to the Riven update that required the database to be reset; new states were added.

Interesting, I have painfully come to realize how huge of a change this Riven update was after all this. They said they'd announced it on their Discord, but I wasn't a member of it yet, but even if I had been I didn't see where they had used @everyone so I don't know if I'd have even been notified. I requested that they make it more clear on the github releases page when a breaking change occurs.

Would it be possible for you to reflect such major changes onto your github releases page too? The extra visibility would help prevent some potential catastrophes in the future, especially to those who have installed DMB but aren't following the original devs/discords.

One last question, is it possible to disable DMB from auto-updating all the contained softwares on every release? I'm currently assuming I just need to hardcode the version in the docker compose file for any software I don't want to update. I might do that in Riven for the time being so that I can have a break from this.

Riven's breaking changes have been the bane of existence! I've requested the same. They use Semantic Versioning, as do I; however, Riven has not incremented the MAJOR field on a breaking change.

As for tracking it on the DMB repo, that's a bit difficult to do when no changes are required for the DMB code to support a Riven breaking change - where typically the indicator would be the MAJOR field for the DMB version.

I-am-PUID-0 commented 2 months ago

Something's even more odd on my side.

  1. I made sure that torrentio is disabled in settings.json, and it is. Restarted, didn't help.
  2. I zapped the alembic directory. Restarted, the directory is not recreated, didn't help.
  3. I zapped the pgsql data directory. Restarted, the data directory is recreated, didn't help.

Relevant part of the logs, for reference:

Sep 25, 2024 16:12:34 - INFO - Attempt 10/10 to fetch settings from http://127.0.0.1:8080/settings/get/all
Sep 25, 2024 16:12:34 - ERROR - Error fetching settings: HTTPConnectionPool(host='127.0.0.1', port=8080): Max retries exceeded with url: /settings/get/all (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f29379c3f10>: Failed to establish a new connection: [Errno 111] Connection refused'))
Sep 25, 2024 16:12:34 - ERROR - Max retries reached. Failed to fetch settings from http://127.0.0.1:8080/settings/get/all
Sep 25, 2024 16:12:34 - ERROR - Error loading Riven settings: No active exception to reraise
Sep 25, 2024 16:12:34 - ERROR - An error occurred in the Riven backend setup: No active exception to reraise
Sep 25, 2024 16:12:34 - INFO - Shutdown signal received. Cleaning up...
Sep 25, 2024 16:12:34 - INFO - Stopping riven_frontend
Sep 25, 2024 16:12:34 - WARNING - riven_frontend process not found or already stopped.
Sep 25, 2024 16:12:34 - INFO - Stopping riven_backend
Sep 25, 2024 16:12:34 - INFO - riven_backend process terminated gracefully.

And this is the relevant part of my docker compose:

services:
  dmb:
    image: iampuid0/dmb:latest
    container_name: dmb
    restart: unless-stopped
    stop_grace_period: 60s
    stdin_open: true
    tty: true
    volumes:
      - /home/nosync/media/dmb/config:/config:rw
      - /home/nosync/media/dmb/log:/log:rw
      - /home/nosync/media/dmb/zurg/RD:/zurg/RD:rw 
      - /home/nosync/media/dmb/zurg/mnt:/data:rshared
      - /home/nosync/media/dmb/riven/data:/riven/backend/data:rw
      - /home/nosync/media/dmb/riven/mnt:/mnt:rw
      - /home/nosync/media/dmb/pgsql/data:/postgres_data:rw
      ## Location for rclone cache if enabled
      - /data/cache/rclone:/cache:rw
      - /home/nosync/media/dmb/zilean/data:/zilean/app/data:rw
    environment:
      - TZ=Europe/Zurich
      - PUID=1000
      - PGID=1000
      - ZURG_ENABLED=true
      - ZURG_UPDATE=true
      - ZURG_LOG_LEVEL=info
      - RD_API_KEY=redacted
      - RCLONE_MOUNT_NAME=dmb
      - RCLONE_DIR_CACHE_TIME=10s
      - RCLONE_CACHE_DIR=/cache
      - RCLONE_VFS_READ_CHUNK_SIZE=1M
      - RCLONE_VFS_READ_CHUNK_SIZE_LIMIT=32M
      - RIVEN_ENABLED=true
      - RIVEN_BACKEND_UPDATE=true
      - RIVEN_FRONTEND_UPDATE=true
      - RIVEN_LOG_LEVEL=info
      - FRONTEND_LOGS=off
      - ORIGIN=https://redacted.proxy.com      
      - PLEX_TOKEN=redacted
      - PLEX_ADDRESS=https://redacted2.proxy.com
      - DUPLICATE_CLEANUP=true
      - CLEANUP_INTERVAL=24
      - SEERR_API_KEY=redacted
      - SEERR_ADDRESS=https://redacted3.proxy.com
      - ZILEAN_ENABLED=true
      - ZILEAN_UPDATE=true
      - DMB_LOG_LEVEL=info
      - COLOR_LOG_ENABLED=true
    devices:
      - /dev/fuse:/dev/fuse:rwm
    cap_add:
      - SYS_ADMIN     
    security_opt:
      - apparmor:unconfined
      - no-new-privileges
    healthcheck:
      start_period: 1m
      interval: 10s
      retries: 3
    networks:
      macvlan-docker:
        ipv4_address: 192.168.50.245

Try the below; just cleaned it up a bit:


  dmb:
    image: iampuid0/dmb:latest
    container_name: dmb
    restart: unless-stopped
    stop_grace_period: 60s
    stdin_open: true
    tty: true
    volumes:
      - /home/nosync/media/dmb/config:/config:rw
      - /home/nosync/media/dmb/log:/log:rw
      - /home/nosync/media/dmb/zurg/RD:/zurg/RD:rw 
      - /home/nosync/media/dmb/zurg/mnt:/data:rshared
      - /home/nosync/media/dmb/riven/data:/riven/backend/data:rw
      - /home/nosync/media/dmb/riven/mnt:/mnt:rw
      - /home/nosync/media/dmb/pgsql/data:/postgres_data:rw
      ## Location for rclone cache if enabled
      - /data/cache/rclone:/cache:rw
      - /home/nosync/media/dmb/zilean/data:/zilean/app/data:rw
    environment:
      - TZ=Europe/Zurich
      - PUID=1000
      - PGID=1000
      - ZURG_ENABLED=true
      - ZURG_UPDATE=true
      - RD_API_KEY=redacted
      - RCLONE_MOUNT_NAME=dmb
      - RCLONE_DIR_CACHE_TIME=10s
      - RCLONE_CACHE_DIR=/cache
      - RCLONE_VFS_READ_CHUNK_SIZE=1M
      - RCLONE_VFS_READ_CHUNK_SIZE_LIMIT=32M
      - RIVEN_ENABLED=true
      - RIVEN_UPDATE=true
      - FRONTEND_LOGS=off
      - ORIGIN=https://redacted.proxy.com      
      - PLEX_TOKEN=redacted
      - PLEX_ADDRESS=https://redacted2.proxy.com
      - DUPLICATE_CLEANUP=true
      - CLEANUP_INTERVAL=24
      - SEERR_API_KEY=redacted
      - SEERR_ADDRESS=https://redacted3.proxy.com
      - ZILEAN_ENABLED=true
      - ZILEAN_UPDATE=true
#      - DMB_LOG_LEVEL=info
      - COLOR_LOG_ENABLED=true
    devices:
      - /dev/fuse:/dev/fuse:rwm
    cap_add:
      - SYS_ADMIN     
    security_opt:
      - apparmor:unconfined
      - no-new-privileges
    healthcheck:
      start_period: 1m
      interval: 10s
      retries: 3
    networks:
      macvlan-docker:
        ipv4_address: 192.168.50.245```
profucius commented 2 months ago

Riven's breaking changes have been the bane of existence! I've requested the same. They use Semantic Versioning, as do I; however, Riven has not incremented the MAJOR field on a breaking change.

As for tracking it on the DMB repo, that's a bit difficult to do when no changes are required for the DMB code to support a Riven breaking change - where typically the indicator would be the MAJOR field for the DMB version.

I see, yes that is quite maddening isn't it. I've talked to other devs in the past about versioning, sometimes they can be staunch about it and it makes little sense to me why not just increment a major version, that's what it's for. Well I just want to say I appreciate you keeping things straight for us here on this repo! And thanks a ton for your diagnostics to help me resolve this issue too.

Edit: Just answered my own question about the frontend/backend versions. Found them in the Riven>About page. 👍

NoSync commented 2 months ago

@NoSync I believe you should open a separate ticket for this, it appears to be a different issue at this point.

It was indeed a case of two separate issues caused by the latest riven update, one of them being the one mentioned in this ticket.

On one hand, the settings.json I had was no longer playing nice with the latest riven version. After I removed it, as you recommended, I got the same crashes as per OP when restoring my pgsql data directory, which were solved by zapping the data directory again. In short, should anybody else stumble upon this, to get everything to work nicely with the latest release both settings.json and the DB data had to be initialized from scratch.

I also cleaned up the logging vars in my docker-compose a little, as suggested by @I-am-PUID-0, although they weren't the culprits.

Thanks to both.

profucius commented 2 months ago

Glad you got it working! A cautionary tale to all: Riven is under heavy development. If they anounce major breaking changes, we should be prepared to delete (or use a file diff app to compare) the settings.json file, as well as potentially the whole PostgreSQL database folder if all else goes fubar. Reminder to keep backups of your working config and database. Locking your versions in the docker compose is also a valid option.

Thanks to PUID as always for being patient with us AND with the Riven devs haha. Much love to all.