Failed syncing database: ConnectionError [SequelizeConnectionError]: database "torrentio" does not exist

[thoohtje]

error.txt Describe the bug Getting error when running docker compose up -d, not starting the scrape.

To Reproduce Running Just started giving this error out of nowhere. What i tried so far docker kill $(docker ps -aq) docker rm $(docker ps -aq) docker rmi -f $(docker images -a -q) docker system prune removing the whole folder from server git clone https://github.com/Gabisonfire/torrentio-scraper-sh.git (running docker compose up -d en watching the logs of the scraper)

Expected behavior Starting the scrape: scraper-1 | Scraper started scraper-1 | [Fri Feb 02 2024 15:20:51 GMT+0000] starting YTS scrape... scraper-1 | Scrapping YTS page 1

Logs See error.txt

Hardware:

• OS ubuntu 22.04
• Server: OpenVZ with AMD EPYC™ 7443P - 4.00 GHZ

Additional context Worked perfectly in the morning (and all the other days i had it running) but now its not working anymore. Cloned https://github.com/kubo6472/torrentio-ghcr and with this built it is working perfectly: scraper-1 | Scraper started scraper-1 | [Fri Feb 02 2024 15:20:51 GMT+0000] starting YTS scrape... scraper-1 | Scrapping YTS page 1 etc etc etc

Thank you already!

[Gabisonfire]

Please post your Postgres logs as well

[thoohtje]

Thanks for responding so fast! Can you tell where to find them? (Sorry :) I am kinda new in this)

[purple-emily]

type docker compose logs postgres into the terminal when you are inside the folder for the project.

[iPromKnight]

Sounds like you could be missing an env var

ENABLE_SYNC=true on the consumer will create the database if it doesn't exist as long as the connection string is correct, as it validates schema on start

[purple-emily]

If this is true can we remove this var and assume its always true?

is there any benefit to setting this to false?

[iPromKnight]

It's a slightly faster startup of the container but I guess it doesn't matter. Just the consumer. Yeah we can remove it

[thoohtje]

I don't have a env folder. Could that be the problem?

[purple-emily]

You are on the pre-rewrite version

[thoohtje]

@purple-emily Log: postgres-1 | PostgreSQL Database directory appears to contain a database; Skipping initialization postgres-1 | postgres-1 | 2024-02-02 15:19:44.319 UTC [1] LOG: starting PostgreSQL 14.10 on x86_64-pc-linux-musl, compiled by gcc (Alpine 13.2.1_git20231014) 13.2.1 20231014, 64-bit postgres-1 | 2024-02-02 15:19:44.319 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 postgres-1 | 2024-02-02 15:19:44.319 UTC [1] LOG: listening on IPv6 address "::", port 5432 postgres-1 | 2024-02-02 15:19:44.323 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" postgres-1 | 2024-02-02 15:19:44.329 UTC [21] LOG: database system was shut down at 2024-02-02 15:05:34 UTC postgres-1 | 2024-02-02 15:19:44.341 UTC [1] LOG: database system is ready to accept connections postgres-1 | 2024-02-02 15:19:46.793 UTC [28] FATAL: database "torrentio" does not exist postgres-1 | 2024-02-02 15:19:49.084 UTC [29] FATAL: database "torrentio" does not exist postgres-1 | 2024-02-02 15:19:51.402 UTC [30] FATAL: database "torrentio" does not exist postgres-1 | 2024-02-02 15:19:53.802 UTC [31] FATAL: database "torrentio" does not exist postgres-1 | 2024-02-02 15:19:56.551 UTC [32] FATAL: database "torrentio" does not exist postgres-1 | 2024-02-02 15:19:59.929 UTC [33] FATAL: database "torrentio" does not exist postgres-1 | 2024-02-02 15:20:05.335 UTC [34] FATAL: database "torrentio" does not exist postgres-1 | 2024-02-02 15:20:14.481 UTC [1] LOG: received fast shutdown request postgres-1 | 2024-02-02 15:20:14.482 UTC [1] LOG: aborting any active transactions postgres-1 | 2024-02-02 15:20:14.488 UTC [1] LOG: background worker "logical replication launcher" (PID 27) exited with exit code 1 postgres-1 | 2024-02-02 15:20:14.491 UTC [22] LOG: shutting down postgres-1 | 2024-02-02 15:20:14.514 UTC [1] LOG: database system is shut down

[purple-emily]

If you haven’t used the project at all. Delete the folder and download/clone again

[thoohtje]

@purple-emily I deleted the folder like twice today, but it didn't change a thing (I even showed my co-worker). I git cloned it again and now i am seeing the env folder XD. Now i know how users feel when they call me and their problem fixes itself while calling me :)

Is it normal that this addon now uses ~4GB of ram?

[thoohtje]

[purple-emily]

@purple-emily I deleted the folder like twice today, but it didn't change a thing (I even showed my co-worker). I git cloned it again and now i am seeing the env folder XD. Now i know how users feel when they call me and their problem fixes itself while calling me :)

Is it normal that this addon now uses ~4GB of ram

It may use more whilst the producer is filling up the message queue and less in between.

4gb sounds like a lot. I was running 60 consumers and using around 6gb of ram. The producer uses the most ram during operation but it’s throttled to not go crazy and fill the queue up with hundreds of thousands of torrents to process. It should fill up a chunk then wait for them to be processed. Rinse and repeat.

[purple-emily]

Can you use top or htop and verify what’s using all the ram?

[thoohtje]

Stopped al the containers and the mem usage is 2.9Gb... I had this problem before. Seems like docker bugging out? I am restarting my server and then rerunning the docker compose -d with the now working docker ;) Will drop feedback asap. Can i close this bug report, while we can keep chatting? (As in i can let the container run for a while to check if i have a mem leak. This will take a while and the original problem is solved. The new problem is not a priority :) )

I also want to say @Gabisonfire @purple-emily @iPromKnight Thank you for helping me out!

[thoohtje]

Mem usage is now at ~1800MB ~3 min running

[purple-emily]

That sounds much more like what I’d expect. Classic turn it off and on again

[purple-emily]

Might be worth updating everything just in case if you’ve had that issue before

sudo apt update && sudo apt upgrade never hurts

[thoohtje]

I will close this bug report, because the orginal problem is solved. Thank you al very much. If i have the same mem problem i will try to reinstall Docker and if i still have the problem. Do you mind if i tag you through this bug / issue report @purple-emily ?

[thoohtje]

btw tried googling the problem for a couple min but couldn't find something to fix it. So offcourse i wil try to google first ;)

[purple-emily]

Sounds good, hope everything runs smoothly!

[thoohtje]

Have a great night / day / weekend!

[purple-emily]

@thoohtje what version of docker you using?

did you install it with snap/flatpack or from package manager?

[thoohtje]

To be fair idk, i think through apt-get. After reinstalling for i third time i didn't even install snapd i see :) Is it better to install docker with snap?

[purple-emily]

No I’d follow the guide on the docker website.

https://docs.docker.com/desktop/install/linux-install/

[thoohtje]

@purple-emily Completly reinstalled docker from scratch using the docker manual: docker engine: Docker desktop: Reinstall freed up about 5GB space ;0 I hope this solves the mem problem but after the install my mem usage is already at 3.7GB:

Rebooting my server and am going to let it run for a couple hours / night. If i still have mem issues and i can't seem to fix them i will comment back here.

BTW does this mean there is 123Mb in use or 123Mb free?

[thoohtje]

@purple-emily Sadly i still have memory problems when using docker ;( Reinstalled the whole server and set everything up from scratch. So only the really needed packages (nano, docker, nginx, ufw, openjdk (for my discord bot, which isn't running) are installed). i am kinda giving up on this sadly (as for now). To many problems (NOT blaming Gabi for this!!) for something i wanted to use as a cheap backup for torrentio. Last thing i am going to try is to install an older version of docker.

Thank you all for the help.

[thoohtje]

Downgraded to docker v24 and the memory issue is gone! But got another problem; database seems te reset after ~12 hours. I also tried to clone the newest release, but the consumer containers are restarting every second. logs says: {"log":"(node:1) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.\n","stream":"stderr","time":"2024-02-04T20:10:20.0638494Z"} {"log":"{\"level\":50,\"time\":1707077420300,\"pid\":1,\"hostname\":\"78720f3ceb23\",\"msg\":\"Failed syncing database: \"}\n","stream":"stdout","time":"2024-02-04T> {"log":"/app/dist/index.cjs:199\n","stream":"stderr","time":"2024-02-04T20:10:20.313934329Z"}

I will hopefully retry this project in a couple of months and see if my/the problems still exist.

[purple-emily]

Can you add these logs to a new issue for someone to look into?

[thoohtje]

@purple-emily Thank you for replying, but i got jackett running through elfhosted and using a public (free) version of torrentio / jackett that someone is hosting on elfhosted.

I probs wil try this project out somewhere in the future. But for now im done trying :)

Thank you again for the help so far.