Gorm and queue errors - Githubissues

Alexandrsv commented 4 months ago

[x] I have checked the existing issues to avoid duplicates
[x] I have redacted any info hashes and content metadata from any logs or screenshots attached to this issue

Describe the bug

After 1,010,731 new torrent distributions are not added

To Reproduce

Steps to reproduce the behavior:

docker compose up -d

Details

``` services: bitmagnet: image: ghcr.io/bitmagnet-io/bitmagnet:latest container_name: bitmagnet ports: # API and WebUI port: - "3333:3333" # BitTorrent ports: - "3334:3334/tcp" - "3334:3334/udp" restart: unless-stopped environment: - POSTGRES_HOST=postgres - POSTGRES_PASSWORD=123 command: - worker - run - --keys=http_server - --keys=queue_server # disable the next line to run without DHT crawler - --keys=dht_crawler depends_on: postgres: condition: service_healthy postgres: image: postgres:16-alpine container_name: bitmagnet-postgres volumes: - ./data/postgres:/var/lib/postgresql/data ports: - "55323:5432" restart: unless-stopped environment: - POSTGRES_PASSWORD=123 - POSTGRES_DB=bitmagnet - PGUSER=postgres shm_size: 1g healthcheck: test: - CMD-SHELL - pg_isready start_period: 20s interval: 10s ```

Expected behavior

A clear and concise description of what you expected to happen.

Environment Information (Required)

Bitmagnet version: v0.9.3
OS and version: ubuntu docker
Browser and version (if issue is with WebUI): [e.g. Chrome 117.0.5938.149]
Please specify any config values for which you have overridden the defaults: (e.g. dht_crawler.save_pieces: true)

Additional context

logs

``` bitmagnet | ERROR queue server/server.go:230 error handling joberrorERROR: duplicate key value violates unique constraint "queue_jobs_fingerprint_status_idx" (SQLSTATE 23505) {"queue": "process_torrent"} bitmagnet | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob bitmagnet | /src/internal/queue/server/server.go:230 bitmagnet | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).start.func2 bitmagnet | /src/internal/queue/server/server.go:169 bitmagnet | ERROR queue server/server.go:214 job failed {"queue": "process_torrent", "error": "job failed to process: runtime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused"} bitmagnet | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob.func1 bitmagnet | /src/internal/queue/server/server.go:214 bitmagnet | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob.(*Query).Transaction.func2 bitmagnet | /src/internal/database/dao/gen.go:196 bitmagnet | gorm.io/gorm.(*DB).Transaction bitmagnet | /go/pkg/mod/gorm.io/gorm@v1.25.10/finisher_api.go:651 bitmagnet | github.com/bitmagnet-io/bitmagnet/internal/database/dao.(*Query).Transaction bitmagnet | /src/internal/database/dao/gen.go:196 bitmagnet | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob bitmagnet | /src/internal/queue/server/server.go:180 bitmagnet | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).start.func2 bitmagnet | /src/internal/queue/server/server.go:169 bitmagnet | ERROR gorm logger/logger.go:72 gorm trace {"location": "/src/internal/queue/server/server.go:226", "error": "ERROR: duplicate key value violates unique constraint \"queue_jobs_fingerprint_status_idx\" (SQLSTATE 23505)", "elapsed": 17.81044, "sql": "UPDATE \"queue_jobs\" SET \"status\"='retry',\"run_after\"='2024-06-22 18:50:06.889',\"ran_at\"='2024-06-22 18:49:50.888',\"error\"='job failed to process: runtime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused' WHERE \"id\" = '79d22423-65cd-****'", "rows": 0} bitmagnet | github.com/bitmagnet-io/bitmagnet/internal/database/logger.(*customLogger).Trace bitmagnet | /src/internal/database/logger/logger.go:72 bitmagnet | gorm.io/gorm.(*processor).Execute bitmagnet | /go/pkg/mod/gorm.io/gorm@v1.25.10/callbacks.go:134 bitmagnet | gorm.io/gorm.(*DB).Updates bitmagnet | /go/pkg/mod/gorm.io/gorm@v1.25.10/finisher_api.go:411 bitmagnet | gorm.io/gen.(*DO).Updates bitmagnet | /go/pkg/mod/gorm.io/gen@v0.3.26/do.go:728 bitmagnet | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob.func1 bitmagnet | /src/internal/queue/server/server.go:226 bitmagnet | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob.(*Query).Transaction.func2 bitmagnet | /src/internal/database/dao/gen.go:196 bitmagnet | gorm.io/gorm.(*DB).Transaction bitmagnet | /go/pkg/mod/gorm.io/gorm@v1.25.10/finisher_api.go:651 bitmagnet | github.com/bitmagnet-io/bitmagnet/internal/database/dao.(*Query).Transaction bitmagnet | /src/internal/database/dao/gen.go:196 bitmagnet | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob bitmagnet | /src/internal/queue/server/server.go:180 bitmagnet | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).start.func2 bitmagnet | /src/internal/queue/server/server.go:169 bitmagnet | ERROR queue server/server.go:230 error handling joberrorERROR: duplicate key value violates unique constraint "queue_jobs_fingerprint_status_idx" (SQLSTATE 23505) {"queue": "process_torrent"} bitmagnet | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob bitmagnet | /src/internal/queue/server/server.go:230 bitmagnet | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).start.func2 bitmagnet | /src/internal/queue/server/server.go:169 bitmagnet | ERROR queue server/server.go:214 job failed {"queue": "process_torrent", "error": "job failed to process: runtime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused"} bitmagnet | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob.func1 bitmagnet | /src/internal/queue/server/server.go:214 bitmagnet | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob.(*Query).Transaction.func2 bitmagnet | /src/internal/database/dao/gen.go:196 bitmagnet | gorm.io/gorm.(*DB).Transaction bitmagnet | /go/pkg/mod/gorm.io/gorm@v1.25.10/finisher_api.go:651 bitmagnet | github.com/bitmagnet-io/bitmagnet/internal/database/dao.(*Query).Transaction bitmagnet | /src/internal/database/dao/gen.go:196 bitmagnet | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob bitmagnet | /src/internal/queue/server/server.go:180 bitmagnet | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).start.func2 bitmagnet | /src/internal/queue/server/server.go:169 ```

scratchmex commented 3 months ago

in my case this is the trace. it isn't related to the number of torrent, I have 7 mil.

log

``` ERROR queue server/server.go:214 job failed {"queue": "process_torrent", "error": "job failed to process: missing 1 info hashes"} github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob.func1 /src/internal/queue/server/server.go:214 github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob.(*Query).Transaction.func2 /src/internal/database/dao/gen.go:196 gorm.io/gorm.(*DB).Transaction /go/pkg/mod/gorm.io/gorm@v1.25.10/finisher_api.go:651 github.com/bitmagnet-io/bitmagnet/internal/database/dao.(*Query).Transaction /src/internal/database/dao/gen.go:196 github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob /src/internal/queue/server/server.go:180 github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).start.func2 /src/internal/queue/server/server.go:169 ERROR gorm logger/logger.go:72 gorm trace {"location": "/src/internal/queue/server/server.go:226", "error": "ERROR: duplicate key value violates unique constraint \"queue_jobs_fingerprint_status_idx\" (SQLSTATE 23505)", "elapsed": 9.325193, "sql": "UPDATE \"queue_jobs\" SET \"status\"='retry',\"run_after\"='2024-07-17 13:41:41.372',\"ran_at\"='2024-07-17 13:41:25.372',\"error\"='job failed to process: missing 1 info hashes' WHERE \"id\" = 'faa8de3d-7920-4760-8f58-988d3383cd01'", "rows": 0} github.com/bitmagnet-io/bitmagnet/internal/database/logger.(*customLogger).Trace /src/internal/database/logger/logger.go:72 gorm.io/gorm.(*processor).Execute /go/pkg/mod/gorm.io/gorm@v1.25.10/callbacks.go:134 gorm.io/gorm.(*DB).Updates /go/pkg/mod/gorm.io/gorm@v1.25.10/finisher_api.go:411 gorm.io/gen.(*DO).Updates /go/pkg/mod/gorm.io/gen@v0.3.26/do.go:728 github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob.func1 /src/internal/queue/server/server.go:226 github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob.(*Query).Transaction.func2 /src/internal/database/dao/gen.go:196 gorm.io/gorm.(*DB).Transaction /go/pkg/mod/gorm.io/gorm@v1.25.10/finisher_api.go:651 github.com/bitmagnet-io/bitmagnet/internal/database/dao.(*Query).Transaction /src/internal/database/dao/gen.go:196 github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob /src/internal/queue/server/server.go:180 github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).start.func2 /src/internal/queue/server/server.go:169 ERROR queue server/server.go:230 error handling joberrorERROR: duplicate key value violates unique constraint "queue_jobs_fingerprint_status_idx" (SQLSTATE 23505) {"queue": "process_torrent"} github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob /src/internal/queue/server/server.go:230 github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).start.func2 /src/internal/queue/server/server.go:169 ```

this part of the db schema is causing the integrity error https://github.com/bitmagnet-io/bitmagnet/blob/6cf825b907f0905100f5f9c5fd18943b26c0833b/migrations/00012_queue.sql#L32

my workaround was to delete the jobs in retry status

delete from public.queue_jobs
where status = 'retry'

blackcaldera commented 2 months ago

this worked a treat! thanks

Wofflo commented 1 month ago

in my case this is the trace. it isn't related to the number of torrent, I have 7 mil.

log this part of the db schema is causing the integrity error

https://github.com/bitmagnet-io/bitmagnet/blob/6cf825b907f0905100f5f9c5fd18943b26c0833b/migrations/00012_queue.sql#L32

my workaround was to delete the jobs in retry status
delete from public.queue_jobs
where status = 'retry'

Could you let me know where you added this line of code? I'm also getting the same error. Do you put it in the compose file?

blackcaldera commented 1 month ago

Install something like pgAdmin and run it as a script on the database

Wofflo commented 1 month ago

Install something like pgAdmin and run it as a script on the database

Thank you! I shall try this tonight after work.

mgdigital commented 1 month ago

Please note this should not be logged as an error, the error log can be ignored and will be silenced in due course. No querying of the database is needed to remedy this - best to just ignore the error until it is silenced in a future version.

Wofflo commented 1 month ago

I eventually managed to get pgAdmin installed on docker and configured (took some time) and managed to run the script. Now after restarting the bitmagnet container stack it's all working again and loading new results :-D thanks for the help everyone!

mgdigital commented 1 month ago

Hey, okay so to put it a different way - any issue you guys are having is unrelated to the Gorm duplicate key error. If you're having to clear out the queue then yes it will allow new items to appear but at the expense of old ones not being indexed properly. Could I check - out of people having this issue, does anyone have TMDB enabled with a personal API key? If you haven't configured anything for TMDB you will be using the default rate limited API key and the queue will run a lot slower. If you set a personal API key (or disable TMDB) you should find the queue will quickly work back down to zero and you'll start seeing new items. The rate limiting "shouldn''t" cause the queue to back up like this, it seems to work okay for most users, I'm just wary of opening the flood gates and getting the default API key banned which would break it for everyone who hasn't got a personal key. An alternative would be to disable TMDB integration altogether for users without a personal API key...

Wofflo commented 1 month ago

I have a personal api key in use generated from my profile page on TMDB. Is this the right place to get one or do you need to be on the developer side of TMDB to obtain it?

mgdigital commented 1 month ago

I have a personal api key in use generated from my profile page on TMDB. Is this the right place to get one or do you need to be on the developer side of TMDB to obtain it?

You can get a key by following the guide here: https://kb.synology.com/en-au/DSM/tutorial/How_to_apply_for_a_personal_API_key_to_get_video_info

You'll know it's been set properly if you don't see this warning on startup:

you are using the default TMDB api key; TMDB requests will be limited to 1 per second; to remove this warning please configure a personal TMDB api key

You can also run bitmagnet config show to check it's been set.

The other possibility is that TMDB is blocked in your country (I suspect from @Alexandrsv 's username they might be in Russia where it is blocked). If this is the case you'd need to disable TMDB altogether with TMDB_ENABLED=false.

It would be helpful to know what the error is on the retry jobs - you can find it in the error column of the queue_jobs table (but you already cleared everything out) - if this happens again please let me know what the error was.

mgdigital commented 4 weeks ago

The duplicate key error should be fixed in https://github.com/bitmagnet-io/bitmagnet/pull/319 I think your queue was running slow due to an issue with TMDB - try either setting a personal API key or disabling TMDB

bitmagnet-io / bitmagnet

Gorm and queue errors #285

Describe the bug

To Reproduce

Expected behavior

Environment Information (Required)

Additional context