bitmagnet-io / bitmagnet

A self-hosted BitTorrent indexer, DHT crawler, content classifier and torrent search engine with web UI, GraphQL API and Servarr stack integration.
https://bitmagnet.io/
MIT License
2.38k stars 94 forks source link

error persisting torrents: extended protocol limited to 65535 parameters #118

Closed kde99 closed 8 months ago

kde99 commented 8 months ago

Please do not open issues asking for general support with setting up the app and configuring Docker, as such requests are too time consuming to handle. All the information you need to get up and running is in the Bitmagnet docs, the Docker docs and on Google. If you'd like to suggest improvements or corrections to the docs then please open a PR. If you think you've found a bug then please help us out by letting us know. :heart:

Have you checked the roadmap on the website, and existing issues, before opening a dupllcate issue? Yes

Describe the bug During normal crawling you get error persisting torrents: extended protocol limited to 65535 parameters every few minutes, and the torrents listed won't show up when searched in the web UI.

To Reproduce Steps to reproduce the behavior:

  1. run the project in docker (docker latest or git HEAD) both produce this error.

Expected behavior Persist torrents.

General (please complete the following information):

Additional context

bitmagnet           | ERROR     gorm    gorm/logger.go:72       gorm trace      {"location": "/build/internal/database/dao/torrents.gen.go:691", "error": "extended protocol limited to 65535 parameters", "elapsed": 194.518714, "sql": "INSERT INTO \"torrents\" (\"info_hash\",\"name\",\"size\",\"private\",\"piece_length\",\"pieces\",\"created_at\",\"updated_at\",\"files_status\") VALUES ('<binary>','NAME',2242728,false,NULL,'','2024-01-25 20:57:20.868','2024-01-25 20:57:20.868','multi'), *** more stuff like before *** ON CONFLICT (\"info_hash\") DO UPDATE SET \"name\"=\"excluded\".\"name\",\"files_status\"=\"excluded\".\"files_status\",\"piece_length\"=\"excluded\".\"piece_length\",\"pieces\"=\"excluded\".\"pieces\"", "rows": 100}
bitmagnet           | github.com/bitmagnet-io/bitmagnet/internal/database/gorm.(*customLogger).Trace
bitmagnet           |   /build/internal/database/gorm/logger.go:72
bitmagnet           | gorm.io/gorm.(*processor).Execute
bitmagnet           |   /go/pkg/mod/gorm.io/gorm@v1.25.5/callbacks.go:134
bitmagnet           | gorm.io/gorm.(*DB).CreateInBatches.func1
bitmagnet           |   /go/pkg/mod/gorm.io/gorm@v1.25.5/finisher_api.go:48
bitmagnet           | gorm.io/gorm.(*DB).Transaction
bitmagnet           |   /go/pkg/mod/gorm.io/gorm@v1.25.5/finisher_api.go:647
bitmagnet           | gorm.io/gorm.(*DB).CreateInBatches
bitmagnet           |   /go/pkg/mod/gorm.io/gorm@v1.25.5/finisher_api.go:60
bitmagnet           | gorm.io/gen.(*DO).CreateInBatches
bitmagnet           |   /go/pkg/mod/gorm.io/gen@v0.3.25/do.go:598
bitmagnet           | github.com/bitmagnet-io/bitmagnet/internal/database/dao.torrentDo.CreateInBatches
bitmagnet           |   /build/internal/database/dao/torrents.gen.go:691
bitmagnet           | github.com/bitmagnet-io/bitmagnet/internal/dhtcrawler.(*crawler).runPersistTorrents
bitmagnet           |   /build/internal/dhtcrawler/persist.go:43
bitmagnet           | ERROR     dht_crawler     dhtcrawler/persist.go:44        error persisting torrents: extended protocol limited to 65535 parameters
bitmagnet           | github.com/bitmagnet-io/bitmagnet/internal/dhtcrawler.(*crawler).runPersistTorrents
bitmagnet           |   /build/internal/dhtcrawler/persist.go:44
mgdigital commented 8 months ago

Thanks @kde99 , I'm hoping this is fixed by https://github.com/bitmagnet-io/bitmagnet/pull/121

I'm a bit confused why you're seeing this as it's not something I see but reducing the batch size for persistence should sort it - let me know how you get on with 0.5.0 which will be released imminently.

BTW it isn't recommended to run from the main branch as that isn't guaranteed to be stable....