The import and database operations have become extremely slow.

[peterzam]

• [x] I have checked the existing issues to avoid duplicates
• [x] I have redacted any info hashes and content metadata from any logs or screenshots attached to this issue

Describe the bug

I am importing 30 million torrents through the /import endpoint, and the database and queue becomes quite slow for the later parts of the process. The import process slows down exponentially as it goes on.

To Reproduce

Steps to reproduce the behavior: Import.

Expected behavior

A clear and concise description of what you expected to happen.

Environment Information (Required)

• Bitmagnet version: 0.8.0
• OS and version: Linux/Debian12
• Please specify any config values for which you have overridden the defaults: -
• Device specifications: 8 threads, 2.3-3.6 GHz processor, 16 GB of memory, and 300 GB of storage.

Additional context

I think the unique property of the info_hash is slowing down the process.

[peterzam]

Database inserts also slow when merging a large number of queries. May be the culprit is checking unique for info_hash.