100 torrents an hour / what is considered normal operation?

[obfuscate27]

I’m trying to understand the behavior I’ve gotten from magnetico.

From the very start it has only ever sent about 300Kb/s and received around 500Kb/s. And at first it was finding on average somewhere between 500-600 torrents an hour.

I accepted this as working normally and then left it alone.

Just under 2 days later something happened and I see that around 8am the second morning the number of torrents found just suddenly dropped to 100 per hour. Since then, no matter what I do, I’m only finding about 100 torrents an hour yet the data rates stay the same.

My setup: Dedicated server running ESXI 6.7. Ubuntu 18.04 server in a vm that has access to all 4 cores @ 3.2ghz, 12GB ram and 100GB hdd space. Internet speed is 1Gbps. At first I was running it through a pfsense vm and an openvpn connection while specifying a port being open.

The very first time I tried to run it I got some errors about something being invalid. That was a quick fix to look up and after starting it like this:

./magneticod --indexer-addr=xxx.xxx.xxx.xxx:xxxxx

and adding an open port to my pfsense config those errors went away.

Assuming everything was ok now I then left it alone for a couple of days only to now be stuck with the 100 torrents an hour problem.

The first thing I tried to fix the problem was to reset everything in a new vm instance of ubuntu and see if that would make things go back to finding 600 torrents an hour again. I would have left it running for another couple of days after doing this but very quickly noticed that it seems permanently stuck at only finding 100 torrents an hour now.

The second thing I tried was running it on a raw IP with no firewall at all to see if that was the problem. Still get 100 torrents an hour though.

CPU usage barely goes above 10% and ram usage stays around the 2GB mark.

Is this normal behavior? Why did I initially get 600 torrents an hour found and now no matter what I do I can’t get above 100? Was 600 normal to begin with? Was that just some sort of fluke and 100 is the real normal?

I read in quite a few places here that magnetico is meant to suck up bandwidth but that has never happened. Has this been changed in recent versions as those threads discussing it seemed to have been made around 2017?

I’m trying to understand what’s going on and what should be normal operation these days.

Any insight appreciated.

[kescherCode]

It used to suck up bandwidth here, but has slowed down here over time, too.

[Glandos]

I used to have a nice ratio, around 0.1 new torrent per second. Now, it's much more like 0.002 torrent per second. I suspect that some clients are banning this kind of behavior. Or some network middlebox. Or, to be optimistic, our databases are near the maximum size :)

Here is my database evolution. Giant steps are merges, so they're not relevant, but you can see that slopes are flattening:

[kescherCode]

@Glandos With an SQLite database larger than 40 GiB, yes, we might be at a point where the database itself has trouble growing further.

[Glandos]

Not for now: magneticod is eating 16% of one core. And it's a small one. The database is on an SSD, so there's also no issue with I/O.

[kescherCode]

Yes, I do also see this trend, but with magneticod sitting at 100% of one CPU core (due to being limited to one core), and barely any writes. Furthermore, my database lies on an HDD.

[DyonR]

I used to have a nice ratio, around 0.1 new torrent per second. Now, it's much more like 0.002 torrent per second. I suspect that some clients are banning this kind of behavior. Or some network middlebox. Or, to be optimistic, our databases are near the maximum size :)

Here is my database evolution. Giant steps are merges, so they're not relevant, but you can see that slopes are flattening: [image removed to preserve space]

I know this question might be slightly off-topic, but is there a guide on how to merge two magnetico databases together easily? I tried some tutorials I found after a quick Google search but they didn't really work for me.

On-topic: My Magnetico discovers about 4k-6k torrents/day.
In the past I used to get more than 20k per day, but for some reason it seems to matter what IP I have...
My Magnetico runs on a VPS, but due migrations and IP-range changes my torrent discovery/day varied a lot after a change.
In January 2020 I discovered 868k torrents in just one month, which is about 28k/day, but later on, something changed with my IP-adres and speed went down.

[kescherCode]

@DyonR to answer your question about merging: @Glandos created a tool for that, available at https://framagit.org/Glandos/magnetico_merge

For me, the reduction in speed happened with the same IP as I had when I started. I do believe that DHT peers are starting to ignore traffic they deem malicious.

[DyonR]

I changed my magneticod settings from the default leech-max-n=50 and indexer-max-neighbors=1000 to:
leech-max-n=190 indexer-max-neighbors=7500
Previously I got 5k torrents on average per day, but now I get 35k-40k because of this change!
The higher the indexer-max-neighbors is set, the more torrents you will find it seems.
The increase of max-neighbors is increased by 7.5, which also did result in 7.5 times more torrents!
I couldn't find any documentation about what these settings exactly do, but it seems to work pretty well if you increase it.

[kescherCode]

@DyonR mine are at --leech-max-n=500 --indexer-max-neighbors=10000

so yes, initially that helps, but it dies down later nonetheless.

[HxBreak]

Is this normally? How can i fix it if something wrong?