search

spesmilo / electrumx

Alternative implementation of spesmilo/electrum-server
MIT License
438 stars 349 forks source link

My (public) ElectrumX server became too popular, and now it often throttles and doesn't let me connect! #129

Open maltokyo opened 3 years ago

maltokyo commented 3 years ago

I have 48GB RAM, and lots of it allocated to ElectrumX (as per the recommended max in the docs), so resources are not an issue. I also have 1000 sessions limit. Often 700 users or more connected. This seems fine as far as server load is concerned, but the throttling begins frequently anyway.

How can I either:

Please give any advice you might have, otherwise, I will just make it all private and only for me, but I would like to avoid that, as long as I have spare server capacity to burn for the good of the network.

SomberNight commented 3 years ago

fine as far as server load is concerned, but the throttling begins frequently anyway.

What kind of throttling do you mean/experience?

maltokyo commented 3 years ago

the log says something like "10 concurrents, throttling to 0", or something to this meaning, and when it starts doing that I cannot connect at all, or sporadically. Now it is slower, so I can connect and cant see these log entries, when I find it again I will paste it here.

SomberNight commented 3 years ago

How does your client connect to your server? What incoming IP does the server see for the client?

maltokyo commented 3 years ago

The client is the standard Electrum wallet. The server log sees/shows my public IP address (the server is in a different location). I use SSL port 50002 only (the server does not listen on 50001 or other ports).

maltokyo commented 3 years ago

So it is happening now, cannot connect at all. Even after a restart which is strange.

Log attached as a screenshot. Sorry only have mobile.

Kindly let me know what I could get to resolve this. Ill soon get so frustrated that I'll just make it private over tor.

Thank you!!

IMG_20210503_025720.jpg

SomberNight commented 3 years ago

Could you test with disabled resource usage limits?

COST_SOFT_LIMIT=0
COST_HARD_LIMIT=0

(see https://electrumx-spesmilo.readthedocs.io/en/latest/environment.html#resource-usage-limits for more info)

maltokyo commented 3 years ago

Thank you, trying now

maltokyo commented 3 years ago

Runs very smoothly now, even with 1000 connections. Looks good.

maltokyo commented 3 years ago

Everything is very stable now. Between 500-1000 connections being served without any hitch, for days now. Perhaps for servers with a lot of resource, a setting like the above would be helpful to be more prominent in the docs (in terns of exactly what it is doing)

SomberNight commented 3 years ago

Well it already has its own section in the docs; that's what I linked to. Or do you mean specifically this COST_SOFT_LIMIT=0; COST_HARD_LIMIT=0 should be mentioned?

Btw I think maybe what might have been the culprit is the default for SESSION_GROUP_BY_SUBNET_IPV4. It defaults to /24; maybe it should be changed to /32. This could explain why your IP got throttled.

maltokyo commented 3 years ago

Well it already has its own section in the docs; that's what I linked to. Or do you mean specifically this COST_SOFT_LIMIT=0; COST_HARD_LIMIT=0 should be mentioned?

Yes - was mentioning to add a comment something like "set these to 0 in case you see 'disconnected whilst throttled' message in the log when you think that you have enough server power and it should not be throttled under any conditions".
At least this way, a solution would be available to people using Google to search for the error text. Just my 2 sats.

Btw I think maybe what might have been the culprit is the default for SESSION_GROUP_BY_SUBNET_IPV4. It defaults to /24; maybe it should be changed to /32. This could explain why your IP got throttled.

That could be true too - because it is running in docker, on its own IP subnet, so the requests from the internet or the data center network might be treated differently.