slush0 / stratum-mining-proxy

Application providing bridge between old HTTP/getwork protocol and Stratum mining protocol.
399 stars 280 forks source link

Extranonce slots are full, please disconnect some miners #16

Open alphaonex86 opened 11 years ago

alphaonex86 commented 11 years ago

Hello, I have: 2013-04-23 15:51:26,901 INFO proxy stratum_listener.submit # [14ms] Share from 'pool' accepted, diff 1 2013-04-23 15:51:27,149 INFO proxy stratum_listener.submit # [14ms] Share from 'pool' accepted, diff 1 2013-04-23 15:51:27,602 INFO proxy stratum_listener.submit # [15ms] Share from 'pool' REJECTED: (21, u'Stale work', None) 2013-04-23 15:51:28,067 INFO proxy stratum_listener.submit # [14ms] Share from 'pool' REJECTED: (21, u'Stale work', None) 2013-04-23 15:51:29,751 INFO proxy stratum_listener.submit # [14ms] Share from 'pool' REJECTED: (21, u'Stale work', None) 2013-04-23 15:51:29,840 INFO proxy stratum_listener.submit # [14ms] Share from 'pool' accepted, diff 1 2013-04-23 15:51:30,085 INFO proxy stratum_listener.submit # [14ms] Share from 'pool' REJECTED: (21, u'Stale work', None) 2013-04-23 15:51:31,243 INFO proxy stratum_listener.submit # [15ms] Share from 'pool' accepted, diff 1 2013-04-23 15:51:32,745 INFO proxy getwork_listener._on_authorized # Worker 'pool' asks for new work 2013-04-23 15:51:33,539 INFO proxy getwork_listener._on_authorized # Worker 'pool' asks for new work 2013-04-23 15:51:35,481 INFO proxy stratum_listener.submit # [14ms] Share from 'pool' accepted, diff 1 2013-04-23 15:51:38,814 INFO proxy stratum_listener.submit # [14ms] Share from 'pool' REJECTED: (21, u'Stale work', None) 2013-04-23 15:51:41,580 INFO proxy jobs.submit # Submitting d6642a20 2013-04-23 15:51:41,595 WARNING proxy getwork_listener._on_submit # [14ms] Share from 'pool' accepted, diff 1 2013-04-23 15:51:43,635 INFO proxy client_service.handle_event # New job 13d8-1a8 for prevhash a8e7003d, clean_jobs=False 2013-04-23 15:51:44,987 INFO proxy stratum_listener.submit # [14ms] Share from 'alphaonex86_ultracopier' accepted, diff 1 2013-04-23 15:51:47,896 ERROR protocol protocol.process_failure # [Failure instance: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! /usr/lib64/python2.7/site-packages/stratum-0.2.12-py2.7.egg/stratum/services.py:81:call /usr/lib64/python2.7/site-packages/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py:137:maybeDeferred /usr/lib64/python2.7/site-packages/stratum-0.2.12-py2.7.egg/stratum/services.py:78:_run /usr/lib64/python2.7/site-packages/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py:1213:unwindGenerator --- --- /usr/lib64/python2.7/site-packages/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py:1070:_inlineCallbacks /home/stratum-mining-proxy/mining_libs/stratum_listener.py:138:subscribe /home/stratum-mining-proxy/mining_libs/stratum_listener.py:108:_get_unused_tail ] None 2013-04-23 15:51:47,934 INFO stats stats.print_stats # 59 peers connected, state changed 3 times 2013-04-23 15:51:47,983 ERROR protocol protocol.process_failure # [Failure instance: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! /usr/lib64/python2.7/site-packages/stratum-0.2.12-py2.7.egg/stratum/services.py:81:call /usr/lib64/python2.7/site-packages/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py:137:maybeDeferred /usr/lib64/python2.7/site-packages/stratum-0.2.12-py2.7.egg/stratum/services.py:78:_run /usr/lib64/python2.7/site-packages/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py:1213:unwindGenerator --- --- /usr/lib64/python2.7/site-packages/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py:1070:_inlineCallbacks /home/stratum-mining-proxy/mining_libs/stratum_listener.py:138:subscribe /home/stratum-mining-proxy/mining_libs/stratum_listener.py:108:_get_unused_tail How fix? Cheer,

alphaonex86 commented 11 years ago

Can you increase the max connected client to 32Bits? Cheer,

slush0 commented 11 years ago

Do you have more than 250 clients connected to Stratum proxy?

alphaonex86 commented 11 years ago

In reallity more than 20000 try use my proxy just now, I target 100 000 into few months.

slush0 commented 11 years ago

Current limit is ~250 connections, but raising this limit is very high in my TODO. I'll keep you updated.

slush0 commented 11 years ago

Btw this is limitation of Stratum sub-connections, you can try to run miners on getwork protocol until I fix it.

alphaonex86 commented 11 years ago

Have you more details for that's? --fix-protocol + http://X.X.X.X:8332 ? I'm very low to bandwith (less than 1% of my server bandwith)

slush0 commented 11 years ago

Yes, it should be enough.

alphaonex86 commented 11 years ago

I have do multiple server instance (200 server) to have a quick and better fix, seam be correct?

slush0 commented 11 years ago

Yes, you can run multiple instances of mining proxy, for example behind haproxy.

alphaonex86 commented 11 years ago

Yes but it's http proxy with load balancing, no stratum proxy.

slush0 commented 11 years ago

haproxy is TCP proxy as well, I'm using it for balancing on the pool.

alphaonex86 commented 11 years ago

I can use haproxy as front end, and cascade of stratum proxy (255 stratum proxy connected on one stratum proxy connected on my worker?) With cgminer 2.11.3 as client, I try know if I need enable: --no-midstate --real-target It's not very clear.

slush0 commented 11 years ago

no-midstate and real-target are used only for getwork. If you connects cgminer to stratum proxy with stratum protocol, you don't need to worry about this.

alphaonex86 commented 11 years ago

With casade for stratum proxy, it not work. When do you think it's solved? I lost 6x of my income...

alphaonex86 commented 11 years ago

https://github.com/invl/stratum-mining-proxy It's a good start I think to have 32Bits, no?

soundasleep commented 11 years ago

This is occuring right now for me on the WeMineLTC modified proxy. I'm only using three clients on a LAN (the proxy is providing getwork to give-me-ltc stratum), it's occuring when I connect to stratum port. Connecting to getwork port works fine. There aren't that many active ports (netstat displays 5 on :getwork, 0 on :stratum).

markudevelop commented 11 years ago

Got an update on this one? I got the same issue?

alphaonex86 commented 11 years ago

slush0: haproxy loop (connect, disconnect) each second to ping the tcp server. That's reveal a bug with your stratum proxy: more and more shares is rejected, after a 24, near 90% is rejected (mostly bellow difficulty) until I restart the server.

markudevelop commented 11 years ago

@alphaonex86 hey had any luck fixing this? Using the invl stratum proxy it should be around up to 65k workers for stratum but I get issues after 2k peers not sure if it's server fault or script. Did you get something working?

alphaonex86 commented 11 years ago

Same, after 1k of peers I have issue and I need restart by cron each 6h... but the code seam ok. In my case other service (apache, lighttpd) have 20k peers without problem (sysctl linux tuned).

haoyixin commented 11 years ago

i have arranged it by haproxy for more then 40 instances of stratum proxy for loadbalance,it could hold more then 10000 clients on a server with 2 cores and 2GB memory.

haoyixin commented 11 years ago

As GIL working,one instance only works on one core,so many instances will work more effectively.

propostaff commented 11 years ago

Any news about increase maximum connection to strtum proxy?

sterion75 commented 10 years ago

I'm getting this issue on the latest proxy even though it's supposed to handle more then 250. I have less than 250 miners but it seems stale connections might not be removed maybe?

slush0 commented 10 years ago

sterion75: Proxy prints number of currently connected peers to console. Can you please provide such log, so I can identify the problem? In my laboratory tests proxy succesfully handled hundreds of live stratum connections.

sterion75 commented 10 years ago

It only does that when the proxy first starts. It normally will say around 112. I don't see it mention amount of connections at any other point? Am I able to log this some how?

sterion75 commented 10 years ago

changed 1 times 2014-01-24 16:22:02,960 INFO stats stats.print_stats # 3 peers connected, state changed 1 times 2014-01-24 16:22:02,960 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:02,976 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:02,976 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:02,976 INFO stats stats.print_stats # 4 peers connected, state changed 1 times 2014-01-24 16:22:02,976 INFO stats stats.print_stats # 3 peers connected, state changed 1 times 2014-01-24 16:22:02,976 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:02,990 INFO stats stats.print_stats # 2 peers connected, state changed 1 times 2014-01-24 16:22:02,990 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:02,990 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,006 INFO stats stats.print_stats # 1 peers connected, state changed 1 times 2014-01-24 16:22:03,006 INFO stats stats.print_stats # 2 peers connected, state changed 1 times 2014-01-24 16:22:03,023 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,038 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,038 INFO stats stats.print_stats # 1 peers connected, state changed 1 times 2014-01-24 16:22:03,131 INFO stats stats.print_stats # 2 peers connected, state changed 1 times 2014-01-24 16:22:03,147 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,147 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,147 INFO stats stats.print_stats # 1 peers connected, state changed 1 times 2014-01-24 16:22:03,194 INFO stats stats.print_stats # 2 peers connected, state changed 1 times 2014-01-24 16:22:03,210 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,210 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,210 INFO stats stats.print_stats # 1 peers connected, state changed 1 times 2014-01-24 16:22:03,302 INFO stats stats.print_stats # 2 peers connected, state changed 1 times 2014-01-24 16:22:03,319 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,319 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,335 INFO stats stats.print_stats # 1 peers connected, state changed 1 times 2014-01-24 16:22:03,490 INFO stats stats.print_stats # 2 peers connected, state changed 1 times 2014-01-24 16:22:03,506 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,522 INFO stats stats.print_stats # 3 peers connected, state changed 1 times 2014-01-24 16:22:03,522 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,522 INFO stats stats.print_stats # 2 peers connected, state changed 1 times 2014-01-24 16:22:03,522 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,536 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,536 INFO stats stats.print_stats # 1 peers connected, state changed 1 times 2014-01-24 16:22:03,584 INFO stats stats.print_stats # 2 peers connected, state changed 1 times 2014-01-24 16:22:03,599 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,599 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,599 INFO stats stats.print_stats # 1 peers connected, state changed 1 times

slush0 commented 10 years ago

Hm, proxy thinks there're only two peers connected, so this really looks like a bug. I'm reading the code again and again and I don't see any edge case when this should appear...

Output of modabekov's command would be useful to see how many connections are real...

stephenc01 commented 10 years ago

I use haproxy infront of multiple instances of the stratum proxy to support high loads

slush0 commented 10 years ago

Yes, haproxy works nicely, but looks that @sterion75 hit some weird bug in proxy and I'd like to know what's wrong there.

sterion75 commented 10 years ago

I have a hunch it happens when the upstream pool we are connecting to is busy. As the pool has been quiet the last week or two the problem hasn't been occuring but it has occured on other pools when they have been busy. I know exactly how many miners are using the proxy as it is behind a firewall and only the machines I have set up use it :)

alphaonex86 commented 10 years ago

@slush0: Same bug here with and without haproxy.