Closed jotapea closed 2 years ago
You might want to try stopping counterblock entirely and reparse counterblock via : ‘fednode reparse counterblock’
Thank you @jdogresorg, trying it now...
Update on the sync @jdogresorg.
First some context. I am running this on an AWS EC2 instance. The first time I tried 'fednode reparse counterblock' the process was interrupted (CTRL+C) because I realized it was going to take a long time. Then, I tried a couple more times like running it as a background process, but ultimately just decided to run it in the foreground but disabling my computer's sleep to avoid losing connection.
Next day the connection was lost, and the terminal was showing the following error after INFO:blockfeed: Block: 695930 of 742264 [message height=6141080]
:
ERROR:blockfeed: Unhandled exception while processing block. Rolling back, waiting 3 seconds and retrying. Error was: mongodb:27017: [Errno -3] Temporary failure in name resolution
Traceback (most recent call last):
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/pool.py", line 500, in connect
sock = _configured_socket(self.address, self.opts)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/pool.py", line 426, in _configured_socket
sock = _create_connection(address, options)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/pool.py", line 395, in _create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/gevent/_socketcommon.py", line 268, in getaddrinfo
return get_hub().resolver.getaddrinfo(host, port, family, socktype, proto, flags)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/gevent/resolver_thread.py", line 64, in getaddrinfo
return self.pool.apply(_socket.getaddrinfo, args, kwargs)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/gevent/pool.py", line 300, in apply
return self.spawn(func, *args, **kwds).get()
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/gevent/event.py", line 373, in get
return self.get(block=False)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/gevent/event.py", line 363, in get
return self._raise_exception()
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/gevent/event.py", line 343, in _raise_exception
reraise(*self.exc_info)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/gevent/hub.py", line 38, in reraise
raise value.with_traceback(tb)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/gevent/threadpool.py", line 207, in _worker
value = func(*args, **kwargs)
socket.gaierror: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/counterblock/counterblock/lib/blockfeed.py", line 317, in process_cp_blockfeed
result = parse_block(block_data)
File "/counterblock/counterblock/lib/blockfeed.py", line 139, in parse_block
result = parse_message(msg)
File "/counterblock/counterblock/lib/blockfeed.py", line 116, in parse_message
result = function['function'](msg, msg_data) or None
File "/counterblock/counterblock/lib/modules/assets.py", line 625, in parse_balance_change
}, sort=[("block_index", pymongo.DESCENDING), ("_id", pymongo.DESCENDING)])
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/collection.py", line 1006, in find_one
for result in cursor.limit(-1):
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/cursor.py", line 1090, in next
if len(self.__data) or self._refresh():
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/cursor.py", line 1012, in _refresh
self.__read_concern))
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/cursor.py", line 850, in __send_message
**kwargs)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/mongo_client.py", line 798, in _send_message_with_response
exhaust)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/mongo_client.py", line 809, in _reset_on_error
return func(*args, **kwargs)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/server.py", line 88, in send_message_with_response
with self.get_socket(all_credentials, exhaust) as sock_info:
File "/root/.pyenv/versions/3.6.2/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/server.py", line 152, in get_socket
with self.pool.get_socket(all_credentials, checkout) as sock_info:
File "/root/.pyenv/versions/3.6.2/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/pool.py", line 539, in get_socket
sock_info = self._get_socket_no_auth()
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/pool.py", line 577, in _get_socket_no_auth
sock_info = self._check(sock_info)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/pool.py", line 627, in _check
return self.connect()
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/pool.py", line 512, in connect
_raise_connection_failure(self.address, error)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/pool.py", line 65, in _raise_connection_failure
raise AutoReconnect(msg)
pymongo.errors.AutoReconnect: mongodb:27017: [Errno -3] Temporary failure in name resolution
Traceback (most recent call last):
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/pool.py", line 500, in connect
sock = _configured_socket(self.address, self.opts)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/pool.py", line 426, in _configured_socket
sock = _create_connection(address, options)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/pool.py", line 395, in _create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/gevent/_socketcommon.py", line 268, in getaddrinfo
return get_hub().resolver.getaddrinfo(host, port, family, socktype, proto, flags)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/gevent/resolver_thread.py", line 64, in getaddrinfo
return self.pool.apply(_socket.getaddrinfo, args, kwargs)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/gevent/pool.py", line 300, in apply
return self.spawn(func, *args, **kwds).get()
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/gevent/event.py", line 373, in get
return self.get(block=False)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/gevent/event.py", line 363, in get
return self._raise_exception()
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/gevent/event.py", line 343, in _raise_exception
reraise(*self.exc_info)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/gevent/hub.py", line 38, in reraise
raise value.with_traceback(tb)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/gevent/threadpool.py", line 207, in _worker
value = func(*args, **kwargs)
socket.gaierror: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/counterblock/counterblock/lib/blockfeed.py", line 317, in process_cp_blockfeed
result = parse_block(block_data)
File "/counterblock/counterblock/lib/blockfeed.py", line 139, in parse_block
result = parse_message(msg)
File "/counterblock/counterblock/lib/blockfeed.py", line 116, in parse_message
result = function['function'](msg, msg_data) or None
File "/counterblock/counterblock/lib/modules/assets.py", line 625, in parse_balance_change
}, sort=[("block_index", pymongo.DESCENDING), ("_id", pymongo.DESCENDING)])
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/collection.py", line 1006, in find_one
for result in cursor.limit(-1):
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/cursor.py", line 1090, in next
if len(self.__data) or self._refresh():
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/cursor.py", line 1012, in _refresh
self.__read_concern))
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/cursor.py", line 850, in __send_message
**kwargs)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/mongo_client.py", line 798, in _send_message_with_response
exhaust)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/mongo_client.py", line 809, in _reset_on_error
return func(*args, **kwargs)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/server.py", line 88, in send_message_with_response
with self.get_socket(all_credentials, exhaust) as sock_info:
File "/root/.pyenv/versions/3.6.2/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/server.py", line 152, in get_socket
with self.pool.get_socket(all_credentials, checkout) as sock_info:
File "/root/.pyenv/versions/3.6.2/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/pool.py", line 539, in get_socket
sock_info = self._get_socket_no_auth()
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/pool.py", line 577, in _get_socket_no_auth
sock_info = self._check(sock_info)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/pool.py", line 627, in _check
return self.connect()
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/pool.py", line 512, in connect
_raise_connection_failure(self.address, error)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/pool.py", line 65, in _raise_connection_failure
raise AutoReconnect(msg)
pymongo.errors.AutoReconnect: mongodb:27017: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/gevent/greenlet.py", line 534, in run
result = self._run(*self.args, **self.kwargs)
File "/counterblock/counterblock/lib/processor/__init__.py", line 28, in wrapped_f
return func(*args, **kwargs)
File "/counterblock/counterblock/lib/blockfeed.py", line 322, in process_cp_blockfeed
my_latest_block = config.mongo_db.processed_blocks.find_one(sort=[("block_index", pymongo.DESCENDING)])
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/collection.py", line 1006, in find_one
for result in cursor.limit(-1):
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/cursor.py", line 1090, in next
if len(self.__data) or self._refresh():
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/cursor.py", line 1012, in _refresh
self.__read_concern))
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/cursor.py", line 850, in __send_message
**kwargs)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/mongo_client.py", line 781, in _send_message_with_response
server = topology.select_server(selector)
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/topology.py", line 142, in select_server
address))
File "/root/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pymongo/topology.py", line 118, in select_servers
self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: mongodb:27017: [Errno -3] Temporary failure in name resolution
<Greenlet at 0x7f6ed96096d0: wrapped_f> failed with ServerSelectionTimeoutError
After this, the instance was no longer reachable, so I had to reboot it to be able to connect. It seems like the instance had crashed, because bitcoin was out of sync. And I also noticed that for 'fednode ps' it was now showing:
federatednode_counterblock_1 start.sh Up 14100/tcp, 14101/tcp, 14102/tcp, 0.0.0.0:4100->4100/tcp,:::4100->4100/tcp, 4101/tcp,
4102/tcp
federatednode_counterblock_run_4 start.sh Up 14100/tcp, 14101/tcp, 14102/tcp, 4100/tcp, 4101/tcp, 4102/tcp
federatednode_counterblock_run_6 start.sh Up 14100/tcp, 14101/tcp, 14102/tcp, 4100/tcp, 4101/tcp, 4102/tcp
These "_run" services I was unable to stop or kill, as they just reappeared as different processes.
Finally, I waited for everything to sync and then re-run 'fednode reparse counterblock' and it got up to the following, with the error that keeps repeating:
counterblock_1 | INFO:blockfeed: Block: 432550 of 742357 [message height=1995169]
counterblock_1 | WARNING:assets: Credit/debit of 5 where asset ('PEPEMILLION') does not exist. Ignoring...
counterblock_1 | ERROR:blockfeed: Unhandled exception while processing block. Rolling back, waiting 3 seconds and retrying. Error was:
counterblock_1 | Traceback (most recent call last):
counterblock_1 | File "/counterblock/counterblock/lib/blockfeed.py", line 317, in process_cp_blockfeed
counterblock_1 | result = parse_block(block_data)
counterblock_1 | File "/counterblock/counterblock/lib/blockfeed.py", line 139, in parse_block
counterblock_1 | result = parse_message(msg)
counterblock_1 | File "/counterblock/counterblock/lib/blockfeed.py", line 116, in parse_message
counterblock_1 | result = function['function'](msg, msg_data) or None
counterblock_1 | File "/counterblock/counterblock/lib/modules/assets.py", line 531, in parse_issuance
counterblock_1 | assert tracked_asset is not None
counterblock_1 | AssertionError
counterblock_1 | WARNING:database: Pruning to block 432551 ...
counterblock_1 | ERROR:blockfeed: Unhandled exception while processing block. Rolling back, waiting 3 seconds and retrying. Error was:
counterblock_1 | Traceback (most recent call last):
counterblock_1 | File "/counterblock/counterblock/lib/blockfeed.py", line 317, in process_cp_blockfeed
counterblock_1 | result = parse_block(block_data)
counterblock_1 | File "/counterblock/counterblock/lib/blockfeed.py", line 139, in parse_block
counterblock_1 | result = parse_message(msg)
counterblock_1 | File "/counterblock/counterblock/lib/blockfeed.py", line 116, in parse_message
counterblock_1 | result = function['function'](msg, msg_data) or None
counterblock_1 | File "/counterblock/counterblock/lib/modules/assets.py", line 531, in parse_issuance
counterblock_1 | assert tracked_asset is not None
counterblock_1 | AssertionError
counterblock_1 | WARNING:database: Pruning to block 432551 ...
How should I proceed?
Hello, the best thing you can do is to reinstall counterblock or make it start from scratch. To do the latter, follow these steps:
1.- list all containers in your server:
docker ps -a 2.- find the container ids (first column in the list) of "federatednode_counterblock" and "mongodb", with the first 3 characters of each id should be enough 3.- stop counterblock by executing the following (COUNTERBLOCK_ID is the "federatednode_counterblock" id you found in 1.): docker stop COUNTERBLOCK_ID 4.- Execute mongodb console by doing: docker exec -it MONGODB_ID mongo 5.- The console should be running with a "Welcome to the MondoDB shell" text in the first line. Type: show dbs 6.- You should see a list with the counterblock database called "counterblockd". Now type: use counterblockd 7.- "Switched to db counterblockd" should appear. Delete the database executing: db.dropDatabase() 8.- Now exit mongo console by executing: exit 9.- Restart counterblock by executing: docker restart COUNTERBLOCK_ID && docker logs --follow COUNTERBLOCK_ID
With the last command you are restarting counterblock and also following any logs it prints, make sure counterblock starts parsing blocks, parsing should be fast (not like counterparty-lib)
Thank you very much @pataegrillo for the detailed instructions.
I did all your steps (and also removed the "_run" containers), waited for everything up to counterparty to sync, then ran "sudo docker restart COUNTERBLOCK_ID" (sudo required for permissions) and finally ran "sudo docker logs --follow COUNTERBLOCK_ID".
For docker logs I see the following repeating:
WARNING:util: Got call_jsonrpc_api request error: result is None -- is the 'http://counterparty:4000/api/' endpoint operational? -- Waiting 3 seconds before trying again...
WARNING:util: Got call_jsonrpc_api request error: result is None -- is the 'http://counterparty:4000/api/' endpoint operational? -- Waiting 3 seconds before trying again...
WARNING:util: Got call_jsonrpc_api request error: result is None -- is the 'http://counterparty:4000/api/' endpoint operational? -- Waiting 3 seconds before trying again...
WARNING:blockfeed: Cannot contact counterparty-server (via get_running_info): 'NoneType' object is not subscriptable
WARNING:util: Got call_jsonrpc_api request error: result is None -- is the 'http://counterparty:4000/api/' endpoint operational? -- Waiting 3 seconds before trying again...
WARNING:util: Got call_jsonrpc_api request error: result is None -- is the 'http://counterparty:4000/api/' endpoint operational? -- Waiting 3 seconds before trying again...
WARNING:util: Got call_jsonrpc_api request error: result is None -- is the 'http://counterparty:4000/api/' endpoint operational? -- Waiting 3 seconds before trying again...
But for fednode tail counterblock
it seems to be working:
counterblock_1 | INFO:blockfeed: Block: 313390 of 742573 [message height=117411]
counterblock_1 | INFO:blockfeed: Block: 313400 of 742573 [message height=117507]
counterblock_1 | INFO:blockfeed: Block: 313410 of 742573 [message height=117670]
The fednode services I have running are:
Name Command State Ports
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
federatednode_addrindexrs-testnet_1 cargo run --release -- -vvvv Exit 137
federatednode_addrindexrs_1 cargo run --release -- -vvvv Up 0.0.0.0:8432->8432/tcp,:::8432->8432/tcp
federatednode_armory-utxsvr-testnet_1 start.sh Exit 0
federatednode_armory-utxsvr_1 start.sh Exit 0
federatednode_bitcoin-testnet_1 start.sh Exit 137
federatednode_bitcoin_1 start.sh Up 18332/tcp, 18333/tcp, 0.0.0.0:28832->28832/tcp,:::28832->28832/tcp, 0.0.0.0:8332->8332/tcp,:::8332->8332/tcp, 8333/tcp
federatednode_counterblock-testnet_1 start.sh Exit 0
federatednode_counterblock_1 start.sh Up 14100/tcp, 14101/tcp, 14102/tcp, 0.0.0.0:4100->4100/tcp,:::4100->4100/tcp, 4101/tcp, 4102/tcp
federatednode_counterparty-testnet_1 start.sh Exit 0
federatednode_counterparty_1 start.sh Up 14000/tcp, 0.0.0.0:4000->4000/tcp,:::4000->4000/tcp
federatednode_counterwallet_1 start.sh Exit 0
federatednode_mongodb_1 docker-entrypoint.sh mongod Up 127.0.0.1:27017->27017/tcp
federatednode_redis_1 docker-entrypoint.sh redis ... Up 6379/tcp
federatednode_xcp-proxy-testnet_1 docker-entrypoint.sh /bin/ ... Exit 0
federatednode_xcp-proxy_1 docker-entrypoint.sh /bin/ ... Up 0.0.0.0:8097->8097/tcp,:::8097->8097/tcp
And when I try to start counterwallet, it shows the following error:
counterwallet_1 | 2022/06/27 15:26:34 [emerg] 8#0: host not found in upstream "counterblock-testnet:14100" in /etc/nginx/sites-enabled/counterwallet.conf:10
counterwallet_1 | nginx: [emerg] host not found in upstream "counterblock-testnet:14100" in /etc/nginx/sites-enabled/counterwallet.conf:10
counterwallet_1 | 2022/06/27 15:29:36 [emerg] 9#0: host not found in upstream "counterblock-testnet:14100" in /etc/nginx/sites-enabled/counterwallet.conf:10
counterwallet_1 | nginx: [emerg] host not found in upstream "counterblock-testnet:14100" in /etc/nginx/sites-enabled/counterwallet.conf:10
federatednode_counterwallet_1 exited with code 0
Here trying to give as much information as possible to help identify if there is still some issue present, because fednode tail counterblock
does seem to be working.
Normally, you shouldn't use docker with sudo, for that you need to add your user to the docker group in linux. "sudo adduser YOUR_USER docker" and reconnect to the server should do the trick.
It's strange that "fednode tail counterblock" gives you a different result from "docker logs --follow COUNTERBLOCK_ID". Please, check that you don't have 2 counterblock instances, use "docker ps -a".
Counterwallet is telling you that it can't find counterblock-testnet and you are not running testnet containers, so it's fine, but counterwallet (mainnet) should work. Are you trying to access to counterwallet testnet?
Ok got it.
The counterwallet error does make sense.
About the different results for counterblock I can confirm there is only a single instance for docker ps. What I will do is wait to see how far fednode tail counterblock
goes. If it is unable to sync, I'll make it restart from scratch by doing it with the correct non-sudo user...
Following up here. The above recommendations did work (https://github.com/CounterpartyXCP/counterblock/issues/191#issuecomment-1166725616), only the last step (9) I used fednode start counterblock
and fednode tail counterblock
instead of doing the docker commands.
Also FYI, I wasted a bunch of time with a 'bad' counterblock sync that was taking forever, it seemed like mongod was the reason as it was always in 90+% CPU use. But after following the steps to cleanup, rebooting the machine and restarting counterblock from scratch the sync completed in less than 24hrs.
Closing this issue.
Hello! I have been following the 'full' federated node setup instructions (https://counterparty.io/docs/federated_node/), and most services are already in sync, except counterblock.
The following error keeps repeating, seen by running
fednode tail counterblock
:Any guidance on the best way to learn Counterparty's code will also be appreciated.