Closed simonzg closed 2 months ago
Any updates on this? Thanks
Any updates on this? Thanks
did u manage to solve it? having the same issue
@iamacook @falvaradorodriguez @moisses89
I am also encountering this issue. I am not sure why but I get 503 on /v1/owners/:safeAddress/safes
sometimes when I change a Tx Service URL in one of my chains.
Restarting safe-client-gateway
helps to solve the issue when it happens.
I am also encountering this issue. I am not sure why but I get 503 on
/v1/owners/:safeAddress/safes
sometimes when I change a Tx Service URL in one of my chains.Restarting
safe-client-gateway
helps to solve the issue when it happens.
hi, could you provide some clue.
for me, the UI can't even load the network unless I did some routing specifically like:
location = /cgw/v1/chains {
proxy_pass http://SAFEINFRA_NGINX_IP:8000/cfg/api/v1/chains;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
After I did above, the network can load.
but i cant proceed to next page to create safe with below error until today not sure what I did to make it work other than rebuilding docker image since i saw there is recent new release
503 Service Unavailable
http://ip:8000/cgw/v1/owners/0x76c87B76b9B79d43F0CADCB987AC4e826e493eB0/safes
and today can proceed to create safe page but unable to create
Could you check that the transaction service url is right in the config-service? The CGW is taking that url from the config service.
Restarting safe-client-gateway helps to solve the issue when it happens.
It is because the chain info is cached and it contains the url of the transaction service.
Could you check that the transaction service url is right in the config-service? The CGW is taking that url from the config service.
Restarting safe-client-gateway helps to solve the issue when it happens.
It is because the chain info is cached and it contains the url of the transaction service.
@moisses89 I think the URL is correct for me, could help troubleshoot the problem I stated above? or we just wait for new release?
My transaction service URL in the config-service
http://localhost:8000/txs
root@containers:/opt/safe-infrastructure# curl http://localhost:8000/txs -L
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8"/>
<title>Safe Transaction Service API</title>
This address has safe but can't even load lol
The issue is not related with a Safe cannot be loaded if not the service would return 404 error (Safe doesn't exist).
My transaction service URL in the config-service http://localhost:8000/txs
This configuration is wrong, cannot be localhost, because in such case the CGW container will try to get access to a tx-service hosted in the CGW container, where is not.
The 503 from the CGW is basically that the CGW is not able to request information to the tx-service.
I think the solution for you would be configure correctly the URL for the tx-service in the config-service and get the logs from the CGW container to debug the error if still persist.
@moisses89 , localhost is just a test because previously I have error too with below config
http://nginx:8000/txs
But now I tried http://nginx:8000/txs
again with below result:
Loading http://IP:8000/welcome got this in browser console:
GET http://IP:8000/cgw/v1/owners/0xb25c9b23Cff086E1FbC1A2f030AaffC665AAd6E2/safes 500 (Internal Server Error)
CGW logs:
{"build_number":"65f4026","level":"error","message":{"chain_id":null,"client_ip":"172.30.187.89","detail":"[\n {\n \"code\": \"invalid_type\",\n \"expected\": \"string\",\n \"received\": \"null\",\n \"path\": [\n \"results\",\n 0,\n \"pricesProvider\",\n \"chainName\"\n ],\n \"message\": \"Expected string, received null\"\n },\n {\n \"code\": \"invalid_type\",\n \"expected\": \"string\",\n \"received\": \"null\",\n \"path\": [\n \"results\",\n 0,\n \"pricesProvider\",\n \"nativeCoin\"\n ],\n \"message\": \"Expected string, received null\"\n }\n]","method":"GET","origin":null,"path":"/v1/owners/0xb25c9b23Cff086E1FbC1A2f030AaffC665AAd6E2/safes","response_time_ms":86.41635600000154,"route":"/v1/owners/:ownerAddress/safes","safe_app_user_agent":null,"status_code":500},"request_id":"45e0ccf9-b08b-4828-8090-644c80291ae4","timestamp":"2024-07-04T06:25:28.503Z","version":"v1.49.0"}
Seems like now is 500 instead of 503
Could you check the response from transaction service to the following request?
/api/v1/owners/0xb25c9b23Cff086E1FbC1A2f030AaffC665AAd6E2/safes
maybe is not returning anything.
Is the service indexing? You would be able to check it seeing the logs of the worker indexer or checking the mastercopies from django admin.
Could you check the response from transaction service to the following request?
/api/v1/owners/0xb25c9b23Cff086E1FbC1A2f030AaffC665AAd6E2/safes
maybe is not returning anything. Is the service indexing? You would be able to check it seeing the logs of the worker indexer or checking the mastercopies from django admin.
@moisses89 , seems indexing but not the erc_20, to be honest the running_locally.md did not state anything that we have to configure for transaction service.
last 100 lines of txs-worker-indexer logs:
2024-07-04 08:24:14,616 [INFO] [4e0f866d/index_new_proxies_task] Start indexing of new proxies
2024-07-04 08:24:14,634 [INFO] [4e0f866d/index_new_proxies_task] ProxyFactoryIndexer: Processing almost updated addresses
2024-07-04 08:24:14,647 [INFO] [4e0f866d/index_new_proxies_task] Proxy indexing found 0 proxies
2024-07-04 08:24:14,654 [INFO] [4e0f866d/index_new_proxies_task] Task safe_transaction_service.history.tasks.index_new_proxies_task[4e0f866d-08a3-4574-a381-4dc141b33a45] succeeded in 0.03851256100460887s: (0, 4)
2024-07-04 08:24:15,089 [INFO] [2ecc9e9c/index_safe_events_task] Start indexing of Safe events
2024-07-04 08:24:15,099 [INFO] [2ecc9e9c/index_safe_events_task] Find Safe events processed 0 events
2024-07-04 08:24:15,106 [INFO] [2ecc9e9c/index_safe_events_task] Task safe_transaction_service.history.tasks.index_safe_events_task[2ecc9e9c-ff21-4508-954c-d5fdd6eeeae1] succeeded in 0.018187063978984952s: (0, 0)
2024-07-04 08:24:20,088 [INFO] [c216608b/index_safe_events_task] Start indexing of Safe events
2024-07-04 08:24:20,098 [INFO] [c216608b/index_safe_events_task] SafeEventsIndexer: Processing not updated addresses
2024-07-04 08:24:20,109 [INFO] [c216608b/index_safe_events_task] Find Safe events processed 0 events
2024-07-04 08:24:20,117 [INFO] [c216608b/index_safe_events_task] Task safe_transaction_service.history.tasks.index_safe_events_task[c216608b-e17e-4fa6-b609-68c3395c936c] succeeded in 0.028513255063444376s: (0, 3)
2024-07-04 08:24:23,612 [INFO] [df99da7f/index_erc20_events_task] Start indexing of erc20/721 events
2024-07-04 08:24:23,634 [INFO] [df99da7f/index_erc20_events_task] Erc20EventsIndexer: Processing almost updated addresses
2024-07-04 08:24:23,642 [WARNING] [df99da7f/index_erc20_events_task] Erc20EventsIndexer: Value error retrieving events from-block=0 to-block=0 : {'code': -32000, 'message': 'maximum [from, to] blocks distance: 10000'}
2024-07-04 08:24:23,642 [INFO] [df99da7f/index_erc20_events_task] Erc20EventsIndexer: block_process_limit set back to 1
2024-07-04 08:24:23,651 [INFO] [df99da7f/index_erc20_events_task] Task safe_transaction_service.history.tasks.index_erc20_events_task[df99da7f-4be5-4367-9fd2-44231b6c2441] retry: Retry in 15s: FindRelevantElementsException('Request error retrieving events from-block=0 to-block=0')
2024-07-04 08:24:24,655 [INFO] [e95a6df0/index_erc20_events_task] Start indexing of erc20/721 events
2024-07-04 08:24:24,675 [INFO] [e95a6df0/index_erc20_events_task] Erc20EventsIndexer: Processing almost updated addresses
2024-07-04 08:24:24,684 [WARNING] [e95a6df0/index_erc20_events_task] Erc20EventsIndexer: Value error retrieving events from-block=0 to-block=0 : {'code': -32000, 'message': 'maximum [from, to] blocks distance: 10000'}
2024-07-04 08:24:24,684 [INFO] [e95a6df0/index_erc20_events_task] Erc20EventsIndexer: block_process_limit set back to 1
2024-07-04 08:24:24,694 [INFO] [e95a6df0/index_erc20_events_task] Task safe_transaction_service.history.tasks.index_erc20_events_task[e95a6df0-0f9d-4c80-a090-c1549ccd492f] retry: Retry in 15s: FindRelevantElementsException('Request error retrieving events from-block=0 to-block=0')
2024-07-04 08:24:25,089 [INFO] [8be2c704/index_safe_events_task] Start indexing of Safe events
2024-07-04 08:24:25,108 [INFO] [8be2c704/index_safe_events_task] SafeEventsIndexer: Processing not updated addresses
2024-07-04 08:24:25,119 [INFO] [8be2c704/index_safe_events_task] Find Safe events processed 0 events
2024-07-04 08:24:25,126 [INFO] [8be2c704/index_safe_events_task] Task safe_transaction_service.history.tasks.index_safe_events_task[8be2c704-5d6e-44a5-a4c2-a816717856bd] succeeded in 0.03787820891011506s: (0, 3)
2024-07-04 08:24:25,802 [INFO] [d0deecff/index_erc20_events_task] Start indexing of erc20/721 events
2024-07-04 08:24:25,855 [INFO] [d0deecff/index_erc20_events_task] Erc20EventsIndexer: Processing almost updated addresses
2024-07-04 08:24:25,868 [WARNING] [d0deecff/index_erc20_events_task] Erc20EventsIndexer: Value error retrieving events from-block=0 to-block=0 : {'code': -32000, 'message': 'maximum [from, to] blocks distance: 10000'}
2024-07-04 08:24:25,868 [INFO] [d0deecff/index_erc20_events_task] Erc20EventsIndexer: block_process_limit set back to 1
2024-07-04 08:24:25,878 [INFO] [d0deecff/index_erc20_events_task] Task safe_transaction_service.history.tasks.index_erc20_events_task[d0deecff-c61f-4121-b89f-81361538e351] retry: Retry in 15s: FindRelevantElementsException('Request error retrieving events from-block=0 to-block=0')
2024-07-04 08:24:26,694 [INFO] [5e57e4d8/index_erc20_events_task] Start indexing of erc20/721 events
2024-07-04 08:24:26,713 [INFO] [5e57e4d8/index_erc20_events_task] Erc20EventsIndexer: Processing almost updated addresses
2024-07-04 08:24:26,721 [WARNING] [5e57e4d8/index_erc20_events_task] Erc20EventsIndexer: Value error retrieving events from-block=0 to-block=0 : {'code': -32000, 'message': 'maximum [from, to] blocks distance: 10000'}
2024-07-04 08:24:26,721 [INFO] [5e57e4d8/index_erc20_events_task] Erc20EventsIndexer: block_process_limit set back to 1
2024-07-04 08:24:26,730 [ERROR] [5e57e4d8/index_erc20_events_task] Task safe_transaction_service.history.tasks.index_erc20_events_task[5e57e4d8-dfb4-4f3f-9995-9988ef717431] raised unexpected: FindRelevantElementsException('Request error retrieving events from-block=0 to-block=0')
Traceback (most recent call last):
File "/app/safe_transaction_service/history/indexers/events_indexer.py", line 150, in _find_elements_using_topics
return self._do_node_query(addresses, from_block_number, to_block_number)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/safe_transaction_service/history/indexers/erc20_events_indexer.py", line 106, in _do_node_query
transfer_events = self.ethereum_client.erc20.get_total_transfer_history(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/gnosis/eth/ethereum_client.py", line 719, in get_total_transfer_history
for event in self.slow_w3.eth.get_logs(parameters):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/web3/eth/eth.py", line 456, in get_logs
return self._get_logs(filter_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/web3/module.py", line 75, in caller
result = w3.manager.request_blocking(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/web3/manager.py", line 330, in request_blocking
return self.formatted_response(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/web3/manager.py", line 293, in formatted_response
raise ValueError(error)
ValueError: {'code': -32000, 'message': 'maximum [from, to] blocks distance: 10000'}
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/celery/app/trace.py", line 453, in trace_task
R = retval = fun(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/celery/app/trace.py", line 736, in __protected_call__
return self.run(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/celery/app/autoretry.py", line 60, in run
ret = task.retry(exc=exc, **retry_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/celery/app/task.py", line 736, in retry
raise_with_context(exc)
File "/usr/local/lib/python3.12/site-packages/celery/app/autoretry.py", line 38, in run
return task._orig_run(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/safe_transaction_service/history/tasks.py", line 110, in index_erc20_events_task
) = Erc20EventsIndexerProvider().start()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/safe_transaction_service/history/indexers/ethereum_indexer.py", line 465, in start
) = self.process_addresses(
^^^^^^^^^^^^^^^^^^^^^^^
File "/app/safe_transaction_service/history/indexers/ethereum_indexer.py", line 422, in process_addresses
raise e
File "/app/safe_transaction_service/history/indexers/ethereum_indexer.py", line 403, in process_addresses
elements = self.find_relevant_elements(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/safe_transaction_service/history/indexers/events_indexer.py", line 201, in find_relevant_elements
log_receipts = self._find_elements_using_topics(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/safe_transaction_service/history/indexers/events_indexer.py", line 168, in _find_elements_using_topics
raise FindRelevantElementsException(
safe_transaction_service.history.indexers.ethereum_indexer.FindRelevantElementsException: Request error retrieving events from-block=0 to-block=0
2024-07-04 08:24:29,616 [INFO] [1bad93d2/index_new_proxies_task] Start indexing of new proxies
2024-07-04 08:24:29,641 [INFO] [1bad93d2/index_new_proxies_task] ProxyFactoryIndexer: Processing almost updated addresses
2024-07-04 08:24:29,675 [INFO] [1bad93d2/index_new_proxies_task] Proxy indexing found 0 proxies
2024-07-04 08:24:29,682 [INFO] [1bad93d2/index_new_proxies_task] Task safe_transaction_service.history.tasks.index_new_proxies_task[1bad93d2-2bb2-4b40-9976-88a3f9546814] succeeded in 0.06662204395979643s: (0, 5)
2024-07-04 08:24:30,090 [INFO] [9405ca3c/index_safe_events_task] Start indexing of Safe events
2024-07-04 08:24:30,104 [INFO] [9405ca3c/index_safe_events_task] SafeEventsIndexer: Processing not updated addresses
2024-07-04 08:24:30,118 [INFO] [9405ca3c/index_safe_events_task] Find Safe events processed 0 events
2024-07-04 08:24:30,125 [INFO] [9405ca3c/index_safe_events_task] Task safe_transaction_service.history.tasks.index_safe_events_task[9405ca3c-b625-497d-b8f8-703a70c4dd1c] succeeded in 0.03583054605405778s: (0, 3)
This one we add ourself manually lol
My case might be different than @lance-web3
I've checked and the transaction service is working fine http://safegw.meter.io:8000/txs/api/v1/owners/0x14b27D8DC12E59a9904DaC6d17D33B8de2E80e66/safes/
but the cgw still throws a 503 error with this link http://safegw.meter.io:8000/cgw/v1/owners/0x14b27D8DC12E59a9904DaC6d17D33B8de2E80e66/safes
And this is the configs in cfg for my network
@moisses89
My case might be different than @lance-web3
I've checked and the transaction service is working fine http://safegw.meter.io:8000/txs/api/v1/owners/0x14b27D8DC12E59a9904DaC6d17D33B8de2E80e66/safes/
but the cgw still throws a 503 error with this link http://safegw.meter.io:8000/cgw/v1/owners/0x14b27D8DC12E59a9904DaC6d17D33B8de2E80e66/safes
@moisses89
Can you ensure that the CGW have access to the transacción service? What is the tx-service URL that you have configures in the config-service?
I've configured http://safegw.meter.io:8000/txs
as transaction service uri in config-service, as shown in the screenshot here:
And this url is public accessible and CGW should have no problem accessing it. I've checked inside container safe-infrastructure-cgw-web-1
and it has no issue access it:
I've configured
http://safegw.meter.io:8000/txs
as transaction service uri in config-service, as shown in the screenshot here:And this url is public accessible and CGW should have no problem accessing it. I've checked inside container
safe-infrastructure-cgw-web-1
and it has no issue access it:
Ok thanks, this helps. If It was configured just now maybe you need to restart the cgw-web and the cgw-redis to clean cached data from the config service. Would help me if you can share the CGW logs with the error messages from the CGW pod.
I restarted cgw-web and cgw-redis but the 503 error is still there.
and the logs from safe-infrastructure-cgw-web-1
looks like this:
{"build_number":"b2bd0b2","level":"info","message":{"chain_id":null,"client_ip":"172.31.25.116","detail":null,"method":"GET","origin":null,"path":"/v1/chains","response_time_ms":3.780128999998851,"route":"/v1/chains","safe_app_user_agent":null,"status_code":200},"request_id":"a32fe9fd-d870-4eab-a092-838831824abe","timestamp":"2024-07-04T17:45:34.003Z","version":"v1.43.0"}
{"build_number":"b2bd0b2","level":"error","message":{"chain_id":null,"client_ip":"172.31.25.116","detail":"Service unavailable","method":"GET","origin":null,"path":"/v1/owners/0x14b27D8DC12E59a9904DaC6d17D33B8de2E80e66/safes","response_time_ms":248.06489399999919,"route":"/v1/owners/:ownerAddress/safes","safe_app_user_agent":null,"status_code":503},"request_id":"47dc6bc9-f648-4b3e-955f-ae62db5dbec6","timestamp":"2024-07-04T17:45:42.274Z","version":"v1.43.0"}
Let me know if you want to check anything else.
In addition, I pulled all the latest docker images and tried again, still the same 503 error.
The safe-infrastructure-cgw-web-1
container could access transaction service api with no error:
And I turned on the debug log and hopefully this time log of safe-infrastructure-cgw-web-1
contains extra information for 503:
{"build_number":"5e06389","level":"debug","message":{"field":"undefined_undefined","key":"chains","type":"cache_hit"},"request_id":"21464c88-0451-4942-8e0b-fd9fa5574278","timestamp":"2024-07-04T21:34:19.818Z","version":"v1.50.0"}
{"build_number":"5e06389","level":"debug","message":{"field":"","key":"8453_owner_safes_0x14b27D8DC12E59a9904DaC6d17D33B8de2E80e66","type":"cache_miss"},"request_id":"21464c88-0451-4942-8e0b-fd9fa5574278","timestamp":"2024-07-04T21:34:19.819Z","version":"v1.50.0"}
{"build_number":"5e06389","level":"debug","message":{"field":"","key":"82_owner_safes_0x14b27D8DC12E59a9904DaC6d17D33B8de2E80e66","type":"cache_miss"},"request_id":"21464c88-0451-4942-8e0b-fd9fa5574278","timestamp":"2024-07-04T21:34:19.821Z","version":"v1.50.0"}
{"build_number":"5e06389","level":"error","message":{"chain_id":null,"client_ip":"172.31.12.90","detail":"Service unavailable","method":"GET","origin":null,"path":"/v1/owners/0x14b27D8DC12E59a9904DaC6d17D33B8de2E80e66/safes","response_time_ms":6.483422999968752,"route":"/v1/owners/:ownerAddress/safes","safe_app_user_agent":null,"status_code":503},"request_id":"21464c88-0451-4942-8e0b-fd9fa5574278","timestamp":"2024-07-04T21:34:19.824Z","version":"v1.50.0"}
I think there are new fields mandatory in the config service, could you check if they are all filled? Could you share the current version of each service? CGW, ConfigService and Transactionservice? Also would help to ensure if the CGW can access to the chain information, asking from the CGW the chains list. GET /v1/chains
Thanks for your reply. After checking the settings for chains in the config service, I noticed that there's an invalid record for Base
network that I added for testing weeks ago, after deleting this record, everything is working now.
The 503 issue was there before I added the Base
network, so I don't know exactly what was the cause then. All I did recently was pull docker images to use the latest versions, and add new fields in container_env_files/cgw.env
according to https://github.com/safe-global/safe-infrastructure/pull/144 , and it magically works.
Versions that I'm using: CGW: v1.50.0 TransactionService: v5.2.2 ConfigService: v2.83.0
I'll mark this issue as closed for now. Thank you so much for your time @moisses89
@moisses89 mine 503 issue solved too after i pull the new cgw image and follow some of the steps above but i have new issue when creating safe which i will reference in here: https://github.com/safe-global/safe-infrastructure/issues/147
I followed the instrucitons here with only one change: instead of using localhost, I decided to use my own domain (safegw.meter.io). I've replaced localhost with safegw.meter.io in all the env files under
container_env_files
of course.The UI is working and I created a safe wallet on it, the transaction went through and safe created successfully. But UI failed to load the safe, when I open the console on chrome, it shows that this url is 503:
I've checked my ChainInfo and confirm it has the correct EIP-3770 shortname, and confirm the proxy factory and master copy address are in the transaction service db, as mentioned in issue #141 . I tried to read the code of client-gateway and looks like the request on /cgw/v1/owners/:safeAddress/safes depends on transaction-service, so I tested http://safegw.meter.io:8000/txs/api/v1/owners/0x14b27D8DC12E59a9904DaC6d17D33B8de2E80e66/safes and it works.
I've attached the log of safe-infrastructure-cgw-web-1, but other than
Service Unavailable
it doesn't have anything helpful.Any suggestions would be much appreciated!