search

jup-ag / jupiter-swap-api

The jupiter swap api binary
97 stars 24 forks source link

WARN jupiter_core::add_market: Could not get account for market.... #50

Open dancamarg0 opened 2 months ago

dancamarg0 commented 2 months ago

Today we noticed all of our jupiter servers were throwing this warning in the logs, suggesting we couldn't consume new markets from the Europa server

Restarting the jupiter-swap-api service fixed it somehow...

I'm not sure where the issue lies, but as a suggestion could you add a prometheus metric under the --metrics-port that reports a boolean when we lose connection to the europa server? also maybe another metric that counts amount of warnings and errors from add_market and europa_client so we can build alerts and act quicker if this happens again

ZhengYuTay commented 2 months ago

yeap, sorry about that, we had an issue where it stopped sending new markets to the client. It was resolved.

dancamarg0 commented 1 month ago

Today I saw a big wave of these warnings again with

WARN tower_http::trace::on_failure: response failed classification=Status code: 500 Internal Server Error latency=0 ms

Are 500 a warning when our instances throw 500 internal server error to the client? Or it means another thing?

These wave of warnings seem to co-relate with widespread issues in our server. Speaking of today we were having very weird price impacts for some of the routes and they just got fixed after restarting our instances