search

gnosis / prediction-market-agent-tooling

Tools to benchmark, deploy and monitor prediction market agents.
GNU Lesser General Public License v3.0
17 stars 2 forks source link

Simplify our logging #335

Open kongzii opened 2 months ago

kongzii commented 2 months ago

The patch_logger function is starting to be quite complicated.

Could it be somehow simpler?

Currently, the main complication is that (1) GCP needs a specific format for logs to be parsed correctly and (2) if something is logging multi-line messages, it breaks the formatting so we have to replace all new lines with a space.

Another complication is that there are 4 different logging modules, we like loguru, libraries usually uses just Python's logging module, warning library is sometimes used instead of logging.warning method and a lot of 3-rd party libraries just go with a simple print, so we need to patch all of these.

Theoretically, we could (hopefully somehow) re-configure the default behaviour in GCP Logs, where unformatted messages would be INFO instead of ERROR by default. But is that a good approach? (theoretically we could miss some important errors)

kongzii commented 2 months ago

For some reason patching isn't working correctly in our API pods and some warnings messages are still shown as errors in GCP.

To stop the fake alerts, I added this condition to the alerting policy:

NOT "sys:1: ResourceWarning: unclosed <ssl.SSLSocket"