search

ecadlabs / tezos_exporter

A Prometheus metrics exporter for monitoring a Tezos node
https://www.ecadlabs.com/tezos_exporter
MIT License
17 stars 3 forks source link

Logs filled with `monitoring mempool operations` errors #30

Open danielelisi opened 3 years ago

danielelisi commented 3 years ago

We're running the tezos_exporter with the following parameters

$ tezos_exporter -tezos-node-url http://mainnet_node:8732 -bootstraped-poll-interval 10s -bootstraped-threshold 8

I noticed that the Docker logs are filled up with the following error monitoring mempool operations errors

2021-07-16T07:26:28.106254565-04:00 time="2021-07-16T11:26:28Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=applied
2021-07-16T07:26:28.106669729-04:00 time="2021-07-16T11:26:28Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_refused
2021-07-16T07:26:28.107129163-04:00 time="2021-07-16T11:26:28Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=refused
2021-07-16T08:39:26.391385079-04:00 time="2021-07-16T12:39:26Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_delayed
2021-07-16T08:39:26.391999862-04:00 time="2021-07-16T12:39:26Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=applied
2021-07-16T08:39:26.392285558-04:00 time="2021-07-16T12:39:26Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_refused
2021-07-16T08:39:26.392457936-04:00 time="2021-07-16T12:39:26Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=refused
2021-07-16T10:42:42.524004869-04:00 time="2021-07-16T14:42:42Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_delayed
2021-07-16T10:42:42.526903714-04:00 time="2021-07-16T14:42:42Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_refused
2021-07-16T10:42:42.526914804-04:00 time="2021-07-16T14:42:42Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=refused
2021-07-16T10:42:42.526919314-04:00 time="2021-07-16T14:42:42Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=applied
2021-07-16T11:00:46.003275218-04:00 time="2021-07-16T15:00:46Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_delayed
2021-07-16T11:00:46.003496976-04:00 time="2021-07-16T15:00:46Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_refused
2021-07-16T11:00:46.003812822-04:00 time="2021-07-16T15:00:46Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=applied
2021-07-16T11:00:46.004339906-04:00 time="2021-07-16T15:00:46Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=refused
2021-07-16T11:01:16.024557636-04:00 time="2021-07-16T15:01:16Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_delayed
2021-07-16T11:01:16.093522609-04:00 time="2021-07-16T15:01:16Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=refused
2021-07-16T11:01:16.093550228-04:00 time="2021-07-16T15:01:16Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=applied
2021-07-16T11:01:16.093864275-04:00 time="2021-07-16T15:01:16Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_refused
2021-07-16T11:21:29.582456888-04:00 time="2021-07-16T15:21:29Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=applied
2021-07-16T11:21:29.583075061-04:00 time="2021-07-16T15:21:29Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_delayed
2021-07-16T11:21:29.583397827-04:00 time="2021-07-16T15:21:29Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_refused
2021-07-16T11:21:29.583713013-04:00 time="2021-07-16T15:21:29Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=refused
2021-07-16T11:21:59.599926892-04:00 time="2021-07-16T15:21:59Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_delayed
2021-07-16T11:21:59.653021270-04:00 time="2021-07-16T15:21:59Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=refused
2021-07-16T11:21:59.655589310-04:00 time="2021-07-16T15:21:59Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=applied
2021-07-16T11:21:59.655789996-04:00 time="2021-07-16T15:21:59Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_refused
2021-07-16T11:38:04.885606549-04:00 time="2021-07-16T15:38:04Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_refused
2021-07-16T11:38:04.886143873-04:00 time="2021-07-16T15:38:04Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=applied
2021-07-16T11:38:04.886596387-04:00 time="2021-07-16T15:38:04Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_delayed
2021-07-16T11:38:04.886968352-04:00 time="2021-07-16T15:38:04Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=refused
jevonearth commented 3 years ago

When a node is bootstrapping, the /chains/main/mempool/monitor_operations RPC is unavailable. That RPC will 404 until the node has synced fully.

@danielelisi Can you check if these error occur on nodes that are bootstrapping only, or also on fully synced nodes?

danielelisi commented 3 years ago

@danielelisi Can you check if these error occur on nodes that are bootstrapping only, or also on fully synced nodes?

These logs are present on fully bootstrapped node as well. I tried to correlate them to node unsync events but there is no correlation.

For example these logs are taken from a Tezos node fully synched with mainnet. The node last unsync event was at around 00:30AM UTC. The logs below show the mempool operation error being logged almost every hour. log timestamps are in UTC time

time="2021-08-04T11:16:18Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=applied
time="2021-08-04T11:16:18Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_delayed
time="2021-08-04T11:16:18Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_refused
time="2021-08-04T11:16:48Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=refused
time="2021-08-04T14:56:06Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_refused
time="2021-08-04T14:56:36Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_delayed
time="2021-08-04T14:56:36Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_refused
time="2021-08-04T15:14:44Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=refused
time="2021-08-04T15:14:44Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=applied
time="2021-08-04T15:39:21Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=applied
time="2021-08-04T16:57:20Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=refused
time="2021-08-04T16:57:58Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=applied
time="2021-08-04T16:57:58Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_delayed
time="2021-08-04T16:57:58Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=refused
time="2021-08-04T17:18:29Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=applied
time="2021-08-04T17:18:29Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_refused
time="2021-08-04T17:18:29Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_delayed
time="2021-08-04T17:18:29Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=refused
danielelisi commented 3 years ago

Could this be somewhat related to #23 ? That issue also reports parsing errors on mempool endpoints which aligns with the docker error logs. AFAIK json: cannot unmarshal is a Go parsing error

copolycube commented 3 years ago

Hello, I just witnessed exactly the same logs after upgrading my mainnet and granadanet octez tezos nodes (docker images) from tezos/tezos:v9.7 to tezos/tezos:v10.0 .

$ sudo docker-compose  logs --tail=50 -tf tezos-node-exporter
Attaching to mainnet_tezos-node-exporter_1
tezos-node-exporter_1    | 2021-08-20T18:44:34.989839875Z time="2021-08-20T18:44:34Z" level=info msg="starting mempool monitor" pool=branch_delayed
tezos-node-exporter_1    | 2021-08-20T18:44:34.989842582Z time="2021-08-20T18:44:34Z" level=info msg="tezos_exporter starting..." address=":9489"
tezos-node-exporter_1    | 2021-08-20T18:53:02.216883370Z time="2021-08-20T18:53:02Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_refused
tezos-node-exporter_1    | 2021-08-20T18:53:02.216924004Z time="2021-08-20T18:53:02Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_delayed
tezos-node-exporter_1    | 2021-08-20T18:53:02.216989824Z time="2021-08-20T18:53:02Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=applied
tezos-node-exporter_1    | 2021-08-20T18:53:02.217361052Z time="2021-08-20T18:53:02Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=refused
tezos-node-exporter_1    | 2021-08-20T19:07:03.525559151Z time="2021-08-20T19:07:03Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_refused

I had the same with those 2 versions (tried upgrading to the latest one I found a ref for on github)

For info, this is my docker-compose.yml extract for the tezos-exporter 

  tezos-node-exporter:
    #image: ecadlabs/tezos_exporter
    image: ecadlabs/tezos_exporter:v0.0.1-beta.14
    command: --tezos-node-url http://node:${node_port_rpc}
    ports:
      - ${prom_tezos_exporter}:9489
    networks:
      - node
    restart: always

(with variables node_port_rpc and prom_tezos_exporter defined in the local .env

copolycube commented 3 years ago

I just tried the suggestion from error #23 and the RPC mentionned there is answering correctly in my case (my node's RPC is on 8733)

$ curl -s -D - http://localhost:8733/chains/main/mempool/monitor_operations?applied
HTTP/1.1 200 OK
content-type: application/json
transfer-encoding: chunked

[...]