darold / pgbadger

A fast PostgreSQL Log Analyzer
http://pgbadger.darold.net/
PostgreSQL License
3.51k stars 349 forks source link

cloudsql logs parsing is broken #704

Closed thomasLeclaire closed 2 years ago

thomasLeclaire commented 2 years ago

Hello!

cloudsql logs files parsing is fully broken since today. following the commits from today: I can see that all the lines are now reported as invalid json :

DEBUG: Unknown jsonlog line format: {"insertId":"s=3be20b1689d846398afc4035ab865652;i=34d27fa2;b=3cbceca4ac8e433d9cf6a8fc657d24e2;m=34a6fb72d2;t=5d11300d4cb1c;x=631012c1bb51fc3d-0-0@a1","logName":"projects/proj/logs/cloudsql.googleapis.com%2Fpostgres.log","receiveTimestamp":"2021-11-18T16:54:02.653315461Z","resource":{"labels":{"database_id":"dbid","project_id":"proj","region":"europe-west4"},"type":"cloudsql_database"},"severity":"INFO","textPayload":"2021-11-18 16:53:52.082 UTC [1287570]: [836532-1] db=dbname,user=username LOG: duration: 0.139 ms execute : SELECT DISTINCT sodata_products.provider, sodata_customers.provider_customer_ref, sodata_conversion_types.name FROM \"public\".\"sodata_conversion_snippets\" INNER JOIN \"public\".\"sodata_products\" ON \"public\".\"sodata_products\".\"id\" = \"public\".\"sodata_conversion_snippets\".\"product_id\" INNER JOIN \"public\".\"sodata_customers\" ON \"public\".\"sodata_customers\".\"id\" = \"public\".\"sodata_products\".\"customer_id\" INNER JOIN \"public\".\"sodata_conversion_types\" ON \"public\".\"sodata_conversion_types\".\"id\" = \"public\".\"sodata_conversion_snippets\".\"conversion_type_id\" WHERE \"public\".\"sodata_conversion_snippets\".\"snippet\" IS NOT NULL AND \"sodata_products\".\"state\" NOT IN ($1, $2) AND \"sodata_products\".\"provider\" = $3 AND \"sodata_customers\".\"provider_customer_ref\" = $4","timestamp":"2021-11-18T16:53:52.082716Z"}

i'm using such command : pgbadger --explode --format=jsonlog --outdir=/usr/share/output_pgbadger/html --incremental --jobs 4 --retention=0 --exclude-db=postgres --exclude-db=cloudsqladmin --exclude-user=postgres --wide-char --extra-files --prettify-json /usr/share/input_pgbadger/cloudsql.googleapis.com/postgres.log/$date/*json --verbose;

darold commented 2 years ago

Right, cloudsql log with single line json was broken, commit cdffbec fixes this issue. Please confirm.