search

axiomhq / cli

The power of Axiom on the command line.
https://axiom.co
MIT License
41 stars 11 forks source link

Ingestion problems after #25 #33

Closed lukasmalkmus closed 3 years ago

lukasmalkmus commented 3 years ago

A regression was introduced with https://github.com/axiomhq/cli/commit/f534579be074e2dee71824e3331df09463f79a1b which causes ingestion to fail:

logs.json:

{"time": "17/May/2015:08:05:32 +0000", "remote_ip": "93.180.71.3", "remote_user": "-", "request": "GET /downloads/product_1 HTTP/1.1", "response": 304, "bytes": 0, "referrer": "-", "agent": "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.21)"}
{"time": "17/May/2015:08:05:32 +0000", "remote_ip": "93.180.71.3", "remote_user": "-", "request": "GET /downloads/product_1 HTTP/1.1", "response": 304, "bytes": 0, "referrer": "-", "agent": "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.21)"}
{"time": "17/May/2015:08:05:32 +0000", "remote_ip": "93.180.71.3", "remote_user": "-", "request": "GET /downloads/product_1 HTTP/1.1", "response": 304, "bytes": 0, "referrer": "-", "agent": "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.21)"}

cat ~/Downloads/logs.json | axiom ingest lukas-logs:

Error: could not ingest "stdin" into dataset "lukas-logs": API error 400: 400 Bad Request: "invalid event at index 17: ReadObject: expect { or , or } or n, but found t, error found in #10 byte of ...|:42 +00{\"time\": \"17/|..., bigger context ...|{\"time\": \"17/May/2015:08:05:42 +00{\"time\": \"17/May/2015:08:05:42 +0000\", \"remote_ip\": \"|..."

Having just a single event inside logs.json fails silently, without ingesting and raising an error.

lukasmalkmus commented 3 years ago

The PR just fixed the second issue mentioned:

Having just a single event inside logs.json fails silently, without ingesting and raising an error.

The previous issue still exists :(

lukasmalkmus commented 3 years ago

Only occurs with a given file size. It seems like the json content Is cut mid-line. To reproduce, add like 18 lines of the data from above and try to ingest.