search

axiomhq / cli

The power of Axiom on the command line.
https://axiom.co
MIT License
42 stars 11 forks source link

Ingest Error: 504 API Timeout #190

Closed RoloEdits closed 9 months ago

RoloEdits commented 11 months ago

Twice in the last 3 months has ingest stopped due to an error on timeout and a failure to reconnect. The fix has been to just keep deployments going at smaller intervals, but for some of our longer uptime services, the timeouts stopping all ingest has been confusing.

This is the error:

Error: could not ingest "stdin" into dataset "NAME": API error 504: Gateway Timeout

The deployments are done inside of a Docker container with this initiating the CLI:

ENTRYPOINT ./APP | axiom ingest DATASET

I'm not sure what would be causing the API timeout multiple times like this, but perhaps there could be a --continue-on-error flag that would keep trying to reconnect through all errors and just keep logging them to the console? The error was only logged once so its unknown how many times it took to give up trying. At least with a continual log we could see the number of attempts.

lukasmalkmus commented 10 months ago

Thanks for the report.

There can be various reasons for a 504. Axiom Go, which powers Axiom CLI has a sensible retry-mechanism but apparently not every 504 can be resolved adequately. I'll be checking out the parameters we're using over there but nonetheless, --continue-on-error might be a nice addition for ingest and stream.

RoloEdits commented 10 months ago

Unsure if the same issue as the error message is different but I am seeing this:

Error: could not ingest "stdin" into dataset "DATASET": Post "https://api.axiom.co/v1/datasets/DATASET/ingest": read tcp IP:PORT -> IP:443: read: connection reset by peer

It took 8 days this time before it was unable to send to ingest anymore.

lukasmalkmus commented 10 months ago

@RoloEdits Is CLI invoked exactly as you described above?

ENTRYPOINT ./APP | axiom ingest DATASET

No extra command line arguments or any specific environment variables set?

RoloEdits commented 10 months ago

CLI invocation has no flags, and AXIOM_TOKEN env is set.

RoloEdits commented 10 months ago

Got a new message, this time a 503 Error.

Error: could not ingest "stdin" into dataset "DATASET": API error 503: Service Unavailable
lukasmalkmus commented 9 months ago

@RoloEdits Ups, forgot to ping you about it but can you try the latest release with --continue-on-error applied?

RoloEdits commented 9 months ago

We have moved over to using https://vector.dev/ and sending to the axiom sink as it has had no issues ingesting. I will have to look into getting some services back to the axiom cli, but its pretty busy currently. Not sure how soon I can get that going.