logdna / logdna-agent-v2

The blazingly fast, resource efficient log collection client
https://logdna.com
MIT License
63 stars 46 forks source link

Logdna-agent v3 fail to start with "thread 'main' panicked at 'called `Option::unwrap()` on a `None` value'" error #219

Open jaszxg opened 2 years ago

jaszxg commented 2 years ago

Hi LogDNA team, im seeing logdna-agent crash with the following error on bootup:

[2021-09-10T17:00:36Z INFO fs::tail] initialized "regional-snapshot-86dd4dcf77-t6hjd" with offset 0 thread 'main' panicked at 'called Option::unwrap() on a None value', /opt/rust/cargo/git/checkouts/logdna-rust-da7bb1a0a2df3f84/d7adef1/src/request.rs:274:49

I am not sure if this is related, but further investigating, i find that one of the log file that logdna-agent tries to tail has a log line but in the logdna-agent log it seems to think it is empty:

[2021-09-13T02:15:01Z INFO  fs::tail] initialize event for file "image-rest-server-5c5b957fff-qsj46", target "/var/log/at/image-rest-server-5c5b957fff-qsj46"
[2021-09-13T02:15:01Z INFO  fs::tail] initialized "image-rest-server-5c5b957fff-qsj46" with offset 0

But looking at the size of the file

[logdna@logdna-agent-atracker-lsgwb at]$ wc -l  image-rest-server-5c5b957fff-qsj46
1 image-rest-server-5c5b957fff-qsj46

which indicates there is a single line

This is observed in logdna-agent v3.1 and later

c-nixon commented 2 years ago

@jaszxg We think we've identified where this panic was coming from. I don't know if you still have the reproduction environment available, but can you confirm that the LOGDNA_INGESTION_KEY environment variable was set and the LOGDNA_HOST variable was either set or completely unset (not set to empty) when you ran the agent?

jzxg commented 2 years ago

Yes they were set, in fact the workaround we used was to add the extensionless files to the inclusion rules, which worked

jzxg commented 2 years ago

Sorry I did not update with latest findings, but the issue turn out to be that after Logdna-agent v3.1, a regression was introduced such that it no longer supports extensionless files. This was acknolwedged by the logdna team to be a bug.