hrxi / tracing-loki

A tracing layer for shipping logs to Grafana Loki
Apache License 2.0
50 stars 20 forks source link

Crash when running in docker #4

Closed Christoph-AK closed 2 years ago

Christoph-AK commented 2 years ago

Hey there, I want to migrate my logging to loki, and successfully got my code to use this crate locally. I can see all the interesting things in grafana, all my test run, everything is great.

Then I deploy to AWS EC2 with docker-compose, and my app won't run anymore. I don't even get panic, the process is just killed. I only get a 139 exit code, which apparently is a docker code for a SEGV (https://stackoverflow.com/a/69763409).

After adding a few printlns to my code, it seems that this line is responsible for the crash:

    let (layer, task) = tracing_loki::layer(
        loki_url,
        vec![host, app, log_env].into_iter().collect(),
        Default::default(),
    )?;

I basically copied it from the examples. Do you know what could be going wrong here?

Christoph-AK commented 2 years ago

Possibly related: https://github.com/awslabs/aws-lambda-rust-runtime/issues/270

Christoph-AK commented 2 years ago

Aaand that's indeed the problem. Reqwest crashes in docker with native-tls. The reqwest::Client::builder().build() in Backgroundtask::new() straight up segfaults. Did some 'debugging' in my fork.

I really like @greaka s approach to this: https://github.com/hrxi/tracing-loki/compare/master...greaka:tracing-loki:master

They feature gates their tls dependency, so consumers of tracing-loki get to decide which version they want to run.

Could we get that merged? Maybe with native-tls as default for backwards compatibilty?

hrxi commented 2 years ago

I'd accept a pull request that does this.

hrxi commented 2 years ago

@Christoph-AK @greaka

tracing-loki 0.2.1 has been released, fixing this bug.

Christoph-AK commented 2 years ago

Works perfectly, thank you ❤️