Logging doesn't work

[amitdixit]

I have followed the steps from https://medium.com/hepsiburadatech/fluent-logging-architecture-fluent-bit-fluentd-elasticsearch-ca4a898e28aa throughly but not able to succeed for last 3 days. I have done all the steps as pert the post. The only difference is my FLuentd and Elastic Kibana are running in the docker. Need your help in this.

https://github.com/amitdixit/efk-dotnet.git

[error] [output:forward:forward.0] no upstream connections available [2022/08/08 12:02:05] [ warn] [engine] failed to flush chunk '1-1659960106.171172984.flb', retry in 34 seconds: task_id=9, input=tail.0 > output=forward.0 (out_id=0) [2022/08/08 12:02:05] [error] [output:forward:forward.0] no upstream connections available [2022/08/08 12:02:05] [ warn] [engine] failed to flush chunk '1-1659960103.32201405.flb', retry in 11 seconds: task_id=7, input=tail.0 > output=forward.0 (out_id=0) [2022/08/08 12:02:05] [ warn] [engine] failed to flush chunk '1-1659960113.32502984.flb', retry in 20 seconds: task_id=13, input=tail.0 > output=forward.0 (out_id=0) [2022/08/08 12:02:07] [error] [output:forward:forward.0] no upstream connections available [2022/08/08 12:02:07] [ warn] [engine] failed to flush chunk '1-1659960115.540334596.flb', retry in 6 seconds: task_id=14, input=tail.0 > output=forward.0 (out_id=0) [2022/08/08 12:02:08] [error] [output:forward:forward.0] no upstream connections available [2022/08/08 12:02:08] [error] [output:forward:forward.0] no upstream connections available [2022/08/08 12:02:08] [ warn] [engine] failed to flush chunk '1-1659960123.39019777.flb', retry in 6 seconds: task_id=18, input=tail.0 > output=forward.0 (out_id=0) [2022/08/08 12:02:08] [error] [output:forward:forward.0] no upstream connections available [2022/08/08 12:02:08] [ warn] [engine] failed to flush chunk '1-1659960105.160793833.flb', retry in 39 seconds: task_id=8, input=tail.0 > output=forward.0 (out_id=0) [2022/08/08 12:02:08] [ warn] [engine] failed to flush chunk '1-1659960124.610357817.flb', retry in 11 seconds: task_id=19, input=tail.0 > output=forward.0 (out_id=0) [2022/08/08 12:02:09] [error] [output:forward:forward.0] no upstream connections available [2022/08/08 12:02:09] [ warn] [engine] failed to flush chunk '1-1659960088.428308125.flb', retry in 15 seconds: task_id=0, input=tail.0 > output=forward.0 (out_id=0) [2022/08/08 12:02:10] [error] [output:forward:forward.0] no upstream connections available

[amitdixit]

Finaaly fixed it. So the issue was I needed to use host.docker.internal instead of 127.0.0.1. Also had to tweak Elasticsearch config files. I will update the repository soon.

[senvardarsemih]

Hi @amitdixit, glad to hear you solved it. The main reason for the "no upstream connections available" situation is usually about fluentd cannot reach elasticsearch. If you are in docker environment, it is necessary to make sure that the network settings are configured correctly. Anyway, I'm so glad you figured it out. See you if you run into any other problems.

[amitdixit]

Thanks Semih. Sure will do. So after 3 days I was able to log. For those who are stuck below are the steps.

1. In my case my app and Fluentbit daemon are running in K8S , Elasticsearch, Fluentd, Kibana are running in Docker.
2. Elastic search and Kibana version is 8.3.3 (Latest as of now, Many things have changed from version 7.**),
3. Fluentd is missing the Elasticsearch plugin ; hence we need to implement a custom Fluentd image with configuration and plugins.
4. If using Docker we need to use appropriate Host like host.docker.internal.
5. Also just for sake of debugging just make ssl related security in elstic.yml file as false.

For reference sharing the repo. https://github.com/amitdixit/efk-dotnet.git