search

elastic / connectors

Source code for all Elastic connectors, developed by the Search team at Elastic, and home of our Python connector development framework
https://www.elastic.co/guide/en/enterprise-search/master/index.html
Other
61 stars 120 forks source link

Not possible to pipe logs for connectors to file #1460

Open artem-shelkovnikov opened 10 months ago

artem-shelkovnikov commented 10 months ago

Bug Description

It's not possible to pipe logs for running connectors to a file.

To Reproduce

Steps to reproduce the behavior:

  1. Run make run >> logs.txt or bin/elastic-ingest >> logs.txt with installed service
  2. Observe the content of logs.txt
  3. File is empty

Expected behavior

All server logs are piped to the file

seanstory commented 10 months ago

I think this is a STDERR thing. My bash is rusty, but I feel like you need a 2> to get STDERR. This works:

Seans-MacBook-Pro:connectors-python seanstory$ bin/elastic-ingest 2> output.txt
Seans-MacBook-Pro:connectors-python seanstory$ cat output.txt 
[FMWK][10:27:48][INFO] Running connector service version 8.11.0.0
[FMWK][10:27:48][INFO] Loading config from /Users/seanstory/Desktop/Dev/connectors-python/connectors/../config.yml
[FMWK][10:27:48][INFO] Preflight checks...
...
[FMWK][10:27:48][INFO] Bye
seanstory commented 10 months ago

also, nohup bin/elastic-ingest works as expected.

artem-shelkovnikov commented 10 months ago

IMO this should be default behaviour though - normal logs to stdout, error logs to stderr.

Curious - why do we have this behaviour override?

seanstory commented 10 months ago

Looks like python's logging module defaults to STDERR. Seeing SO posts like: https://stackoverflow.com/a/14058475 I think we could easily change this to STDOUT if that feels important. Or we could expose this as a configuration.

artem-shelkovnikov commented 10 months ago

Yeah it's an easy change, but IMO really important for predictability of service (and I often needed to pipe logs to a file, so I expect there will be other people with same expectations)