search

kedro-org / kedro

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
https://kedro.org
Apache License 2.0
9.53k stars 877 forks source link

Support for remote logging #548

Closed fkromer closed 3 years ago

fkromer commented 3 years ago

Description

It seems like python-anyconfig is used in kedro to configure logging. python-anyconfig supports file based logging only.

Context

File based logging may introduce overhead in case you want to process logs. Distributed logging is a quite common these days. Many developers are familiar with it.

Possible Implementation

Possible Alternatives

mmchougule commented 3 years ago

@fkromer are you thinking of using something like InfluxDB to push logs to? It can then be used in Grafana to monitor them. I did this in one of my projects.

fkromer commented 3 years ago

@mmchougule Exactly.

mmchougule commented 3 years ago

I think in that case, I did the following simple things to push logs to influxDB:

  1. Install telegraf plugin, create telegraf conf file.
  2. Configure telegraf conf file. Created custom_pattern logstash-style grok pattern to parse logs from this file.
  3. Update logging.yml in kedro to match this pattern.

Then, running telegraf, was able to push logs successfully to InfluxDB. Once logs are in the DB, we can do other things with it.

Are you thinking of a different approach or somehow streamlining this approach? Possibly with a new kedro plugin?

fkromer commented 3 years ago

At work I don't need integration ATM. But I thought about experimenting w.r.t. this in a private side project. Thanks for the hints. I'd use an influxDB client for Python and declare kedro internal logging to be disabled in logging.yml. One could wrap the logic into a kedro plugin of course. But I don't have the time to this in the near future. Probably it's worth to be put onto the roadmap however.

yetudada commented 3 years ago

Hi @fkromer, let us know how the experimentation went. We'd be happy to see a contribution in the form of a Kedro plugin. We're going to close this issue now but let us know if we should reopen it.