search

dtinit / data-transfer-project

The Data Transfer Project makes it easy for platforms to build interoperable user data portability features. We are establishing a common framework, including data models and protocols, to enable direct transfer of data both into and out of participating online service providers.
https://dtinit.org/docs/dtp-what-is-it
Apache License 2.0
3.57k stars 482 forks source link

Logging and Monitoring enhancements #607

Open jimmarino opened 6 years ago

jimmarino commented 6 years ago

We have discussed on various occasions improving our logging capabilities as well as providing hooks for monitoring tools such as Prometheus, Graphite, DataDog, New Relic, etc. We may be able to meet both needs with "Monitor" interface that accepts callbacks (lambdas).

For logging my preference would be to remove all logging implementations (in parts SLF4J is used) so as not to dictate a particular provider. Configuration of many of these logging frameworks also cause problems when they use statics.

Instead, the runtime will make the following interface available through Guice injection (to extensions using that framework) and as a DTP service extension:

https://gist.github.com/jimmarino/f0bcf7d49ce12d1b9edf71a0e0a918f2

Code that wishes to emit monitoring events can do so by having an instance of that interface injected or passed to it.

The implementation of that interface would multiplex to zero or more extensions. For example, we could include a simple console monitoring extensions. Other implementations could forward to SLF4J or Log4J or Java logging.

For logging example use could be:

@Inject Monitor monitor;

public void doSomething() { // ...
monitor.error(()-> "Some error was thrown while doing something", error); }

I also believe that interface should be suitable for monitoring applications such as Graphite since objects containing runtime statistics could be emitted instead of errors.

Comments, ideas, suggestions?

rtannenbaum commented 6 years ago

@holachuy or @olsona could you document why SLF4J was used for data-transfer-project/libraries/logging again? IIRC there was a reason we couldn't use a more extensible logger.

@jimmarino this would replace the current Logger.java, right?

Also, one thing to keep in mind re a monitor extension approach is that many of these logging & monitoring frameworks are supported by multiple cloud platforms/extensions, but cloud platforms often have a "cloud native" logging/monitoring framework, such as StackDriver for Google Cloud Platform.

Perhaps we could implement these Monitors as extensions but bundle one in as default for a CloudExtension, or introduce a notion of compatibility or default another way, or just leave it up to the distribution writers to choose a compatible/default framework for the cloud extension they use.

jimmarino commented 6 years ago

@rtannenbaum That's correct, it would replace logger.

I agree we should support other cloud logging and monitoring options. As a baseline, I think we could start with a simple console output implementation since many of those options have ways to pipe console output. We can also add extensions for common implementations.