[Sandbox] Loggie

[ethfoo]

Application contact emails

fuyi1@corp.netease.com, hzzhoujianding@corp.netease.com

Project Summary

Loggie is a lightweight, cloud-native, event-driven data collector, transformer and aggregator.

Project Description

Loggie is a cloud-native, event-driven data collector and aggregator that is designed to be lightweight and high-performing. It is written in Golang and uses a pluggable architecture that allows users to easily configure multiple pipelines using custom sources, interceptors, and sinks. Loggie can be deployed as either an Agent or an Aggregator, and its pipelines can be connected to form large-scale data flows.

One of the key benefits of Loggie is its ability to support various types of data streams and to normalize, transform, and process data in real-time. It is also scalable, with the ability to handle large volumes of data without requiring significant resources. Additionally, Loggie is designed to be isolated, meaning that different data streams do not influence each other, providing a more stable and secure transmission.

In the context of Kubernetes, Loggie can be easily configured using CRD (Custom Resource Definitions) to create standard streaming data transmission pipelines. It can be used in a variety of scenarios, including collecting container and host logs, aggregating and forwarding data, and collecting Kubernetes events. Overall, Loggie is an ideal tool for managing and processing streaming data in a cloud-native environment.

Org repo URL

https://github.com/loggie-io

Project repo URL

https://github.com/loggie-io/loggie

Additional repos

No response

Website URL

https://loggie-io.github.io/docs-en/

Roadmap

https://loggie-io.github.io/docs-en/getting-started/roadmap/roadmap-2023/

Roadmap context

No response

Contributing Guide

https://loggie-io.github.io/docs-en/developer-guide/contributing/

Code of Conduct (CoC)

https://github.com/loggie-io/loggie/blob/main/CODE-OF-CONDUCT.md

Adopters

No response

Contributing or Sponsoring Org

Netease

Maintainers file

https://github.com/ethfoo, https://github.com/mmaxiaolei

IP Policy

• [X] If the project is accepted, I agree the project will follow the CNCF IP Policy

Trademark and accounts

• [X] If the project is accepted, I agree to donate all project trademarks and accounts to the CNCF

Why CNCF?

Loggie is a rapidly developing project with a strong focus on user experience and creative ideas. We have received positive feedback from companies currently utilizing Loggie and believe that joining the Cloud Native Computing Foundation (CNCF) will increase the visibility of the project, attract more users and contributors, and help us to continually improve and make Loggie more widely applicable. We believe that joining the CNCF will provide us with valuable resources and support as we strive to make Loggie an easy-to-use and highly effective tool for a wide range of scenarios.

Benefit to the Landscape

As a cloud-native log collector, Loggie has unique capabilities for collecting container logs in Kubernetes environments. Unlike Fluentd and other log collectors, which require mounting and collecting from all standard output paths, Loggie allows users to flexibly choose which pods to collect logs from using the LogConfig Custom Resource Definition (CRD). In addition to collecting standard output logs, Loggie can also collect log files within the container without requiring any modifications to the pod.

One of the key advantages of Loggie is that it includes a built-in Kubernetes operator/controller, eliminating the need to deploy additional operator components. Users can simply create a LogConfig/ClusterLogConfig CRD to perform individual log collection configuration, format processing, and data conversion for each workload. Additionally, the use of Interceptor/Sink CRDs allows users to quickly build a log transmission architecture for collecting, aggregating, and forwarding logs.

Beyond log collection, Loggie's multi-pipeline and source->interceptor->sink processing architecture make it a versatile tool for collecting, processing, aggregating, and forwarding a wide range of events. In event-driven scenarios or situations where multiple event sources need to be connected, Loggie can be used to collect and convert events to cloud events or opentelemetry logs format. This makes Loggie an valuable addition to the CNCF, providing a powerful tool for log and event collection, processing, and aggregation.

Cloud Native 'Fit'

Based on the idea of cloud-native Operator/Controller, Loggie provides a better solution for transmitting data such as logs in Kubernetes.

Cloud Native 'Integration'

Prometheus

• Loggie primarily focuses on log collection, but also supports the conversion of logs into metrics such as Nginx access logs. Similar to mtail, Loggie supports log processing and structuring, and supports data aggregation over a period of time into metrics. The converted metrics can then be sent to Prometheus via Loggie's prometheusExporter sink or prometheusRemoteWrite sink for further analysis.
• Loggie has the capability to detect and match log data according to rules and can trigger alerts through its integration with Prometheus AlertManager via the alertWebhook sink.
• Loggie also supports the collection of Prometheus metrics as an agent on each node, with the ability to perform pre-aggregation and processing of metric data, providing an additional method for Prometheus to collect metrics.

opentelemetry

Loggie incorporates a robust transformer interceptor that can be used to comply with the log standard format defined by OpenTelemetry. Additionally, Loggie plans to implement a fluentforwardreceiver to facilitate seamless integration with the OpenTelemetry Collector.

Grafana Loki

Loggie has a loki sink, so Loggie can directly collect logs and send them to Loki for storage, and can use Grafana to query and retrieve logs.

Cloud Native Overlap

Fluentd/Filebeat

One of the main scenarios of Loggie is log collection and transmission. Compared to similar projects, Loggie has distinct design and user experience features such as:

• Loggie is a log collection and transmission tool that utilizes a cloud-native design, implemented as a Kubernetes Operator. This design allows for a more user-friendly and powerful log collection experience within Kubernetes environments. (please refer to the Benefit to the Landscape section above)
• Loggie also features a Multi-Pipeline design, which provides stronger isolation in complex production environments. This feature enables separate deployment as a data transmission transit and aggregation solution.
• Loggie boasts improved performance and a more lightweight design compared to similar projects. This allows for faster data transmission and a more efficient use of resources.

Similar projects

Fluentd

Product or Service to Project separation

N/A

Project presentations

No response

Project champions

No response

Additional information

No response

[dims]

There are a number of projects/tools in CNCF around monitoring/logging/tracing can you please write about any intersection/cooperation between the existing projects and the one you are proposing?

[ethfoo]

There are a number of projects/tools in CNCF around monitoring/logging/tracing can you please write about any intersection/cooperation between the existing projects and the one you are proposing?

Thanks for the suggestion. Have added to the Cloud Native 'Integration' and Cloud Native Overlap section.

[amye]

/vote-sandbox

[git-vote[bot]]

Vote created

@amye has called for a vote on [Sandbox] Loggie (#20).

The members of the following teams have binding votes:

Team
@cncf/cncf-toc

Non-binding votes are also appreciated as a sign of support!

How to vote

You can cast your vote by reacting to this comment. The following reactions are supported:

In favor	Against	Abstain
👍	👎	👀

Please note that voting for multiple options is not allowed and those votes won't be counted.

The vote will be open for 7days. It will pass if at least 66% of the users with binding votes vote In favor 👍. Once it's closed, results will be published here as a new comment.

[amye]

/check-vote

[git-vote[bot]]

Vote status

So far 18.18% of the users with binding vote are in favor (passing threshold: 66%).

Summary

In favor	Against	Abstain	Not voted
2	0	0	9

Binding votes (2)

User	Vote	Timestamp
cathyhongzhang	In favor	2023-05-09 22:27:54.0 +00:00:00
rochaporto	In favor	2023-05-09 18:55:28.0 +00:00:00

Non-binding votes (2)

| User | Vote | Timestamp | | ---- | :---: | :-------: | | fengchj | In favor | 2023-05-10 2:53:25.0 +00:00:00 | | zhujf1989 | In favor | 2023-05-10 3:11:36.0 +00:00:00 |

[amye]

/check-vote

[git-vote[bot]]

Vote status

So far 36.36% of the users with binding vote are in favor (passing threshold: 66%).

Summary

In favor	Against	Abstain	Not voted
4	1	0	6

Binding votes (5)

User	Vote	Timestamp
rochaporto	In favor	2023-05-09 18:55:28.0 +00:00:00
TheFoxAtWork	In favor	2023-05-15 14:29:55.0 +00:00:00
nikhita	Against	2023-05-15 14:42:47.0 +00:00:00
kgamanji	In favor	2023-05-15 15:56:31.0 +00:00:00
cathyhongzhang	In favor	2023-05-09 22:27:54.0 +00:00:00

Non-binding votes (2)

| User | Vote | Timestamp | | ---- | :---: | :-------: | | fengchj | In favor | 2023-05-10 2:53:25.0 +00:00:00 | | zhujf1989 | In favor | 2023-05-10 3:11:36.0 +00:00:00 |

[amye]

/check-vote

[git-vote[bot]]

Vote status

So far 45.45% of the users with binding vote are in favor (passing threshold: 66%).

Summary

In favor	Against	Abstain	Not voted
5	2	0	4

Binding votes (7)

User	Vote	Timestamp
TheFoxAtWork	In favor	2023-05-15 14:29:55.0 +00:00:00
dzolotusky	In favor	2023-05-16 22:07:47.0 +00:00:00
nikhita	Against	2023-05-15 14:42:47.0 +00:00:00
rochaporto	In favor	2023-05-09 18:55:28.0 +00:00:00
kgamanji	In favor	2023-05-15 15:56:31.0 +00:00:00
cathyhongzhang	In favor	2023-05-09 22:27:54.0 +00:00:00
mauilion	Against	2023-05-16 15:01:48.0 +00:00:00

Non-binding votes (2)

| User | Vote | Timestamp | | ---- | :---: | :-------: | | fengchj | In favor | 2023-05-10 2:53:25.0 +00:00:00 | | zhujf1989 | In favor | 2023-05-10 3:11:36.0 +00:00:00 |

[git-vote[bot]]

Vote closed

The vote did not pass.

54.55% of the users with binding vote were in favor (passing threshold: 66%).

Summary

In favor	Against	Abstain	Not voted
6	2	0	3

Binding votes (8)

User	Vote	Timestamp
@rochaporto	In favor	2023-05-09 18:55:28.0 +00:00:00
@RichiH	In favor	2023-05-17 8:51:37.0 +00:00:00
@dzolotusky	In favor	2023-05-16 22:07:47.0 +00:00:00
@nikhita	Against	2023-05-15 14:42:47.0 +00:00:00
@cathyhongzhang	In favor	2023-05-09 22:27:54.0 +00:00:00
@mauilion	Against	2023-05-16 15:01:48.0 +00:00:00
@TheFoxAtWork	In favor	2023-05-15 14:29:55.0 +00:00:00
@kgamanji	In favor	2023-05-15 15:56:31.0 +00:00:00

Non-binding votes (2)

| User | Vote | Timestamp | | ---- | :---: | :-------: | | @fengchj | In favor | 2023-05-10 2:53:25.0 +00:00:00 | | @zhujf1989 | In favor | 2023-05-10 3:11:36.0 +00:00:00 |