loggie-io / loggie

A lightweight, cloud-native data transfer agent and aggregator
https://loggie-io.github.io/docs-en/
Apache License 2.0
1.26k stars 167 forks source link

feat: support rocketmq sink #530

Closed hansedong closed 1 year ago

hansedong commented 1 year ago

Proposed Changes:

Which issue(s) this PR fixes:

The main purpose of this PR is to support the RocketMQ Sink.

In some scenarios, RocketMQ has higher write performance than Kafka when there are a large number of topics. Our company's internal MQ mainly uses RocketMQ.

I have used and verified the rocketmq Sink function in my K8s cluster, and it works properly. I am looking forward to adding this Sink into Loggie (so that I don't need to maintain a separate version of Loggie within the company).

Regarding unit testing, I noticed that there aren't many examples for other Sinks in this area. If needed, I am happy to supplement them but may require some examples and assistance. Thank you very much!

Additional documentation:

I don't know where to add the document 😓, please tell me.

ethfoo commented 1 year ago

Welcome! Please submit your document PR to this link: https://github.com/loggie-io/docs. I will merge it when we release a new version.

ethfoo commented 1 year ago

@hansedong The CI build has failed. Please run the 'make fmt' command locally and resubmit your changes.

hansedong commented 1 year ago

@hansedong The CI build has failed. Please run the 'make fmt' command locally and resubmit your changes.

It has been resolved.

hansedong commented 1 year ago

@ethfoo I've been committing code for quite some time now, and up to this point, the code is working fine. However, my branch differs a lot from the main branch. Just now, I've committed again to resolve some code conflicts. Please approve the workflow. I wanted to ask, is there a plan to merge this into the mainline?

ethfoo commented 1 year ago

@hansedong Please take a look at reivew comments above.

hansedong commented 1 year ago

@ethfoo

Welcome! Please submit your document PR to this link: https://github.com/loggie-io/docs. I will merge it when we release a new version.

I've already submitted a PR for loggie/docs.

The CI build has failed. Please run the 'make fmt' command locally and resubmit your changes.

In addition, I've followed your instructions, executed the make fmt command and resubmitted the PR (to avoid introducing too many meaningless commits, I force pushed the commit).

Please take a look at reivew comments above.

I'm sorry, could you clarify which one you're referring to?

ethfoo commented 1 year ago

@ethfoo

Welcome! Please submit your document PR to this link: https://github.com/loggie-io/docs. I will merge it when we release a new version.

I've already submitted a PR for loggie/docs.

The CI build has failed. Please run the 'make fmt' command locally and resubmit your changes.

In addition, I've followed your instructions, executed the make fmt command and resubmitted the PR (to avoid introducing too many meaningless commits, I force pushed the commit).

Please take a look at reivew comments above.

I'm sorry, could you clarify which one you're referring to?

@hansedong

image

hansedong commented 1 year ago

@ethfoo I'm very sorry that I didn't see the code suggestion you mentioned before. I am currently working on the two points you mentioned and thank you for bringing them to my attention.

hansedong commented 1 year ago

@ethfoo

The support for RocketMQ sink has been developed and basiclly completed. The commits mainly includes the following aspects:

  1. Support for RocketMQ sink.
  2. Dynamic rendering of tags in addition to topics in RocketMQ sink.
  3. RocketMQ client SDK has its own default logging component. I have made an adaptation so that RocketMQ client can use Loggie's logger, in order to maintain consistency in the way logs are outputted, log formats, and log levels of Loggie.

I have already deployed and conducted functional testing in my own two sets of K8s environments, everything is working fine. Also, the PR https://github.com/loggie-io/docs/pull/160 is updated.

Please kindly review it again and if possible, provide some constructive feedback. Thank you.