Looking for maintainers

[Nevon]

This project was started by Tulio and then maintained mainly by him and I for a good number of years as we worked together on projects that used KafkaJS. Tulio no longer works at a company that uses KafkaJS, and while the company I work for does use KafkaJS, I myself don't. The amount of time and energy this project requires to be successful is more than I have the capacity for, given that it no longer really "scratches my own itch", and as a result I haven't been able to tend the garden for the past year or two.

Given that, I think the best thing to do is to put out a call for maintainers so that I can let go and give someone else the chance to take over the reins.

What you should know

• This package is used a lot, which means that changes must be well-considered and well tested. This is not the kind of project where you spend 30 seconds looking at a PR and then going "lgtm". As a maintainer I believe that helping land contributions is the most important thing you do, both for the technical well-being of the project but also to help attract new contributors and make existing ones stick around.
• The code-base itself is in a pretty good spot. Test coverage is good and I'd say the overall code quality is fine. What I see lacking most is a roadmap for future development and an idea of what KIPs have been implemented and not.
• There are no ongoing costs for CI or other infrastructure. We used to have a continuous long-running service that would test out beta releases of KafkaJS, which was dependent on an AWS sponsorship that has since expired. Everything else is running on Github Actions and Azure Devops Pipeline's free tier.
• The KafkaJS organization also contains a few supporting libraries. While it's great if you're willing to maintain those as well, I don't see that this needs to necessarily be the case.
• Becoming an expert at developing and using KafkaJS does open up opportunities for at least a side-gig if you want it to. Don't expect to quit your day job, but it can bring in some beer money if you're willing to spend some extra time helping folks out. Getting to talk to people in companies using KafkaJS has been quite the highlight, and I've gotten more than one job offer over the years because of it.
• I won't be 100% gone, at least in the mid term. My company still uses KafkaJS and so if there are security issues or features that we really need, I will most likely be involved to some degree. However, my goal would be to transition to a contributor more than a maintainer.
• To be perfectly clear, what this project needs is not more contributions, but project management in terms of adding new collaborators, making releases, deciding on what features to adopt and which not to, providing feedback to contributors etc. It's not about cranking out code but rather making sure that the project stays healthy over time, that new contributors have a good experience and that our users stay happy.

How to become a maintainer

First of all, I'm not actually the owner of this repository, so I can't hand out access to anyone. My idea would be to move the repository to the KafkaJS organization and add new maintainer(s) there. This will come with some practical things to sort out, like setting up NPM publish rights and so on, but it'll make it easier to manage the project in the long run. I haven't had a chance to run this past Tulio recently, but this was our plan when he stepped back some time ago, so I don't think it'll be an issue other than just taking some time to get set up.

That said, maintainership of a project like this isn't for someone's first open-source experience. While the license says that the code comes with no warranty, our users still place some trust in us, so I'm not about to betray that trust by handing the keys over to the first person willing to take them. If you do have some experience contributing to related open-source projects, or ideally even KafkaJS itself, then please leave a comment in this thread if you are interested in becoming a maintainer, along with some contact information.

I don't want to be a maintainer, but I still want to help out

That's great. The best thing you can do is probably help out with issue triage. Even if you don't have the permission to close an issue or merge a PR, it still helps whoever is maintaining the project a lot if someone has done most of the work already by the time they get around to reviewing an issue or PR. You don't need any special permission to do this, and never have.

What I would ask that you please don't do is @ me or Tulio with "Any updates on this?" or "When will this be merged?". I understand the frustration, but it causes a lot more stress and guilt than you might think, so please don't.

I don't want to take over as a maintainer, but I still want to get feature/bugfix X into production

Open-source is all about forking, so go right ahead. KafkaJS is licensed under MIT, so as long as you keep the attribution there's nothing preventing you from doing this.

[catYalere]

I can help you with that @Nevon, but mainly I want to help you with the supporting libraries that are in kafkajs org

[Nevon]

That's great. I saw you were interested in maintaining the confluent-schema-registry lib, so I've created a team with maintenance access to that repository and invited you as a member. Let's use the issue tracker there for working out what we need to do to make it possible to maintain.

[jwmonroe-outschool]

@Nevon My company Outschool is an extensive user of Kafka.js. We are evaluating potentially adopting maintenance of the project as a company with myself and @nuria as the primary contacts.

We had a couple questions about the nature of the role before committing to it. Would you be the right person to talk to about this? Would you prefer discussing these questions here in the issue or through some other medium?

[Nevon]

Here would be ideal, since if you have questions, I bet others will be wondering about those same things as well.

[t-d-d]

I would like to contribute but I can only commit a few hours per month.

On Wed, 30 Aug 2023, 06:00 Tommy Brunn, @.***> wrote:

Here would be ideal, since if you have questions, I bet others will be wondering about those same things as well.

— Reply to this email directly, view it on GitHub https://github.com/tulios/kafkajs/issues/1603#issuecomment-1698494707, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDLW5SRRNNCVJWYUHU7EFTXX3CGBANCNFSM6AAAAAA3ILH26Y . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[nuria]

@Nevon Could you outline a bit what is the commitment as a maintainer, for example: "node version upgrades twice a year which in the past has taken {this} long".

Many thanks for your contributions to this project over the years, we have benefited greatly.

[Nevon]

Could you outline a bit what is the commitment as a maintainer, for example: "node version upgrades twice a year which in the past has taken {this} long".

In my view the main things that are needed, roughly in order of importance, are:

• Reviewing and helping contributors get their PRs merged (or rejected if they are not aligned with the project direction). This depends wildly on how complex the contribution is - sometimes it takes 5 minutes and sometimes it takes several hours over many weeks. It sucks when people contribute improvements but no one is able to take the time to land the change. I would say expect a couple of hours per week on average, but it's not always a steady stream.
• Making regular releases. Historically we've had a stabilization period where we've run beta releases in production to catch issues that slipped through CI, and then made a "stable" release when we feel confident, but this could change to a more continuous release schedule or whatever the maintainers feel is the most sustainable. The release process is mostly automated, but it definitely has some rough edges that could use a bit of work. It's the kind of thing you spend a few hours on once and then it just keeps working for a few years, so not a huge deal, but still needs doing.
• Triaging issues. I don't believe it's necessarily the maintainer's job to debug people's issues, but it is good to at least go through and close invalid issues, label things correctly and so on, just to avoid the issue tracker being a jungle. Again, this is a rabbithole where you can spend hours and hours if you really want to get to the bottom of issues, and perhaps an hour or two a week if you just want to make sure that each issue has at least been looked at and closed/labelled appropriately.
• Related to the first point - providing guidance on what needs to be done in order to implement some feature. Sometimes contributors just open an issue describing the feature they want, then independently implement the solution and it's all good, but most of the time it's their first time contributing to a Kafka client and they need some guidance to figure out how to plan their feature or just get feedback on their idea before implementing it. This doesn't need to be done by a maintainer, but people tend to look to you for this type of support, so be aware that it can be a timesink.
• Maintaining node versions and dependency upgrades - frankly very little time. We don't have any runtime dependencies, so there's not much to worry about. Maybe a few hours per year, whenever older Node versions become unsupported and we need to update our CI to match.

[t-d-d]

Mining what has gone before, how about this for a starting point:

https://chat.openai.com/share/a68b60db-c7a2-493c-b9e8-3ff4502ca20e

On Fri, 1 Sept 2023, 15:46 Tommy Brunn, @.***> wrote:

Could you outline a bit what is the commitment as a maintainer, for example: "node version upgrades twice a year which in the past has taken {this} long".

In my view the main things that are needed, roughly in order of importance, are:

• Reviewing and helping contributors get their PRs merged (or rejected if they are not aligned with the project direction). This depends wildly on how complex the contribution is - sometimes it takes 5 minutes and sometimes it takes several hours over many weeks. It sucks when people contribute improvements but no one is able to take the time to land the change. I would say expect a couple of hours per week on average, but it's not always a steady stream.
• Making regular releases. Historically we've had a stabilization period where we've run beta releases in production to catch issues that slipped through CI, and then made a "stable" release when we feel confident, but this could change to a more continuous release schedule or whatever the maintainers feel is the most sustainable. The release process is mostly automated, but it definitely has some rough edges that could use a bit of work. It's the kind of thing you spend a few hours on once and then it just keeps working for a few years, so not a huge deal, but still needs doing.
• Triaging issues. I don't believe it's necessarily the maintainer's job to debug people's issues, but it is good to at least go through and close invalid issues, label things correctly and so on, just to avoid the issue tracker being a jungle. Again, this is a rabbithole where you can spend hours and hours if you really want to get to the bottom of issues, and perhaps an hour or two a week if you just want to make sure that each issue has at least been looked at and closed/labelled appropriately.
• Related to the first point - providing guidance on what needs to be done in order to implement some feature. Sometimes contributors just open an issue describing the feature they want, then independently implement the solution and it's all good, but most of the time it's their first time contributing to a Kafka client and they need some guidance to figure out how to plan their feature or just get feedback on their idea before implementing it. This doesn't need to be done by a maintainer, but people tend to look to you for this type of support, so be aware that it can be a timesink.
• Maintaining node versions and dependency upgrades - frankly very little time. We don't have any runtime dependencies, so there's not much to worry about. Maybe a few hours per year, whenever older Node versions become unsupported and we need to update our CI to match.

— Reply to this email directly, view it on GitHub https://github.com/tulios/kafkajs/issues/1603#issuecomment-1702872085, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDLW5VXX2LKP32CDL7C2Y3XYHYLJANCNFSM6AAAAAA3ILH26Y . You are receiving this because you commented.Message ID: @.***>

[frankdejonge]

Hi @Nevon, I'd like to volunteer some of my time as well. Although not in the NodeJS/TS community, I have a long track record of maintaining open source projects with large user-bases (Flysystem and EventSauce, ~1% of all packagist packages downloaded historically), so can bring that experience into the group that ends up maintaining this project. I can help out with PR reviews, releases, internal and external documentation, and general implementation of features.

[HeatherFlux]

I may be able to volunteer limited time for maintaining. I keep up with the kips for the most part since my company is built around kafka and heavily use this library. Also I would be happy to help with the migration to the kafkajs foundation repo, I have permission from my company to volunteer time.

[kalesh13]

I am planning to start using this library. Let's get a hang of this library, and I believe, I can contribute to the library in the long run. There is a huge pipeline of monolithic to microservices migrations with me. Hope for the best!

[biswajit415]

Hi @Nevon , This kafkajs library is heavily used in our organization. We are expecting to have new release of this library soon.

[HeatherFlux]

Heyo my company is building out an OSS org and we are forking your KafkaJS package so we can start applying KIPS to it and fix bugs. Since things are frozen here if you are a dev looking to contribute feel free to come over here for now. https://github.com/ExtendOSS/kafkajs

[nuria]

@HeatherFlux why don't you take over the main repo?

[HeatherFlux]

@HeatherFlux why don't you take over the main repo?

I would help but until Nevon or Tulios moves it to the kafkajs org and add maintainers my hands are tied. The fork is there as a stop gap until then

[ThisIsMissEm]

I'd agree that the first step in finding a new set if maintainers would be to move it to the @kafkajs organisation.

[eladhaim]

Hi @Nevon I can help, my company uses kafkajs heavily.

[HeatherFlux]

I'd agree that the first step in finding a new set if maintainers would be to move it to the @kafkajs organisation.

unfortunately the maintainers for Kafkajs also maintain the org.

Published first version https://www.npmjs.com/package/@helloextend/kafkajs Things that I believe need to be done and I'm open to suggestions/building a todo doc in the repo

• Testing in GHA pipelines, my company has tooling we built for this that I can move to the OSS org
• Clean up readme and docs and how to guides
• Start moving updating code based on recent KIPS as well as fix the current issue it has with KRAFT.
• I will also open up in the next weeks the Confluent AWS IaC tooling and api's that we have built, as well as the confluent-schema-registry package security updates.
• Set up a discord for discussion, peer reviewing, etc.

Once this can be moved into the KafkaJS org and when we have more maintainers we can refocus work there, I'm mostly hoping this is a temporary home.

[maxgr0]

@HeatherFlux @Nevon

We at @jucr-io also use KafkaJS heavily. Some things we've built here internally and would be willing to contribute to the project include (but are not limited to):

• A fixed schema registry client complying to confluents Wire format https://github.com/kafkajs/confluent-schema-registry/issues/152
• A wrapper to use KafkaJS and the schema registry client with effect-ts
• A wrapper to use KafkaJS and the schema registry client with fp-ts
• A huge collection of tooling for automating tests and releases with GitHub Actions

Furthermore, I can also help with hands-on coding as well as general maintenance tasks for the project. Another thing to mention: We're quite well funded and would also provide financial support for the newly formed org.

Cheers!🍻

[HeatherFlux]

@maxgr0 Sounds great to us. I will set up a discord this weekend if that works for folks who want a place to discuss and contribute.

[maxgr0]

@HeatherFlux sounds great! Do you post the link here then?

[samuelleach]

Sam from Confluent here.

I wanted to point out that Confluent is now officially supporting a javascript client with this early access release:

https://github.com/confluentinc/confluent-kafka-javascript

From the Readme:

confluent-kafka-javascript is Confluent's JavaScript client for Apache Kafka and the Confluent Platform. This is an early access library. The goal is to provide an highly performant, reliable and easy to use JavaScript client that is based on node-rdkafka yet also API compatible with KafkaJS to provide flexibility to users and streamline migrations from other clients.

[theblinkingusb]

Thanks @samuelleach - Looks like there's a kafkajs migration guide as well which could be helpful for users ITT.

[HeatherFlux]

Nice this is great news!

[frankdejonge]

Just for clarity on what they provide, it's an an API with (partial) compatibility with the configuration options of KafkaJS. The protocol implementation is done by rdkafka. I came to use KafkaJS primarily because it didn't rely on these bindings, as solutions like node-rdkafka were very buggy. Arguably, this was years back and that project wasn't owned by Confluent.

[AnkurGel]

@samuelleach Would you recommend https://github.com/confluentinc/confluent-kafka-javascript if I can use this in production? I would love to use confluent maintained nodejs library for Kafka. Is there a roadmap you can share for your project?

[kkapoorOS]

@Nevon Just wanted to check the state of affairs. Are you still looking for maintainers? I see a few comments on this thread, but not sure if a decision has been made. We're still interested and wanted to touch base cc @nuria @jwmonroe-outschool

@Nevon My company Outschool is an extensive user of Kafka.js. We are evaluating potentially adopting maintenance of the project as a company with myself and @nuria as the primary contacts.

We had a couple questions about the nature of the role before committing to it. Would you be the right person to talk to about this? Would you prefer discussing these questions here in the issue or through some other medium?

[EricMCornelius]

The largest benefit of this project vs. librdkafka wrappers is the pure JavaScript aspect. Avoiding compiling independently for every deployment platform is extremely important for some of us users.

Would be great to have an idea of future direction here if possible.

[emasab]

@EricMCornelius confluent-kafka-javascript already has prebuilt librdkafka binaries for Linux (arm64, amd64) macOS (aarch64) and Windows (x64) so you won't have to build and install the dependency on those platforms

[EricMCornelius]

@emasab - while I appreciate that, it does not solve the single executable deployment scenario. And there are still numerous potential compatibility issues (e.g. glibc vs. libc, corresponding versioning), platform testing, and potential security implications.

Needing to test and ship platform-specific binaries is a significant additional overhead for some use cases - especially for users who are using SEAs for deployment.

I'm just saying it's not a true drop in replacement - and users in this thread should be aware of the potential implications.

[emasab]

@EricMCornelius I understand that a pure JS implementation would be simpler, if it was only for JS. But languages are many, Kafka clients are complex and it's expensive to keep 5 of more different implementations up to date with latest KIPs and bug free.

We don't say that it's a drop in replacement, it's more similar to major release as it needs some code changes, for the imports and the configuration at least.

[EricMCornelius]

@emasab - that's fine - it's just important to make it clear to the community. And I appreciate the work you all are doing.

However, I would argue shifting from an actual javascript implementation to a native library with JS bindings is more than just a major update. It's a complete behavioral change. One not necessarily immediately clear from confluent-kafka-javascript as a name for the project, either.

There remains value in pure javascript implementations for a number of users - and therefore it would behoove us to get back to the main question in this issue - which is what the future maintainer options are for this project.

[sashahohloma]

In my project we use many popular technologies and only Kafka lacks support of native JS client. Also, KafkaJS supports custom partitioner and assigner, but official binding of librdkafka doesn't (there is no createPartitioner field in types and PartitionAssigners is enum except for a function in original KafkaJS).

In my experience, requirements are too strict for already cross-platform NodeJS framework and in some cases installation takes too much time or can't be done successful at all.

KafkaJS is very good library that suits my case and my team prefers to use KafkaJS instead of the official client. All I'm saying is that there is should be a choice between native NodeJS library and official librdkafka binding.