search

jqlang / jq

Command-line JSON processor
https://jqlang.github.io/jq/
Other
30.53k stars 1.58k forks source link

About the jq's release process (Was: Is jq is still alive/maintained ?) #2305

Closed tst2005 closed 1 year ago

tst2005 commented 3 years ago

Hello,

jq is a already powerfull enough for most of use. it does not need new release too often but ... There are lot of PR for bugfixes and interesting improvements.

is @stedolan still maintaining jq ? if it is a "dead" project, Am I alone to think about a fork ?

Regards,

carlsmedstad commented 3 years ago

Also wondering this. It would be a shame if it stopped being developed and maintained. Maintainers, is there anyway to help you out?

wtlangford commented 3 years ago

Hey, so I'm still around- the main issue is that I've been super busy with work and dealing with life this last year that it's been especially hard to find the time/energy to devote to jq.

We'd definitely appreciate help- I've got a couple small changes floating around that I need to finish up which should improve some build behaviors (I think I can make the CI builds more reliable- they keep breaking. Should also be able to fix some things around the math functions that got brought up "recently"). I'll also try and dig up that list of things that needed finishing in order to cut the next release and see if I can set up some issues for it and a milestone so that it's clear what needs doing. (Depending on what's left on that, I might just say we should cut a release and schedule myself some time to do it).

Beyond that, I need to go review all the open PRs- I know I've seen some good stuff come through my email- and also try to clean up the issue tracker some.

tst2005 commented 3 years ago

Thanks for your answers.

@carlsmedstad : IMHO, I don't think it is a shame if @stedolan can not or don't want maintain jq anymore. I prefer saying a big thank for the work already done.

Today I see the stedolan's jq repository has 19.3k stars, 1.1k fork, 121 PR, 574 issues. 121/1100 --> ~10% of forks are PR 574/1100 --> ~50% of forks equal to an issue (bug/ask/feature request/...)

There different solutions to get a projet maintained, :

For now I don't read people answering "I forked and I can maintain jq", @wtlangford you are the closest one but you already told us you are busy.

I'm in the same case. I want a maintained jq, I can contribute but I don't have enough time to be the only one to maintain jq.

When nobody has the time to maintain the projet, we can try to make a github "organisation" like "jq-community" or "jq-maintainers". It imply to work together (it is not always easy).

My personnal goal and possible contribution is targeted on jq builtin

I'm currently using jq 1.5, the only interesting thing I take from jq 1.6 is some builtin (made in jq language it self) then I took them and load on jq 1.5 before using them.

The most work I'm doing around jq is provide a set of usefull functions/samples for my colleages and myself. Some of them could be jq builtin, some other not. jq is powerfull but not always easy to use.

I want more sample in documentation for builtin and more wiki cookbook) for functions that is not built in but interesting enough to exist.

For me, there are a few number of missing feature in jq. The only main one for me is the ability to check if a function is already defined or not. For example: It could be usefull to be able to define a math pow function (made in jq language), only if a "C native" pow not already available.

itchyny commented 3 years ago

I would like to help maintaining this project if you could invite me as a collaborator. Maybe @pkoppstein would be willing to help, too. I have already submitted some pull request, including the fix for the CI (#2141, #2193). Based on my experience of gojq, I believe I have throughout knowledge of jq filters. But I'm afraid there are many difficulties in this project though; changing the behaviors will likely to break some scripts in the world, even if it is likely broken behaviors (like leaf_paths misjudging falcy values #1163 and index returning bytes offsets #1430). But still I can help reviewing document fixes or closing questions.

carlsmedstad commented 3 years ago

@wtlangford Fully understand that, really appreciate the work you do. I'm not too familiar with the project yet but I'll try to get to it this weekend and see where I could help out.

@tst2005 Sorry, I did not intend to come off as negative or demanding. I just wanted to get across that I appreciate the project and would like to help.

wader commented 3 years ago

@wtlangford Thanks for all hard work. Im also willing to help out with review, documenting, testing and whatnot

tst2005 commented 3 years ago

I create a https://github.com/jqmaintainers organisation, @wader @carlsmedstad @itchyny @wtlangford I invited you all to join. EDIT: I also invited @leonid-s-usov @nicowilliams @pkoppstein, any help is welcome ;)

leonid-s-usov commented 3 years ago

Hi!

I feel bad that at some point I fell out of the work that I've started on a couple of core developments here. Back then I had some spare period between jobs but since I've entered my current position I haven't been having any spare time/brain to continue working on this.

I am very sad to see how this amazing project is struggling currently and I would encourage @nicowilliams and @wtlangford to seriously consider picking a responsible maintainer who will be able to put the required effort into moving this forward.

Personally, I am still planning to find some time to finalize my work on a very promising feature that Nico has started. It is going to open the way to extending JQ with custom filters and producers, like fetching stuff from APIs, or running queries against DBs from within the jq script.

davidfetter commented 3 years ago

That maintenance is landing on one person who may or may not actually have the time because they have other obligations such as a day job is precisely the problem. Making a GH organization and seeing to it that we have enough people who can actually do the work would go a very long way in the right direction.

pkoppstein commented 3 years ago

@tst2005 - Some thoughts in response to your post:

1) jq has had pow/2 since 1.5.

2) Since the (official) Cookbook has become a collection of short essays rather than a "library" of ready-to-go functions, one thought would be to create a new page on the wiki for code snippets. Perhaps "library" or "snippets" or "supplemental builtins" would be a suitable title?

3) There are currently 475 jq entries (mostly worked examples) at rosettacode.org (http://rosettacode.org/wiki/Category:Jq).

FractalTree

itchyny commented 3 years ago

Decision such as migrating to an organization should happen after deliberation of current maintainers.

tst2005 commented 3 years ago

@pkoppstein 

1. jq has had pow/2 since 1.5.

I know it is was just a simple example. It can be use to make compat from 1.5 to 1.6 for sql-like function.

2. Since the (official) Cookbook has become a collection of short essays rather than a "library" of ready-to-go functions, one thought would be to create a new page on the wiki for code snippets.  Perhaps "library" or "snippets" or "supplemental builtins" would be a suitable title?

For me there are 3 kind of functions:

3. There are currently 475 jq entries (mostly worked examples) at rosettacode.org (http://rosettacode.org/wiki/Category:Jq).

I'm not sure to understand. What you wanna do with that ?

tst2005 commented 3 years ago

@itchyny

Decision such as migrating to an organization should happen after deliberation of current maintainers.

He didn't react/commit for years. I supposed he does care of jq anymore. I will be happy if he answer us !

wtlangford commented 3 years ago

@itchyny

Decision such as migrating to an organization should happen after deliberation of current maintainers.

He didn't react/commit for years. I supposed he does care of jq anymore. I will be happy if he answer us !

So these days, @nicowilliams and I are the two maintainers. I'll chat with him about it and see what he thinks

For me, there are a few number of missing feature in jq. The only main one for me is the ability to check if a function is already defined or not.

While slightly off-topic for this thread, I do want to point out that this isn't really possible, due to how jq's internal linker works. When we're parsing your code, we just note that you referred to a symbol by name. Later, we do a pass through to try and link your symbols to already defined things. If we can't find a symbol by that name, linking fails. So there's not really a way to do that, without introducing some kind of preprocessor, and that's not something I'm particularly enamored of.

It can be use to make compat from 1.5 to 1.6 for sql-like function.

So one of the nice things about jq is that most of our standard library is defined as jq code (https://github.com/stedolan/jq/blob/master/src/builtin.jq). So as long as it doesn't actually depend on a new language feature, you can just copy over the needed library functions.

That said, per my comments earlier, I'm going to spend some time today doing some cleaning/organizing of issues/PRs so that we can see what's left before a 1.7 release.

pkoppstein commented 3 years ago

@tst2005 -

  1. I mentioned rosettacode.org because you wrote:

I want more sample in documentation for builtin and more wiki cookbook) for functions that is not built in but interesting enough to exist.

  1. Regarding builtin.jq vs extended.jq vs misc.jq - it takes quite a lot of joint discussion and time before anything new makes its way into builtin.jq, and whether an extended.jq will ever make it into the official distribution has not yet even been discussed to my knowledge, so it seems to me that one or perhaps two new pages on the wiki would be helpful for experimentation and discussion, as well as perhaps being quite useful.

  2. Regarding your comment:

He didn't react/commit for years.

You are perhaps referring to the original creator of jq, but there have been other official maintainers, and of course at least one of them (thank you @wtlangford!) is participating in this thread.

wtlangford commented 3 years ago

Regarding builtin.jq vs extended.jq vs misc.jq - it takes quite a lot of joint discussion and time before anything new makes its way into builtin.jq, and whether an extended.jq will ever make it into the official distribution has not yet even been discussed to my knowledge, so it seems to me that one or perhaps two new pages on the wiki would be helpful for experimentation and discussion, as well as perhaps being quite useful.

We have actually discussed it some, but we're not sure how we want to handle it- anything like that would be a breaking change for a lot of existing scripts, and that's something we'd like to avoid. The main reason jq-1.6 is so much slower than 1.5 is due to the sheer number of builtins we added to it and the cost that incurs at load-time. Splitting less-commonly needed builtins out into builtin modules would go a long way towards helping (as we wouldn't load those modules unless you ask for them).

But, as I said- that's going to be breaking behavior and I'm hesitant to commit to it. Beyond that, we had a great contribution back in 2019 (yes, I know our release schedule is slow) that significantly improved startup times back to approximately the performance of jq-1.5, so it's a bit less pressing to my mind.

kasperk81 commented 3 years ago

That maintenance is landing on one person who may or may not actually have the time because they have other obligations such as a day job is precisely the problem. Making a GH organization and seeing to it that we have enough people who can actually do the work would go a very long way in the right direction.

This is a solid point, it does not mean to blame any one individual just a fact about the nature of things in open source, people move on, lose interest, life happens and have (real) morning jobs. An organization with multiple entities makes much sense for project like jq to evolve. Lets face it, it is simply not one individual's job to maintain this huge popular project at this point.

chb0github commented 3 years ago

Eventually all great projects outgrow their original maintainers (hello, Linux). I definitely would encourage whomever still owns this to delegate and allow the community to grow.

I could see a whole community with several branches springing up. I, for one, would love to see a whole repo dedicated to add-on functions that can be installed.

Please consider growing the community and releasing soon

psacawa commented 3 years ago

Can I second those who ask for a new release from the new maintainers? jq has had optional else clause in if statement since 4f6045a in Feb 2020. Since then it doesn't appear in any release. It's pretty strange that such a useful feature is locked away in the master branch where the people who install from package repository can't use it.

chb0github commented 3 years ago

To the owner of this repo: If you simply don't have the time to even delegate, then donate it to something like the apache foundation or the eclipse foundation, that has a great record of running open source projects

Is the owner even still alive? @stedolan - he hasn't made a single comment on this thread

cheister commented 3 years ago

@wtlangford Any updates on doing a release?

chb0github commented 3 years ago

He's dead. We need a fork for someone who can maintain it

vintnes commented 3 years ago

This comment is a rant; accept my apologies and skip it.

@tst2005 @chb0github

Fork it then. This was a complicated project before the portability concerns and the massive adoption. It's a programming language that most of its users don't know is a programming language. It's a functional shell-oriented streaming DSL used by people who write AWS docs and people who think Python is bloat. Its maintenance requirements are unique. Anyone who's capable of wrangling libjq is also capable of making a quarter million a year bumping the version control on some embedded firmware.

I'm on record begging for a new release but if you look at the project history the schedule isn't weird and the build isn't expensive. Emailing everyone subscribed to this repo the same entitled comments over and over does not make anything happen faster. There are some tantalizing features in development but I, for one, would rather see the scope protected than the growth rate explode.

@wtlangford Sorry for contributing to the trash fire

chb0github commented 3 years ago

I depart. Clearly nothing productive here.

tst2005 commented 3 years ago

For a more productive comment...

@wtlangford I supposed you are little busy and do what you can with the free time you have (like most of us!)

1. the release process

Can you consider to make Release candidate ? jq 1.7rc1, 1.7rc2, 1.7rc3 ... before the final jq 1.7 release ? It could be a way to get more feed back about feature, benchmark, etc. If you don't have time, you should maybe delegate the release candidate to others peoples. and keep the final release for your own. You probably be the one to be able to make a clean official release, but we can try to make some release candidate (or "beta" ?) to help you to move to the next release and test what it should be included or not.

2. multiple version support

I'm using jq over the debian package distribution currently I'm using jq 1.5 I'm loading some interesting jq 1.6 built-in function when I need it. I'm not sure but I read somebody told jq 1.6 is less performant than jq 1.5 If it's true, how to deal with this ? Can we consider to maintain multiple branches ? and make backports release, include recent features in an older version My personnal need should only be a jq 1.5 with the jq 1.6 built-in, that can be named jq 1.5.1 or jq 1.5.0+bpo1 ... I just launch the idea. I'm don't know if it is a really good way to do.

Regards,

kasperk81 commented 3 years ago

I really liked the old title and intent of this issue, since it was speaking to what most subscribers of this issue are interested in. For various reasons, jq is a unique and useful tool in this space available across many platforms, which is why I think it deserves dedicated maintainers (preferably under a GitHub org rather than individual user). For sure, we can talk about releases and versioning, once maintainership story is settled down (that is a separate issue).

diafour commented 3 years ago

@tst2005 jq 1.6 is drastically less performant. See #2069. A fix is in master for a long time #1817.

1.7rc.1 is a great idea!

eitsupi commented 2 years ago

Any update?

Paszymaja commented 2 years ago

Hello?

tst2005 commented 2 years ago

Maybe in 1 month for the anniversary date of the issue ? :D

tst2005 commented 2 years ago

happy issue birthday ! https://github.com/jqmaintainers/jq updated, I consider again seriously to fork jq...

kasperk81 commented 2 years ago

yes, lets do that: https://github.com/jqmaintainers/jq/issues/1

matei-dragu commented 2 years ago

yes please - this is becoming embarrassing would it be possible to include the fork in Homebrew / DEB / RPM indexes?

reegnz commented 2 years ago

yes on fork... This is too good of a project to be abandoned like this. But if you're seriously considering a fork, please make the https://github.com/jqmaintainers members public, I don't think we want another one-man-show as a fork, and visibility into who manages things is important for getting approval/adoption. Also consider spreading the responsibility so at least 2-3 people are owning that org.

reegnz commented 2 years ago

@ceresdarkmatter I think it's possible to replace jq with the fork and include it in homebrew/ubuntu/fedora/arch/etc., but first you need a convincing argument that this is not a hostile fork, but a necessary one (eg. original got abandoned). This github issue is definitely a good case for the abandonment argument. Also there has been talk about breaking changes: I think that would be unacceptable for any of those indexes.

tst2005 commented 2 years ago

I created the https://github.com/jqmaintainers (the 30 april 2021) the goal was to break the "one-man project" to move it to many.

I drop the jqmaintainers idea quickly because I read https://github.com/stedolan/jq/issues/2305#issuecomment-830660167 at this moment jq seemed to have 2 maintainers. I had good hope for a new release!

Like many of us, I make lot of things and don't have so many time to maintain jq. if @wtlangford and @nicowilliams don't have enough time to maintan jq I will not blame them!

I'm also a busy man, I will don't have lot of time to maintain jq, but I think i will be able to do more than what is done since the last year : nothing ?. After 1 year of wait, I think it is time to find a better solution.

@reegnz I'm totally agree with you. It is definitively not a hostile fork : I'm trying to find a solution.

@reegnz the current members are @pkoppstein and @wader . I invited 7 peoples in 2021 (chose from people that make answer to this issue) 2 of them join the organisation. i did not realized that it is not public: it seems a github per-people choice. I did not find the way to put them public. I only change my own setting.

vmgolovin commented 2 years ago

Maybe, it would be better to add more collaborators than just @wtlangford and @nicowilliams? For instance, @itchyny has several open PRs, some of them fixing major bugs, like this one: https://github.com/stedolan/jq/pull/2133. He seems to be pretty active with his gojq and has a pretty expressive self-description in his profile :-)

tst2005 commented 2 years ago

@vmgolovin I'm agree without. I already try to group all "motivated" peoples: 3 joined, 5 miss the invitaton. I just relaunched all expired invitation for :

Feel free to join ;)

wader commented 2 years ago

I think @itchyny has shown lots of motivation and knowledge, so he would get my vote if he would agree to lead. I'm also happy to help out with various things.

01mf02 commented 2 years ago

I also think that @itchyny would be a great leader for jq. He has uncovered a number of bugs in my jq reimplementation, jaq, and I'm sad to see that his pull requests to jq that fixed serious bugs were never integrated.

jbrains commented 2 years ago

I'm new here, so please forgive the question.

What needs to happen now for this community to successfully take over maintaining jq? I see some potential for resolution, so I'm not sure what this community is waiting for.

Are things happening, but slowly, because of holidays and busy times? or are things stalled because the next steps aren't clear?

Thank you.

Uj947nXmRqV2nRaWshKtHzTvckUUpD commented 2 years ago

you are assisting to the defintion of "open source". you believe it is descentralized, but in reality it is not. sounds like another bell for me

jbrains commented 2 years ago

So far I have 1 reaction of frustration. Is there more?

tst2005 commented 2 years ago

@jbrains Hello, since I started this thread,

wader commented 2 years ago

@tst2005 Hi, maybe some misunderstanding, I don't have collaborator access to stedolan/jq

But i'm still available to help out. Currently mostly try help triage new issues, fix documentation and report bugs.

tst2005 commented 2 years ago

@wader, sorry my brain made a mistake! it was @wtlangford (starting by a "w" too)

@wtlangford,

wader commented 2 years ago

@tst2005 :) no worries!

jbrains commented 2 years ago

@tst2005 Merci bien de nous avoir mis à jour. :)

Yes, I can continue to use jq as it is. I will also experiment with using https://github.com/01mf02/jaq as an alternative.

even if we fork jq elsewhere, the stedonlan's repository will continue to be the most stared/viewed. People will continue to send issue/RP here. It will spend years to reach the point of visibility to be the place were jq really lives.

Yes, so we should start now! :) If most people are happy to use v1.6, then they can continue; and if the "new primary fork" makes some important improvements, then it will become the standard. I believe that we can simply start. We only need to know who will try to become the leader. :) We can leave an issue here that notifies the world that there is a new fork for anyone who wants to fix problems and add features.

If @wtlangford says that he is too busy, then is it true that the new "community organization" can operate until someone revives this version of the project?

I want to be clear: I can't contribute code to this project, but I can contribute some energy and enthusiasm and maybe some documentation. @wader has worked with me before and maybe he can confirm that I can provide good energy. ;)

wader commented 2 years ago

I stumbled up one this thread https://news.ycombinator.com/item?id=33102129 and i'm guessing cryptonector is @tst2005? if so did you managed to get ahold of stedolan?

tst2005 commented 2 years ago

@wader nope, I'm not cryptonector (I'm not on ycombinator.com).