About the jq's release process (Was: Is jq is still alive/maintained ?)

[tst2005]

Hello,

jq is a already powerfull enough for most of use. it does not need new release too often but ... There are lot of PR for bugfixes and interesting improvements.

is @stedolan still maintaining jq ? if it is a "dead" project, Am I alone to think about a fork ?

Regards,

[wader]

@tst2005 Ok 👍

Doh https://github.com/stedolan/jq/blob/f9afa950e26f5d548d955f92e83e6b8e10cc8438/AUTHORS#L5 it's @nicowilliams :)

[eitsupi]

They certainly seem to be here. https://github.com/jqlang/jq https://github.com/jqlang/jqlang

[chickenandpork]

FWIW, I'm an SRE-aligned eng, drawn to the CI/CD side of things, and contributed to autotools long ago. If there's space to help the CI/CD, I'd love to submit changes there to help that along, and it could accelerate the feature-side. If w can get some donated server-time on Windows, I've got some flavors of Mac, we could get automated test builds for lower/zero cost.

[liquidaty]

Hey everyone, how do we make this happen? I totally agree, it would be great to have a new fork that can more quickly incorporate various updates that are now backlogged.

It seems like that is what https://github.com/jqlang/jq aims to be, correct?

But in that case, I am confused: why does that repo seem to not have actually made any changes / updates past what the original jq repo has? Isn't the whole purpose of the new repo, to be able to catch up with the backlog of updates that the original ones has not kept up with? I get it that some updates are just feature requests at this time, but many others are actual changes that are either sitting there in pull requests or otherwise straightforward to begin chipping away at.

If https://github.com/jqlang/jq is not serving that purpose, then can we either:

• make it serve that purpose
• identify a different repo fork that serves that purpose, or
• create a new repo to serve that purpose?

[liquidaty]

@pkoppstein re your comments:

[...] we've just about reached the point where it would indeed be reasonable to create a "successor fork", that would only incorporate changes that are made with a view to having them incorporated in the stedolan repository. Ideally, such changes would also either reflect changes that have already been adopted in gojq, or would be acceptable to @itchyny for potential adoption in gojq

Obviously, maintaining jq at a high level presents a wide variety of challenges. Perhaps the place to start is to ensure that the initial team of official maintainers has all the required technical skills?

I agree in principle with both of these statements, other than I think it's fine if someone has some and not all the requisite skills (for example, if @chickenandpork can contribute not a lick of C and nothing other than CI/CD shell scripts, I'll take that all day long). I'm not sure how to implement in practice. I suppose we can probably start with half a dozen people whose contributions clearly demonstrate that. Or we could just ask who wants to volunteer-- it may turn out that all our volunteers meet the desired bar. We could also establish a simple agreed-upon approach for making decisions.

Finally-- and this isn't mutually exclusive with other options, and is probably a good idea anyway-- we could begin as a group to review and roughly prioritize PRs (perhaps based on bug vs enhancement, number of likes, etc). From there, we could come up with some set we think should get incorporated ASAP, and then use that as a basis for a near-term jq 1.7 goal. Once that's done, it might be clearer how top optimally structure the admin / maintainer team.

If I don't hear any objection, perhaps in a few days I'll open a "calling anyone who wants to opine on 1.7" issue and see if we can't put some flesh on this. That said, I'd be very happy to hear objection, especially of the "we are already doing this so you don't have to" sort...)

cc: @stedolan @nicowilliams @wtlangford @dtolnay @leonid-s-usov @wader @tst2005 @chickenandpork

[pkoppstein]

@liquidaty wrote:

I think it's fine if someone has some and not all the requisite skills

Of course. Evidently you misunderstood my suggestion, for when I wrote:

Perhaps the place to start is to ensure that the initial team of official maintainers has all the required technical skills?

I simply meant that the combined skills of the team members should be sufficient. My point really is that the necessary set of skills comprises more than just git, github, C, jq internals, and the jq language. Some other relevant realms of knowledge and/or experience that come to mind are: Windows, bison, make, the testing infrastructure (./tests), the documentation infrastructure (including Python and jinja2), the release process, cross-compilation, oniguruma, and the ICU package (decNumber).

Any others?

Regarding the fork - I would suggest considering forking stedolan jq and promoting it immediately to Version 1.6.1 so that it's clear what the "baseline" is. Then the bugs that have already been fixed via PRs or that have been fixed in gojq could be 1.6.2 or the basis for 1.6.2.

Regarding https://github.com/jqlang/jq - I am not sure what the status of that repository is. Since the last commit was made by @nicowilliams perhaps he could chime in?

[chickenandpork]

for example, if @chickenandpork can contribute not a lick of C and nothing other than CI/CD shell scripts

Chicken's more than willing to just bang on CI/CD all day :) (and is flattered by the call-out). There's some things I might add, but generally I'm personally motived for CI/CD tasks, documentation, test-coverage, and I'm willing to volunteer some hardware for CI/CD (Mac and Linux).

[wader]

@liquidaty Hey thanks for organizing. I'm in to help out with whatever needs to be done, and creating a new issue to figure out a plan sounds good.

@mrienstra Did you hear back from @nicowilliams? i pinged in the email thread but no response yet.

[mrienstra]

@wader, alas no, fingers crossed! Might be time to focus efforts on a fork. Hopefully in the near future that will no longer be necessary. Oh cool, just saw #2550, looks like a great start! 👍

[jpmckinney]

FWIW, I'm trying jaq and it is faster and does the complex queries I have (I haven't checked whether 100% compliant): https://github.com/01mf02/jaq

[pkoppstein]

Regarding jaq, @jpmckinney wrote:

I haven't checked whether 100% compliant

jaq is very impressive in many ways, but although it is substantially compliant, it is not, and does not strive to be. The documentation is in fact very up-front about this, so the above-quoted comment is a bit disheartening.

By contrast, gojq is substantially compliant with jq except mainly for bugs and certain "undocumented" features (e.g. _nwise/1) and "implementation" matters, notably (on the plus side) integer precision and (on the negative side) memory management.

[eitsupi]

This issue mey be outdated? (This repo is now jqlang/jq)

[jpmckinney]

Looks like that change happened within the last 24 or 48 hours, yes.

Edit: https://github.com/jqlang/jq/issues/2594#issuecomment-1566198375

[kloczek]

So looks like whole repo with all data/metadata has been transferred to new maintainer 🤔

[xinthose]

He's dead. We need a fork for someone who can maintain it

he hasn't responded or made a commit in over two years, so you could be right

[nicowilliams]

He's dead. We need a fork for someone who can maintain it

he hasn't responded or made a commit in over two years, so you could be right

We have:

• a new home for jq (jqlang/jq) -- a new org, so that we're not dependent on just one person
• new maintainers
• new org owners and admins
• work being done towards making a release

Yes, the old maintainers (myself included) went AWOL, but we're not dead. You can thank @owenthereal for pushing to do this, and @stedolan for transferring the old stedolan/jq repository to the new jqlang org.

[nicowilliams]

So looks like whole repo with all data/metadata has been transferred to new maintainer 🤔

Same maintainers but with more new maintainers, because we the old maintainers were not responsive.

[nicowilliams]

Regarding jaq, @jpmckinney wrote:

I haven't checked whether 100% compliant

jaq is very impressive in many ways, but although it is substantially compliant, it is not, and does not strive to be. The documentation is in fact very up-front about this, so the above-quoted comment is a bit disheartening.

By contrast, gojq is substantially compliant with jq except mainly for bugs and certain "undocumented" features (e.g. _nwise/1) and "implementation" matters, notably (on the plus side) integer precision and (on the negative side) memory management.

I'm bored of C -- it's a very dangerous language. I think Rust is a pretty good choice of a language for a rewrite. Just looking at the README for jaq I see some issues that should be addressed, but nothing serious, and if jaq can grow up to replace this implementation of jq, I think that'd be fantastic.

[jpmckinney]

I think there will always be a place for jq. From jaq's readme:

jaq currently does not aim to support several features of jq, such as:

• Paths
• Modules
• Dates
• SQL-style operators
• Streaming

I assume jaq is fast not only thanks to the author's efforts to make the implemented features faster, but also thanks to the architecture being simpler by excluding some features. It might not be possible to maintain that performance and implement all of jq's features (unless the slower features are made to use an entirely different code path, but then why bother if jq already exists for those needs).

[liquidaty]

I would happily use jaq, if it can support wasm targets (more specifically, can be linked as a library into a wasm module). Maybe this is more of a general Rust matter than jaq; either way I only found the question raised but not answered

[davidfetter]

Regarding jaq, @jpmckinney wrote:

I haven't checked whether 100% compliant

jaq is very impressive in many ways, but although it is substantially compliant, it is not, and does not strive to be. The documentation is in fact very up-front about this, so the above-quoted comment is a bit disheartening. By contrast, gojq is substantially compliant with jq except mainly for bugs and certain "undocumented" features (e.g. _nwise/1) and "implementation" matters, notably (on the plus side) integer precision and (on the negative side) memory management.

I'm bored of C -- it's a very dangerous language. I think Rust is a pretty good choice of a language for a rewrite. Just looking at the README for jaq I see some issues that should be addressed, but nothing serious, and if jaq can grow up to replace this implementation of jq, I think that'd be fantastic.

The latest organizational Rust issue makes that seem like a bad bet, however technically sweet the technology may be.

I get that when you've been doing a lot of C, those B&D languages can start to look great, but it really is a "the grass is greener on the other side of the fence" situation, or at least it has been over the past few decades.

[wader]

Being jq compatible would probably also include re-implement all of jq's CLI argument and behaviours (input handling, exit codes etc) and also in some cases implement some behaviours that might be considered bugs, ex 1+3 as $a | ($a*2) is 8 with jaq, 7 with jq. But i do think it's possible but would be a lot of work.

BTW the author or jaq @01mf02 has written a paper about it https://arxiv.org/abs/2302.10576

[flying-sheep]

I would happily use jaq, if it can support wasm targets (more specifically, can be linked as a library into a wasm module). Maybe this is more of a general Rust matter than jaq

One of the reasons Rust became popular is that it was easy to compile to WASM from very early on. Rust remains one of the best choices to do what you want, I bet one could very quickly set up a small wrapper around jaq that exposes it to JS, e.g. using https://github.com/rustwasm/wasm-pack

The latest organizational Rust issue makes that seem like a bad bet, however technically sweet the technology may be.

Does it though? I’m so bored by any predictable outbreak of internet drama being FUDded into the doom of humankind. Rust is used in the Windows kernel. It’s not going away.

[eitsupi]

Obviously the original problem with this issue has been resolved, so why not create another issue about the rewrite to Rust and close this issue?

[itchyny]

Although we're moving forward on the new jqlang org with new maintainers, we haven't go through the entire releasing process, so let's keep this open.

[tst2005]

Although we're moving forward on the new jqlang org with new maintainers, we haven't go through the entire releasing process, so let's keep this open.

The survive of jq seems now more assured than at the beginning of this thread ;-)

[01mf02]

I would happily use jaq, if it can support wasm targets (more specifically, can be linked as a library into a wasm module). Maybe this is more of a general Rust matter than jaq; either way I only found the question raised but not answered

@liquidaty, the question you have referenced was about having jaq as a "WebAssembly library wrapped in JS on npm". I have never done the "npm" part, but I have already compiled another Rust program of mine to WebAssembly and used it from JS. So this part is definitely doable, also for jaq. I would be really happy if somebody would try it, because having something like jqplay.org for jaq would be quite cool. In particular, it would allow processing all data on the client instead of sending it to the server.

[colindean]

I wish I could react to the close notification so this comment will have to do. Great work, new maintainer team!

[nicowilliams]

About the jq's release process (Was: Is jq is still alive/maintained ?)

Yes! jq has been revived, and now is indeed alive and being maintained again. You can thank... a lot of people for this, including those who asked what was up and even threatened a fork.

[tst2005]

After more than 2 years, I'm really happy to see this issue that can now be closed without hesitation ;-)

[nicowilliams]

After more than 2 years, I'm really happy to see this issue that can now be closed without hesitation ;-)

Just so you know, your efforts to fork jq helped revive jq. Thanks!

[ceresdarkmatter]

Dev Team - Thank you very much! Amazing job!

[jbrains]

This is really good timing for me. I wanted modules last week and I'm using jaq, so suddenly I wanted to use jq again. :)