search

jqlang / jq

Command-line JSON processor
https://jqlang.github.io/jq/
Other
30.54k stars 1.58k forks source link

organizing to release JQ 1.6.2 and JQ 1.7: Your suggestions/comments would be appreciated #2550

Closed liquidaty closed 11 months ago

liquidaty commented 1 year ago

This is an offshoot of #2305. Please read that for background. cc: @stedolan @nicowilliams @wtlangford

I am posting this issue to organize a plan to create a new JQ fork with two primary objectives:

  1. Merge existing bug fixes into a new 1.6.2 release
  2. Merge existing new features into a new 1.7 release

The plan to create a new for is done out of necessity, not desire-- again, see #2305 for details. This is NOT an attempt to hijack or otherwise replace this repo, and the ultimate goal will be to either merge back into this repo, or to otherwise provide a successor with the blessing of this repo's owners/maintainers. If at any point in this process those maintainers would like to merge this effort back into this repo, that would be welcome

As a start, I would propose that we:

  1. agree on a decision-making framework
  2. identify a core set of roles, and folks who can fill said roles, to get this off the ground
  3. agree on a preliminary target timeline
  4. identify existing PRs for bug fixes and enhancements to target for 1.6.2 and 1.7, respectively

Request for comment

This entire issue post is a request for comment to see if we can get this off the ground. I will start with the below proposals. These are certainly incomplete and in need of further refinement / fleshing out, so please offer your suggestions / comments

Decision framework

I'm just going to throw out the simplest way to start: on any decisions that are not obvious, the core team votes (on the other hand, how do we decide the core team? kind of chicken & egg...). Maybe we start with the core team being, whoever are the first few volunteers who are willing commit meaningful time/effort/value to this?

As for decision guidelines, I would propose that a core value is impact-driven prioritization. Surely it is impossible to agree on exactly what outcomes have what impact, but we can also surely agree that a fatal bug fix is more important than a cosmetic one, and an enhancement with 100 likes and dozens of comments is probably more impactful than one with no likes and no comments.

Core set of necessary roles

Folks who can fill these roles?

Please volunteer. I'm happy to pitch in for any role, though surely others are more skilled than I at CI/CD or code review (and probably all the others too). @pkoppstein @wader @tst2005 @chickenandpork (apologies in advance for anyone, which there are sure to be, whom I should have mentioned and missed here-- pls do not take personally but rather blame my sleepiness): any takers?

Preliminary target timelines

Suggest May 1, 2023 for version 1.6.2. If there are more bug-fix PRs than can fit in that time, can always plan for 1.6.3 some time thereafter.

Suggest May 15, 2023 for version 1.7. Similarly, If there are more feature-enhancement PRs than can fit in that time, can always plan for 1.7.1 some time thereafter.

identify existing PRs for bug fixes and enhancements to target for 1.6.2 and 1.7, respectively

Note: I have not spent much time reviewing PRs, so I've just started by exporting the PRs to jq_prs-20230307.json.txt and spitting out in markdown below, and then mostly arbitrarily choosing a few to get started

number title is_bug version
#2548 allow, and strip, comments (#, // or /*) N 1.7
#2547 add utf8tolower, utf8toupper N 1.7
#2546 Fix segmentation fault when using jq in threads Y 1.6.2
#2538 fix: tarball fails distcheck: add files missing from manifest y 1.6.2
#2535 Remove deprecated --nul-output documentation Y 1.6.2
#2523 Added truncatestream builtin function N 1.7?
#2519 C99 compatibility enhancements for the configure script 1.7?
#2505 Build and test using GitHub actions workflow 1.6.2?
#2504 doc: Added information on --indent to --help
#2492 Show missing backslash in manual
#2487 Add !x as syntactic sugar for x | not
#2484 Tidy up nomem_handler implementations to fix SEGVs and ensure more consistency between TLS and non-TLS implementations.
#2482 Fix double-installed doc
#2480 Use right headers and flags in configure
#2474 Document how sort_by() can sort by multiple keys, including an example.
#2473 Clarify in the manual how map_values() behaves differently from map().
#2470 Add examples for input & inputs to manual
#2468 Use autotools standard ${docdir}
#2462 docs: clarify split behavior
#2451 Bump lxml from 4.6.3 to 4.9.1 in /docs
#2434 Fixed typo
#2432 [docs] Use yaml.safeload() in build*.py scripts
#2398 Clarify use of --argjson option
#2391 Add fixes for manual
#2357 Validate module metadata to be an object
#2355 Link to the Onigurama docs
#2336 Fix typo
#2314 Support binary strings, preserve UTF-8 and UTF-16 errors
#2311 Replace bareword error
#2306 Added usage under Powershell to the manual
#2296 Use a regex to parse ISO8601 dates
#2292 Disable some tests when building without regex support
#2285 Add in-place modification (expression) shorthand in objects
#2256 Fix --indent 0 implicitly enabling compact-output
#2255 Added a first fuzzer for integration with OSS-Fuzz.
#2254 Fix number lexer to avoid conflict with object indexing
#2253 Make 0 divided by 0 results in NaN consistently
#2252 Fold modulo operator on constant values and raise zero remainder error quickly
#2249 Enhance @sh to output objects as bash associative array initialisers
#2241 Added base/1 and unbase/1
#2235 fix handling of -0
#2233 Fix typo
#2216 Add an example usage for the inputs builtin
#2212 Manual error -- key should be "foo" without spaces
#2205 add nix and guix to website and fix typo in manual
#2202 Fix strflocaltime on daylight saving time values
#2188 Add Windows installation via scoop
#2183 Build out of tree
#2157 Fix uri format to follow RFC 3986
#2142 Fix string multiplication with a value between 0.0 and 1.0 Y 1.6.2
#2137 allow MS Windows \r\n style line endings and it's (broken) "UTF-8 BOM" Y 1.6.2
#2133 Fix deletion using assigning empty against arrays
#2112 Add debug(exp) builtin
#2108 Remove undefined behavior caught by LLVM 10 UBSAN. Y 1.6.2
#2102 typos in manual.yml
#2099 alloc: properly allocate memory handler per thread
#2098 Adds "tobool" builtin function for converting strings to booleans N 1.7?
#2094 manual: mention 0-based indexes in array slicing
#2093 Fix documentation for default search paths
#2092 spelling: text
#2081 use homebrew addons
#2079 Correct object construction with variables example
#2067 Fedora link returns 404
#2060 Added lpad and rpad
#2059 #618 alternative to dupn
#2010 Fix to incorrect --stream paths
#2004 Rewrite Dockerfile to use Alpine Linux, and produce a minimal image
#2000 Fix assertion when using multiple data imports
#1997 Use decnum for addition
#1993 Better dumping
#1989 Update manual.yml
#1977 Override field color with JQ_COLORS
#1962 [doc] replace find with select function as the example of jq-coded functions
#1961 implement scan/2 function
#1907 Unify/simplify the MultiByteToWideChar() code and add wrappers for open()/stat()
#1900 Optimise size of docker file
#1869 Make last(empty) empty
#1866 Should work on the entire object instead of each key of it.
#1862 Added jv_is_integer_large() to the library.
#1849 add nested array indexing example
#1843 dlopen(), random, I/O, eval, and co-expressions (co-routines)
#1798 Add fromstream_with_dups() builtin (fix #1795)
#1791 Set field color using JQ_COLORS
#1789 Provide a jvp_dump_raw_string function to implement the combination o…
#1767 Patches for AIX Y 1.6.2
#1748 Make reverse more DWIM-y
#1743 Save literal value of the parsed number to preserve it for the output
#1738 Add new builtin fetch
#1736 Fix truncate_stream example
#1733 fixed word 'outputting' in manual.yml v1.5
#1726 Rename gen_ functions to jqgen
#1703 wip: feat: terminate jq immediately after the outgoing pipe closed
#1701 Added is{scalar,value,..}. Fixes leaf_paths bug. Leaf paths Y 1.6.2
#1673 Add support for CRLF line endings in filters N 1.7
#1672 Add scoop installation instructions
#1671 "writing output failed" with non ASCII input
#1654 Make INDEX/2 more efficient
#1643 Add mul builtin
#1631 Fix Dockerfile
#1570 Fixes issue #1569 Y
#1558 Fix a try-catch example Y
#1549 Use an unreachable commit ID instead of a tag
#1537 Update manual.yml
#1527 Update the man page to note that the sort functions are stable
#1523 Update maruku and rake dependencies to latest versions
#1522 Add ruby config dotfiles for docs
#1517 Fix build with linux on travis-ci Y 1.6.2
#1460 Change tutorial to highlight filter part of query versus output part of query
#1458 fix portability problems, found in NetBSD-7.1/amd64 Y 1.6.2
#1454 Example of object creation attribute shortcut
#1440 Revisions to any/2, all/2, IN/1, IN/2 as per #1438
#1327 jv: Add some support for 64 bit ints in a very conservative way (ALTERNATIVE)
#1246 jv: Add some support for 64 bit ints in a very conservative way
#1228 Define format in jq code
#1215 Add support for seccomp
#1201 Add snap packaging support
#1183 More info. Directly install the release binary.
#1169 Fix some typos
#1159 C++ compatibility, C and C++ examples
#1127 Ignore jq.exe, fixed gcc compiler warnings for msys2 (windows).
#1103 Enhance the "add" operator so that, when applied to objects, if two f…
#1102 Allow tonumber to work on strings that contain commas as thousands separators
#1062 project/1, query/1 and unify/1 added to builtin.jq, with tests and documentation
#1060 Mention select() in if-then-else docs
#1058 add debian/ directory
#1032 Dump block
#673 Working module/package system
#458 Move tail call optimisations to compile.c.
wader commented 1 year ago

Great start. Will start going thru issues:

Add a PR comment. Documentation fix.

Has PR #2395, also related to #2216. Documentation fix.

Think the conclusion for master was --nul-output works, there is code for -0 but is treated as a number instead of flag (and --null-input has never been?). Mostly documentation fix, maybe -0 code should be cleaned up?

Documentation fix.

Documentation fix.

BTW also accidentally noticed that github markdown includes the issue/PR title if you put them in a list - #<number>!

Will continue later when i more time.

pkoppstein commented 1 year ago

@liquidaty - Thanks for getting the ball rolling so nicely!

Your post raises a number of issues, but in this response, I'd like to focus on just a few that arise because of various complexities.

First, on governance - I suggest that a "core" group be formed (defined by authority to make changes on github), and that at least initially it proceed on the basis of unanimity.

Second, because there are so many issues, many of which are actually quite tricky in one way or another, I think we need to prioritize carefully and be a bit wary about pre-mature commitments to time lines. I'd suggest focusing on generic "milestones" in the near term, perhaps along these lines:

Version 1.6.1 - the state of things at the time the fork is created.

Version 1.6.2 - minor documentation updates and fixes for bugs or performance issues provided the solutions are known to be quite straightforward.

Version 1.6.3 - incorporate changes of the "low hanging fruit" variety, avoiding new features but possibly addressing at most one important issue, e.g. an issue upon which the resolution of others depends (e.g. the 'builtins.jq' issue (*)).

Version 1.6.4 - additional changes that adhere to the principle of backward compatibility.

My guess is that there will be other releases before 1.7.

(*) I am not quite sure what the current status of the "builtins.jq" issue is exactly, but the problem has to do with the run-time costs of adding additional defs to builtins.jq. The general point of course is that we have to be careful not to introduce unexpected performance penalties.

liquidaty commented 1 year ago

@wader thanks for help reviewing. @pkoppstein I like all those ideas.

Does the JQ repo currently have any built-in benchmarking tests? I see some mentions in #1589 and #2386 but can't tell if anything made it into the repo. If not, maybe maintaining some basic benchmarks should be one of the "roles"-- though maybe it's more of a one-time setup

pkoppstein commented 1 year ago

@liquidaty asked about

built-in benchmarking tests?

Obviously the various tests in the tests/ directory can be used for benchmarking. See also:

mfeit-internet2 commented 1 year ago

1862 and #1752 are of interest to me because they cause problems in a production system and I've been running a patched version for years.

1862 should be pulled into a 1.6.x and #1752 should go into 1.7.

I'll be happy to review, beta test, etc. anything related to either.

hexagonrecursion commented 1 year ago
  1. Where is the fork? Does it already exist? I have looked at several items in https://github.com/stedolan/jq/forks?include=active&page=1&period=2y&sort_by=last_updated but nothing looks like it is actively maintained.
  2. I agree with @pkoppstein: pre-mature commitments to time lines might do more harm than good. I also agree with their milestone criterias
liquidaty commented 1 year ago

The fork has not been created. From my perspective, the main thing that needs to happen first, is to establish the "core" group. This would be the group that has authority to make changes on github, and would initially make decisions based on unanimity.

I can appreciate that pre-mature commitments can be easily overdone, but there needs to be some level of commitment to make this work as a group, otherwise it might as well be a one-person project which defeats the entire purpose. Said another way, as part of determining that core team, we would need to have an understanding of some minimal level of commitment (e.g. obviously, unanimous decision-making can't happen if decision-makers are not available) and some very basic initial milestones or general objectives as to what we are trying to do in what time frame.

I would propose the following issues to be first addressed with the core team. And this is going to be a bit of a chicken and egg process given that we don't have a core team yet, but we have to start somewhere.

  1. what issues / pull request comprise the total universe of potential issues to include for each of the various contemplated upcoming versions
  2. how a change request should be initiated for review / approval; what are the necessary prerequisites
  3. what the QC process will be
  4. how often, at a minimum, each core member can make themselves available to review anything that awaits their feedback on a decision. we don't want anyone to feel like this requires an unreasonable amount of work on their part, and we also don't want to be stuck for 6 months waiting for a core member to opine on whether a pull request should be accepted
  5. what differences we have amongst the team that we should consider so that we are taking advantage of our individual strengths to optimize effectiveness while minimizing burden

This is just a very rough initial proposal so please comment as you see fit. More to the point, assuming we can come to some reasonable agreement about how all this will come together:

More directly to the point: who is up for being part of (or at least considering being part of) the core team? and/or just being part of the team, but non-core? @pkoppstein, @wader, @chickenandpork, @tst2005, @hexagonrecursion, @mfeit-internet2 (or anyone else I missed)?

nicowilliams commented 1 year ago

The biggest problem I have maintaining jq, besides my own lack of time, is the inability to add maintainers w/o @stedolan's help.

Therefore, @owenthereal and I set up https://github.com/jqlang/jqlang a while back in preparation for forking some day. I've invited @stedolan to be the owner of that org. I will invite him again.

What I need right now in order to make that fork a reality is: candidates to be maintainers. To pick from the candidates I guess I'd want a small set of PRs or something to look at.

Mind you, you guys are pretty fed up -as you should be- with me and @stedolan, so if you want to fork using a different org and pick your own maintainers, that's fine. jqlang seems like a fine org name though.

Thoughts?

mrienstra commented 1 year ago

Stoked to hear from you @nicowilliams, your input is very welcome!

My voice shouldn't count for much -- I'm just a rando who got excited about the idea of helping this project get unstuck -- but I'm in favor of https://github.com/jqlang/jq as the new fork. Though it's worth mentioning the existence of https://github.com/jqmaintainers/jq, created by @tst2005.

Again, no particular reason to listen to me, but personally I think @wader & @itchyny should be fast-tracked in as a maintainers. I haven't looked closely at other people's contributions recently -- and again I'm not familiar enough with jq to be any kind of gatekeeper -- but just for convenience, here are some links, in alphabetical order (sorry if I missed anyone who has been active lately, I kinda threw this together):
adoxa PRs branches commits comments
agordon PRs branches commits comments
carlsmedstad PRs branches commits comments
chickenandpork PRs branches commits comments
dgryski PRs branches commits comments
DRMacIver PRs branches commits comments
dtolnay PRs branches commits comments
emanuele6 PRs branches commits comments
hexagonrecursion PRs branches commits comments
itchyny PRs branches commits comments
jankatins PRs branches commits comments
leonid-s-usov PRs branches commits comments
liquidaty PRs branches commits comments
mfeit-internet2 PRs branches commits comments
muhmuhten PRs branches commits comments
orzen PRs branches commits comments
pkoppstein PRs branches commits comments
stagrlee PRs branches commits comments
stroan PRs branches commits comments
tst2005 PRs branches commits comments
wader PRs branches commits comments
wtlangford PRs branches commits comments

To recreate, starting w/ a list of @ mentions, use this regex: ^@(.+)$ ➝ | [$1](https://github.com/$1) | [PRs](https://github.com/stedolan/jq/pulls?q=is%3Apr+author%3A$1) | [branches](https://github.com/$1/jq/branches) | [commits](https://github.com/stedolan/jq/commits?author=$1) | [comments](https://github.com/stedolan/jq/issues?q=commenter%3A$1) |

nicowilliams commented 1 year ago

Thank you for that list @mrienstra! I'll review it tomorrow and invite some of them.

nicowilliams commented 1 year ago

Also, is there a PR for making the Windows build work again? And one for GitHub Actions?

pkoppstein commented 1 year ago

@nicowilliams wrote:

is there a PR for making the Windows build work again? And one for GitHub Actions?

This one seems relevant:

https://github.com/stedolan/jq/issues/2250 AppVeyor continuous integration tests all fail with GPG errors

wader commented 1 year ago

@nicowilliams Hey, nice to hear from you, hope all i well. My vote is for @itchyny also if he is up for it, and based on history maybe it's good to have more then one new maintainer :) i'm up for helping out either as maintainer or contributor. I'm not very familiar with the jq source code but i have done lots of C previously and feel motivated working on it.

mrienstra commented 1 year ago

Quick aside, I appreciate the process @liquidaty is championing in this thread, and I don't want to derail that train of thought. 🚂❤️

owenthereal commented 1 year ago

Therefore, @owenthereal and I set up https://github.com/jqlang/jqlang a while back in preparation for forking some day.

👋 Hello everyone!

I'm absolutely thrilled to see the move to https://github.com/jqlang finally happening 🚀. I can't wait to collaborate with all of you in maintaining and further developing jq!

A little about me: I'm the brains behind https://github.com/owenthereal/jqplay, the interactive playground that's linked directly from the jq documentation page 📚. Excited to bring my experience and passion for jq into this new venture with you all! 💻🔧🌟

liquidaty commented 1 year ago

I don't have any opinion on where the future fork is, and I'm happy to be a core member, or just a helping member, or not a member at all (in fact, this last would be my preference if the objective can be met without me!)-- really, I just want, for my selfish reasons, for the good work that many people have done on jq, as well as at least one of the updates that I've submitted and that is necessary for my purposes, to be available to everyone in a current and centralized single repo/fork so we don't have to all be using our separate forks.

So if the desired location of the fork is jqlang, that's fine with me.

That said, my prior question, which IMHO addresses the number 1 obstacle at this time ("who is up for being part of (or at least considering being part of) the core team? and/or just being part of the team, but non-core?") is still in want of a direct answer from anyone. Though, perhaps the latest comments and enthusiasm are an implicit volunteering to at least be a non-core contributor?

In any case, whatever team wants to maintain this, wherever the fork is housed, will still need to deal with figuring out what are the goals, what is the process and what is the team. On that last question, I would suggest shifting the focus from "who is sufficiently skilled to be a maintainer" to "what do we need to do" followed by "who can help and who is willing to help do those things".

So to try to put some specificity around "what we need to do" next, here is something to start with:

  1. Agree on a set of releases. Let's start with those that @pkoppstein proposed on 3/7
  2. For each release, agree broadly on a universe of issues/PRs applicable to each
  3. Agree on a level of commitment. I would suggest we expect each person to spend an hour or two about once every 2 months. That might sound like a small number, but I'm trying to start with something sustainable assuming we are all busy professionals with personal lives. I would further suggest that we ask for the review team members to commit to this role in 1-year increments. This will make it easier for people to volunteer to help, because the commitment expectation is fixed and reasonably low. At the end of that time, each review member can decide to stay, or should make at least an effort to find a replacement
  4. Agree on a review process. I would suggest:
    • Each PR must have an associated automated test. Someone on the core team will need to review the PR to make sure this condition is met
    • Each PR must pass all automated tests. We should have a git hook for this.
    • Agree on what platforms will be supported; all PRs must pass all automated tests on all platforms. Suggest Linux, Win, MacOS, Emscripten. Intel only (can add ARM/M1/M2 when github runners are available for them)
    • Exceptions can be allowed by "group decision" (whatever that is defined to be: unanimous, etc)
  5. Agree on a review cadence. I would suggest starting with something that is easy for everyone to commit to, such as 1 PR every half month. If that is too slow, it's easy to speed it up later. If we start too fast, we could easily end up back to where we are today.
    • That means that as a group we are reviewing something every 2 weeks
    • Assuming we need at least 3 people to OK any PR, we need at least 12 people on the review team
  6. Agree on what "group decision" means, and what happens if it breaks down
    • does everyone need to review all the code changes and all the QC tests? Or do we specialize? Or rotate? Suggest unanimous but with specialization for actual review. i.e. one person reviews QC tests. One person reviews the actual code. Exceptions for large PRs that have a lot to review. Rotate duties. Votes can be "for", "against", or "abstain". If anyone votes "against", that is binding unless 4 others vote "for" and no others vote "against".
    • unanimous, majority, depends?
    • what is the basic timeline on a per-PR basis? e.g. at start of the month, 2 PRs are chosen, one voted on by 15th, the other by end of month?
    • what if someone whose vote a decision requires is unresponsive for 2 weeks? 2 months? 1 year? what if someone's contributions don't seem to be helping? For whatever reason, I would suggest that someone can be forcibly removed from the team if 5 other members agree that doing so is in the best interests of the jq community
  7. Get the fork ready. This includes making sure the automated QC tests are working for all target platforms. I am not sure this is already properly working today; if not, we will need some help fixing this as a first order of business for the new fork

So: how do those next steps sound, and if within reason (assuming some level of flexibility to refine), who would be willing to help, and which specific items above would you be willing to help with?

Again, I'm not trying to insert myself if I'm not needed. I'm just trying to be a catalyst to us being able to have a current and single jq repo, and offering to help if I can, but more than happy to leave this in more capable hands than mine if that goal can be met without my participation.

liquidaty commented 1 year ago

Hi everyone, thanks for your comments and feedback.

Thus far it doesn't look like we have sufficient (imho) momentum for a "core" team, so I will plan to create a new fork this weekend for the benefit of anyone who would like to use it. If the preferred location for that fork is jqlang and the owners want to make me an admin there, I'm happy to discuss that possibility, otherwise I'll choose a different location and notify this thread.

The first order of business will be to get CI/CD running smoothly, for win, linux and MacOS, after which time I will start looking at releases PRs / etc. If anyone would like to join me or has other comments before then, feel free to post here and I will check back in around the end of the week.

nicowilliams commented 1 year ago

@mrienstra: thanks for that list! I've invited @muhmuhten (accepted), @itchyny, @leonid-s-usov, @dtolnay (though I suspect he's quite busy with Rust), @wtlangford (though he is not interested in continuing as maintainer), @stedolan (as owner, of course), and I'll look more later.

nicowilliams commented 1 year ago

@liquidaty I don't think we need much governance unless there's disagreement.

My preference is that maintainers take any obvious PRs and that they seek consensus among themselves in case of controversy.

nicowilliams commented 1 year ago

Also, the way we did things when we had multiple active maintainers is that the maintainers reviewed each others' PRs and we needed each others' approval to push those, but as for PRs by non-maintainers we took the trivial ones w/o debate and discussed the ones that needed to be discussed.

nicowilliams commented 1 year ago

A bit of wisdom for new maintainers:

chickenandpork commented 1 year ago

I’m still interested in helping; I’m an SRE who enjoys CI/CD challenges.

I have seen statements of interest, but it seems like there’s a plan now :)

pkoppstein commented 1 year ago

@chickenandpork wrote:

I’m still interested in helping

Great. In case you didn't notice, @nicowilliams asked in this thread:

is there a PR for making the Windows build work again?

If you could offer a PR addressing the issue(*), it might even be incorporated into the stedolan version!

Thanks.

(*) Presumably https://github.com/stedolan/jq/issues/2250 (AppVeyor continuous integration tests all fail with GPG errors)

liquidaty commented 1 year ago

Seems like as a start it might be easier to remove the dependency on appveyor, since it adds another layer of applications, logins, authentications, accounts, etc. Is there anything that appveyor provides now, that isn't as easily provided by github ci/cd?

itchyny commented 1 year ago

I would like to join the communication space of maintainers if exists (Slack or Discord?).

pkoppstein commented 1 year ago

@liquidaty wrote:

Is there anything that appveyor provides now [?]

See https://github.com/stedolan/jq/wiki/Installation#windows-using-appveyor

liquidaty commented 1 year ago

@pkoppstein thank you. So is Appveyor used solely for building windows executables? If so, that is extremely easy to replace with a github action that runs mingw64 on vanilla linux. I'd be happy to submit a PR for that, which would kill 2 birds in one stone (fix windows build and remove a "heavy" and otherwise unnecessary dependency).

here's an example of a github action that builds a win version of an executable (and builds and installs a custom jq library!) at https://github.com/liquidaty/zsv via linux+mingw64 as specified in https://github.com/liquidaty/zsv/blob/main/.github/workflows/ci.yml.

pkoppstein commented 1 year ago

@liquidaty - From my point of view, the main significance of Appveyor was that it made it easy for Windows users to snarf any recent "snapshot" of jq of interest, not just the official versions.

wader commented 1 year ago

I would like to join the communication space of maintainers if exists (Slack or Discord?).

Me too

liquidaty commented 1 year ago

OK I've spent a lot more time than I had planned on looking into this appveyor issue and here are my conclusions:

  1. the existing appveyor.xml is out of date. It tries to download non-existent files and apply out of date workarounds etc
  2. I tried many fixes from various threads and thus far don't see any reasonable way to fix the broken issue, and I might not be alone in that view
  3. The whole appveyor issue, and any solution, seem entirely arbitrary and fragile anyway, and offer a very narrow benefit (in theory-- not today since it is broken) for which better and more maintainable alternatives exist
  4. This is a fundamental issue that needs to be fixed for anything to progress in a smooth manner. CI/CD with 99% pass and 1% fail is still of very little value
  5. A better alternative, which even the MSYS developers themselves suggest to @nicowilliams ' question is: "use GitHub Actions instead". It also isn't hard for GitHub actions to be set up to provide access to nightly build artifacts

I also think, it is taking a disporportionate amount of time and attention write all these messages and discussions and attempt to get consensus, vs just fixing the issues. So, I have set up my own repo and am going to remove appveyor and do whatever other changes are needed for github actions to a minimal level of CI/CD. Until those basic changes are done, I am going to keep the repo private, because it's going to be in a broken state and I don't want anyone to try to use it and get turned off by that. Once CI/CD is working in that private repo, I will make it public and then begin porting the obvious and non-controversial PRs into that repo. If and when any of the maintainers of this repo or jqlang would like to take those changes, they are welcome to do so (or make me a maintainer and I'll do it directly).

If anyone would like to help with that, feel free to let me know.

I would like to join the communication space of maintainers if exists (Slack or Discord?).

Me too

nicowilliams commented 1 year ago

If you set up a slack or discord, I'll join it. EDIT: Owen will set it up and schedule it.

The other thing I need to do is set up a mirror from jqlang/jq to stedolan/jq and update the README to note that we're moving issues/PRs/dev activity to jqlang/jq but that stedolan/jq will remain canonical as long as there is one maintainer left with push rights for stedolan/jq.

owenthereal commented 1 year ago

👋 Feel free to join this discord 🎉 : https://discord.gg/yg6yjNmgAC. Please name your server nickname the same as your GitHub username so that we know who we are talking to 😄

pkoppstein commented 1 year ago

@nicowilliams - Thanks for your continuing efforts. I would like to ask a big favor that I'm hoping would not take too much of your time - to tag the current version with an "official" release number (e.g. 1.6.1). It would make a huge difference in several respects. I won't go into the details as I'm sure you can easily surmise them. (Of course, if you have time to sneak in any additional enhancements before creating 1.6.1, so much the better :-)

If you are don't have time to create a new version yourself, is there any chance that someone else could do so?

wader commented 1 year ago

A bit of wisdom for new maintainers:

Thanks for sharing. I got a bit curious about your wisdoms as i've tried (and failed) various ways to extend jq while working on https://github.com/wader/fq

  • don't add any more non-function-like language elements like try/catch -- try/catch should have been a function try(exp; catch) and try(exp; catch; finally)

After writing lot of jq code i think i'm quite satisfied what it currently has. But if i could dream there are some things i've thought about that might be neat/useful:

But as i said, i'm very pleased with current jq.

  • ditto break
  • don't add any more polymorphism -- polymorphism in a dynamically-typed language creates problems

By polymorphism you mean additional types or how different type interact with each other? both?

  • do not add new types unless those are essentially sub-types of JSON types

    • for example, a "binary data" type should be indistinguishable from an array of small integers 0-255

    (this was something I discussed with @stedolan and he was quite adamant about this, and now so am I).

For fq i've added a binary type that can have both byte or bit units. First I naively exposed the type the to "standard jq" but it turned out be a bad idea, lots of weird behaviors happened (ex type/0 returned "binary"). This was later changed so the the type was indistinguishable to a string unless you use some new functions the know about the duality.

If I get some motivation and time it would be interesting to try the binary as array of ints idea for fq. It already kind of works like that if you use explode/0 or tobytes/0 (turns things into binary).

So I agree, adding a new type that is not JSON is probably a bad idea.

nicowilliams commented 1 year ago

@wader

By polymorphism you mean additional types or how different type interact with each other? both?

I mean adding behaviors to operators that work on multiple input types.

For fq i've added a binary type that can have both byte or bit units.

JSON only has the types that jq has, but we can have a binary type that is represented in jv/jq as an array of small unsigned integers (0..255) and internally use a byte array to represent it. Then we can have tobinary and toutf8 functions to convert from strings to binary and vice-versa, and we can have tobinary also accept arrays of numbers, and we can have a toarray to convert binary values that wouldn't allow setting/appending values that are not small integers (which would otherwise raise an error if the array was a binary). As well we'd have hex and base32 and base64 codecs. Maintaining the fiction that binary is an array of small integers is important when it comes to doing JSON I/O.

[...] lots of weird behaviors happened (ex type/0 returned "binary").

Exactly!

Binary has to be a "sub-type" of array (or string, but I think really of array). If a UTF-8 string is converted to binary, then that could be treated as a sub-type of string as long as you don't set or append elements that render it non-UTF-8, but checking that every time would be annoying, so I think it's best to go with binary as a sub-type of array.

Farmbuyer commented 1 year ago

@liquidaty If you're still looking for bugfix PRs, #1588 and #2116 are the same bug with trivially easy test cases (an error reporting routine is asserting and crashing).

chickenandpork commented 1 year ago

@chickenandpork wrote:

I’m still interested in helping

Great. In case you didn't notice, @nicowilliams asked in this thread:

is there a PR for making the Windows build work again?

If you could offer a PR addressing the issue(*), it might even be incorporated into the stedolan version!

Thanks.

(*) Presumably #2250 (AppVeyor continuous integration tests all fail with GPG errors)

@pkoppstein I hadn't noticed the response back in May; I haven't owned a windows box for likely 19 years, so I'm the opposite of helpful with windows. Now, bazel, that's got my interest, and it might re-enable some Windows compatibility. I cut a quick message into #general on the discord.

itchyny commented 11 months ago

Since jq 1.7 has been released, I'm closing this issue.