List of publicly released modules?

[joelpurra]

Is there a list of modules yet? Until there's a package system, I'd love to see people drop a line here for whatever they have released.

At some point I will clean up the shell scripts in har-heedless and har-dulcify, which downloads/creates, transforms and analyzes data based on HAR files.

[nicowilliams]

There aren't any yet. We probably need a package manager first. A few weeks from now we might have one.

[pkoppstein]

@joelpurra askd:

Is there a list of modules yet?

jq's concepts of "module" and "package" are still evolving, but there are plenty of jq snippets. A few are at the jq Cookbook. Some useful snippets are scattered throughout these trouble tickets. Perhaps some of them should be salvaged by adding them to the Cookbook or somewhere else, e.g. as a "gist". To get the ball rolling, I created one such snippet: https://gist.github.com/37c1311075d2993982be.git

It would be easy enough to add a Resources or Snippets page to the wiki, or should we leave them for google to find?

[joelpurra]

Fixed links (for browser usage) jq cookbook https://github.com/stedolan/jq/wiki/jq%20Cookbook combine.jq https://gist.github.com/pkoppstein/37c1311075d2993982be

Thanks!

[joelpurra]

Started writing my own library/package/module/modules: jq-hopkok. I'll try and dump more stuff there later on. Since I'm currently writing my code straight into .sh files, this is how the project will start out. https://github.com/joelpurra/jq-hopkok

[michaelmior]

Unfortunately I haven't been able to find any examples of a module which I can actually get working. combine.jq is posted above is the only example I could find using the new module system. I can't seem to get it working though. I've dumped it in ~/.jq and then tried various invocations such as

jq -L ~/.jq 'combine::demo'

Any suggestions on getting this to work? In any case, I think the docs need to be clearer on how modules work.

[wtlangford]

The docs most definitely need improvement regarding the module system, but since the system's not been part of a major release (1.5 isn't yet out), we've put the documentation off in favor of other things we felt were more pressing.

At any rate, what you're missing is the import statement.

jq 'import combine; demo'

If you put your library in ~/.jq/, you don't need to add the -L, since ~/.jq/ is in the default search chain.

For some reason, the default behavior for import is to bring the module's contents into the current namespace. @nicowilliams is this intentional? I thought we'd decided not to do that.

I'm adding a docs tag to this and flagging it for 1.5 to make sure we get the docs updated.

[michaelmior]

~/.jq doesn't seem to be included in the search path by default for me. (I built this version of jq off the master branch earlier today.)

$ jq 'import combine; demo'         
jq: error: module not found: combine

jq: 1 compile error

When adding the path, the example works fine. However, I expected that using combine::demo would alleviate the need for the import altogether. Any reason this can't be the case?

[wtlangford]

You are treating ~/.jq as a folder, right? You have ~/.jq/combine.jq? The reason behind the required import statement is somewhat complex, but a sufficiently simple version is that we wanted it to be explicit where non-builtin filters came from. Also, the import statement sports a search path argument.

That being said, I'm still confused as to why combine:: isn't needed. It feels slightly like a bug. On Oct 8, 2014 3:53 PM, "Michael Mior" notifications@github.com wrote:

~/.jq doesn't seem to be included in the search path by default for me. (I built this version of jq off the master branch earlier today.)

$ jq 'import combine; demo' jq: error: module not found: combine

jq: 1 compile error

When adding the path, the example works fine. However, I expected that using combine::demo would alleviate the need for the import altogether. Any reason this can't be the case?

— Reply to this email directly or view it on GitHub https://github.com/stedolan/jq/issues/535#issuecomment-58416719.

[pkoppstein]

@wtlangford wrote:

I'm still confused as to why combine:: isn't needed.

@nicowilliams can elaborate but essentially it's because combine.jq does not declare itself to be a module.

More specifically, Nico did not want to introduce a special syntax for the "import" statement to mean "import into the current namespace".

I think that's a perfectly fine approach -- so long as you think of "import FOO" in the sense of "require FOO" rather than "import this file and force it to be encapsulated into a module named FOO".

[wtlangford]

I just went to play with this to better understand it. The interesting thing I've noticed is that even if the imported file declares itself as a module, it does not get encapsulated when we go to use it. @nicowilliams Is this a bug or is this intentional?

[michaelmior]

@wtlangford Yes, ~/.jq is a folder containing combine.jq. As far as being explicit about non-builtin filters, I don't see how requiring the combine:: prefix is any less explicit. None of the builtin filters are namespaced in this way.

[wtlangford]

@michaelmior it is worth noting that I got quite busy towards the end of the development for the module system and wasn't part of the final stages, so what I said may have been off the mark entirely. I had intended for something like

import combine {"search": "/path/to/some/combine"}; # this path would often, be relative
combine::demo

But things seem to have changed after I was away. Looks like if you don't do import combine as SomeIdent {search: "/the/path"};, then the contents of combine are hoisted up to the current namespace. Which is a fine behavior, just not the one I was expecting. @pkoppstein implied that that is only the intended behavior when imported code doesn't declares itself as a module (no module combine {metadataObject}; inside the combine.jq file). That being said, the hoisting appears to always occur.

[michaelmior]

@wtlangford Just speaking from a user perspective without any knowledge of jq internals or how the module system was developed, what would be most convenient for me would be able to jq combine::demo at the command line. I would expect jq to look in the search path for a module named combine and find the filter named demo. For me, requiring an explicit import every time the command is invoked removes a lot of the benefit of using modules (less typing).

[pkoppstein]

@wtlangford wrote:

That being said, the hoisting appears to always occur.

It seems to me that that is a bug: the "hoisting" should only occur if there is no declaration (module statement) in the file.

That is:

1) a vanilla "import" statement (i.e. one without an "as" clause) should simply cause a file to be loaded; 2) whether or not a module is created by a vanilla "import" statement should depend on the contents of the loaded file.

[pkoppstein]

@wtlangford wrote:

If you put your library in ~/.jq/, you don't need to add the -L, since ~/.jq/ is in the default search chain.

I believe that is the plan, but at present, unfortunately, that is not the case (using "master"), presumably because ~/.jq has long been assumed to be a file. In fact, currently "master" does not have a default at all!

[wtlangford]

That's interesting. I wonder why we haven't yet done that. Well, I suppose it is time I stop making statements from memory without checking them wrt the module system. Interestingly, we still attempt to load ~/.jq as a file in master. That's an interesting action, but we probably left that in to ease the shift from 1.4 to 1.5. At least, that's what I'm going with :)

@michaelmior I'm not seeing a way to do that without being particularly hacky. We definitely intended for you to have to say import someModule before referencing it.

[pkoppstein]

@michaelmior - Please note that you don't have to use the "import" statement at all to load a file like combine.jq. Assuming a sufficiently modern shell, you can write something like this:

jq -n -M -f <(cat combine.jq; echo demo)

That won't save any typing, but using this approach, you can use regular old jq 1.4, and you can also use curl:

$ /usr/local/bin/jq -n -M -f <(curl -Ss https://gist.githubusercontent.com/pkoppstein/37c1311075d2993982be/raw/combine.jq; echo demo)
{
  "id": 123,
  "son": [
    "Son1",
    "Son2"
  ],
  "surname": "S"
}

[michaelmior]

@pkoppstein My main hope for using the module system was to save typing. If an explicit import is required, I would probably just fall back to writing shell scripts which is a little unfortunate. I think implicit import of modules would be necessary for my use case (reducing keystrokes).

I'm sure this has already been well-considered, but I'm curious what the complications are for doing automatic imports. My expectation is that this would only happen when a module is explicitly referenced (identified by ::). Then I would expect the function name to be split on :: and looked up in the search path by folders with the second-last component being the file name and the last component being the function.

[pkoppstein]

@michaelmior - I'll let others speak to any implementation issues there may with respect to your proposal, but I would like to point out that what you're asking for (triggering of an "import") might not achieve what you expect! Here's why:

Since combine.jq is not a module, importing it into the main namespace will result in "demo" being defined, but not "combine::demo". Thus, according to your basic proposal, the incantation "combine::demo" would result in "demo" being defined, but not "combine::demo", so the incantation would fail.

Of course I realize the entire system could be designed differently, but design is, alas, more difficult than tinkering.

As for reducing keystrokes -- the key to big savings is writing scripts. In this regard, did you know that you can use jq as the shebang processor? See https://github.com/stedolan/jq/wiki/FAQ

The other key to keystroke convenience is being able to open a shell window within your editor, but that's a topic beyond the scope of this webspace.

[michaelmior]

@pkoppstein You're right that I forgot combine.jq is not a module. However, I would also assume that the automated import would only work if the file exists and it's a module.

[nicowilliams]

Wow, I've a lot to catch up with...

[nicowilliams]

On Wed, Oct 08, 2014 at 11:27:53AM -0700, William Langford wrote:

For some reason, the default behavior for import is to bring the module's contents into the current namespace. @nicowilliams is this intentional? I thought we'd decided not to do that.

It was either that or provide some syntax for it. I'm fine with either way.

[nicowilliams]

On Wed, Aug 06, 2014 at 03:25:09PM -0700, pkoppstein wrote:

@joelpurra askd:

Is there a list of modules yet?

jq's concepts of "module" and "package" are still evolving, [...]

For 1.5 I've only one more thing to do w.r.t. modules:

• prepend the importer's directory to the default searchpath

This will allow us to leave all versioning to an external pkg manager.

[wtlangford]

@nicowilliams Can you confirm that the intended behavior for a basic import (no as clause) is to lift content to the current namespace if the imported file is not a module and to create a module if the imported file is a module?

Or, alternatively:

a.jq

def test: 3;

b.jq

module b;
def test2: 2;

$ jq 'import a; test'
3
$ jq 'import b; b::test2'
2

The current behavior is that things always get brought into the current namespace, regardless of the file that gets imported. If that's the case, why do we have the module statement?

[pkoppstein]

@nicowilliams - Just to be clear - I think we're all agreed that if the imported file declares itself to be a module, then that should be respected in the absence of an "as" specification.

Also -- could you please let us know what the plans for ~/.jq are? I believe the plan was that for 1.5, a file named ~/.jq would not be read in (by default), and that a directory named ~/.jq would have a special status. I'm hoping that ~/.jq will (by default) be included as if by -L ~/.jq.

Thanks.

[joelpurra]

@nicowilliams - Just to be clear - I think we're all agreed that if the imported file declares itself to be a module, then that should be respected in the absence of an "as" specification.

I'll repeat my opinion: no module keyword is necessary - it's merely cluttering code files.

See #566 Semantic version utility where I've stated similar things. There's also #491 Enhancement request: integrated package manager.

[pkoppstein]

@joelpurra wrote:

no module keyword is necessary - it's merely cluttering code files.

In the absence of a different "import" mechanism, the module keyword is necessary to distinguish "file inclusion" from "module importation".

More importantly, the module keyword is necessary for the module system to be declarative. Of course I realize there are different conceptions of modules and modularity, and perhaps Joel would rather that jq's namespace system should not have any of the trappings of a module system, but I note that F# for example has both "namespace" and "module" keywords for declaring namespaces and modules.

[nicowilliams]

@pkoppstein If ~/.jq is a file, it gets imported, otherwise if it's a directory then it gets to be part of the default search path.

[nicowilliams]

@nicowilliams https://github.com/nicowilliams Can you confirm that the intended behavior for a basic import (no as clause) is to lift content to the current namespace if the imported file is not a module and to create a module if the imported file is a module?

I'm open to suggestions. I see three options:

• no namespace pollution
• use extra syntax to request namespace pollution
• if no "as" clause is the syntax for requesting namespace pollution

Each has something to recommend for and against it. Several folks have stated conflicting preferences. I am not partial to the first even though I think one should mostly not import into one's modules' namespaces. I am neutral as to the last two.

The current behavior is that things always get brought into the current namespace, regardless of the file that gets imported. If that's the case, why do we have the module statement?

To provide other metadata. At any rate, the name of a default namespace to import into should be derived from the name imported, not from the imported module's declaration. The principle here being: important effects should be lexically visible where it's most convenient to the reader.

[nicowilliams]

On Monday, October 13, 2014, Joel Purra notifications@github.com wrote:

@nicowilliams https://github.com/nicowilliams - Just to be clear - I think we're all agreed that if the imported file declares itself to be a module, then that should be respected in the absence of an "as" specification.

I'll repeat my opinion: no module keyword is necessary - it's merely cluttering code files.

For the time being it's only use should be for metadata that jq doesn't use today, and might never, but which might be convenient to keep in-band (as opposed to out-of-band in a separate file) (obviously this is not suitable for commit hashes, content hashes, version numbers, but it might be OK for author, license, history, and other metadata such as hompage URIs, etcetera...). It may well be utterly useless.

[joelpurra]

Just released three small packages. Wanted to show the kind of granularity I'd like to see in packages - the smaller and focused the package, the easier it is to re-use it. https://github.com/joelpurra/jq-fallbacks https://github.com/joelpurra/jq-object-sorting https://github.com/joelpurra/jq-key-counter

Try them with jqnpm and let me know if there are any questions! https://github.com/joelpurra/jqnpm

Any jqnpm compatible packages are also added to the jqnpm wiki -- add yours!

[joelpurra]

Released four new packages:

• https://github.com/joelpurra/jq-bugfix-jq-552 (detection/workaround for #552)
• https://github.com/joelpurra/jq-zeros
• https://github.com/joelpurra/jq-stress
• https://github.com/joelpurra/jq-dry

Because copy-pasting is bad, I also made a project template/generator a part of jqnpm:

jqnpm generate <github username> <package name> "<one sentence to describe the package>"    Generate a jq/jqnpm package skeleton in a subfolder.    Package name: all lowercase, separate words with a dash '-'.    Package name example: cool-tool

Code goes in jq/main.jq, tests in tests/all.sh,

[joelpurra]

Three more packages today!

• https://github.com/joelpurra/jq-disarray
• https://github.com/joelpurra/jq-counter-buckets
• https://github.com/joelpurra/jq-ratio-buckets

Using jqnpm generate and taking parts of jq scripts from one of my master's thesis projects. Doesn't take long to write tests for a few small functions at a time either, and I know my messy thesis code will be awesome once I'm done. Hope you guys find these lego pieces useful too =)

[nichtich]

And another one:

• https://github.com/nichtich/jq-jsonpointer#readme

I also found

• https://github.com/stanch/jq-modules

Nevertheless we'd better manage the list in the wiki or on a dedicated git repository as list of repository URLs to automatically fetch metadata from.

[nichtich]

I also started a jq module to process Wikidata dumps. Listing new modules in this thread does not scale so let's collect links to existing modules at https://github.com/stedolan/jq/wiki/Modules instead!

[nicowilliams]

Thanks @nichtich!

[nicowilliams]

Longer term we could have a repository... I'm not sure I want to get to deal with the problems that come with that... -- think NPM...

[dubiouscript]

:dark_sunglasses: + just found this : https://github.com/fiatjaf/awesome-jq