Structured JEPs

[innovate-invent]

I recommend we establish a structured format for JEPs that allows the markdown to be baked into html documentation to be included in the site. Only some elements would be included from the JEP and additionally some elements would only exist for the purpose of inclusion in the documentation and would not directly contribute to the language spec.

This will require a actions workflow to automatically bake the JEPs and commit it to the jmespath.site repo.

All existing specification should also be loaded into a JEP-0 to include in this pipeline

[springcomp]

That is an awesome idea. I must admit reStructuredText is completely foreign to me. But yes, I get what you mean. For now I have been preparing what I hope to contribute to the upcoming specification with the let() function. So far, I have identified the following scenarios:

Specification text update

As in the Third-Party Functions JEP, just some boilerplate text that updates some paragraphs.

This may probably be updated manually as it appears to be too specific.

New functions

This seems to be the easiest to structure for automatically including a subset of the JEP to the specification. I think there needs to be:

• Function quick summary paragraph.
• Function signature.
• Function details explaination.
• Function examples.

Each function now lives in its own file so the change is narrowly scoped.

Grammar updates

This would need a new chapter to the spec. Maybe we need to split each concept / chapter in its own file as for the functions. Maybe if we tackle something like the Regular Expressions discussion.

As for the grammar itself, I do not see yet how to achieve this. But I think grammar updates are less frequent than new functions so if we automate the most frequent scenario, that's great.

My personal comfort zone is to use PowerShell scripts in GitHub actions.

[springcomp]

I have no idea how to convert Markdown back to Sphinx code for inclusion to the spec, though 🤷‍♂️.

[innovate-invent]

I definitely have to dig into sphinx a bit more. I was thinking of possibly moving away from sphinx but its various document output format support might be important.

My major motivation for this is to remove the need to maintain two copies of the spec. The JEPs should be a single source of truth and automatons should reformat that for consumption. I think this is why the JEPs were originally in the site repo.

[springcomp]

Hello @innovate-invent Nolan,

I think we are ready to start generating various iterations of the specification. @jdevillard has implemented a simple mechanism to switch to different versions of the specification in the web site.

I’m curious how you envision this is going to work.

Maybe the JEP-11 Lexical Scoping is separate because it has already been written. Let’s take the JEP-15 Third-Party Functions JEP as a first try to generate something.

There are two ways I can see this working:

1. Each pull request to the JEP repository triggers an automation that merges the contents of the JEP to the existing / current specification.rst and function include files to the JMESPath.site repository. That way, each new JEP adds to the existing specification.

2. Each pull request to the JEP repository triggers an automation that merges all the JEP repository contributions to create a target specification.rst that is pushed to the JMESPath.site repository. That way, JEP repository is the source of truth and specification can always be regenerated from the JEPs. However it comes with additionaly complexity to re-work past proposals.

I would like to tackle JEP-15 and try and help you setup the new automation.

[springcomp]

@innovate-invent ,

I think by analyzing more this issue and the latest commits, we may have a slightly different view for how to achieve the common goal to generate the specification from the JEPs.

There are multiple ways to skin a cat, so I think we should reach a common understanding before going any further. Here are the multiple ways I can see to approach this problem.

First, for the sake of the discussion, my understanding of the common goal, is to generate a JMESPath specification from JEPs automatically.

Target Format

• The JMESPath site uses Sphinx’s .rst format. So we could generate .rst files.
• Or we could (partially) upgrade the web site to support (in whole or in part) .md Markdown specifications.

My view is to reach an initial stage, a baseline that we could improve in the future to publish a first iteration of the spec as quickly as possible to build on the momentum and the energy that I currently have.

Since the website uses .rst format, my view is to continue using this format for the time being. My view is to slightly improve from day one by:

• Splitting the raw ABNF grammar file to its own text file. This is already done in the current version of the site.
• Splitting function documentations to their own files and include them from the main specification file. This is already done in the current version of the site.
• Splitting further each topic to its own file. This would have the main specification be an empty shell that includes most of the cross-referenced files.

JEPs to Specification

As alluded to in the previous post, we could do either of two things:

• Have all JEPs be the source of truth and generate a version of the specification from JEPs. To change the specification, we would need to fix at the JEP level and regenerate.

• Have the latest JEP pull request generate a version of the specification using the (current | previous) version of the specification as input. Based on the approach used from the previous question, the current | previous version of the specification could be in either Sphinx or Markdown format.

Miscellaneous

• I also considered using literate programming to intersperse portions of the JEPs to be extracted automatically to the target.
• Or we could not generate anything at all but have the contributor to the JEPs write the various necessary files (functions, topics) to be included as is in the documentation. This would work for new files, but we would need to find a way to alter existing text in the documentation.

Can you share your views on those items ?

[innovate-invent]

Sorry, I am still in the middle of preparing for interviews for a new job. My plan was to convert the GRAMMAR file to markdown using that regex I included in the PR. The functions would be handled separately by having some sort of structured format for detailing the function signature and description.

I was thinking to simply have a subfolder full of yaml files for this, each one representing a function. I am not against extracting this from the JEP itself though.

We need something with a similar structure (I was going to optionally extract the function name from the file name):

name: merge
topic: object manipulation
args:
#  required:
#    - name: 
#      type:
#      desc:
  optional:
    - name: argument
      type: [object]
      desc: Object to act as base from which all others override
    - name: "$..."
      type: [object]
      desc: Objects whose key-values override the previous
returns:
  type: [object]
  desc: Object with all key-values from the input
desc: |
  Accepts 0 or more objects as arguments, and returns a single object with subsequent objects merged. Each subsequent 
  object's key/value pairs are added to the preceding object. This function is used to combine multiple objects into one. You can 
  think of this as the first object being the base object, and each subsequent argument being overrides that are applied to the 
  base object.
examples:
- args: [{"a": "b"}, {"c": "d"}]
  returns: {"a": "b", "c": "d"}
- args: [{"a": "b"}, {"a": "override"}]
  returns: {"a": "override"}
- args: [{"a": "x", "b": "y"}, {"b": "override", "c": "z"}]
  returns: {"a": "x", "b": "override", "c": "z"}

'type' keys in this structure are lists of valid types in the format specified:

Similarly how arrays can specify a type within a list using the array[type] syntax, expressions can specify their resolved type using expression->type syntax. This means that the resolved type of the function argument must be an expression that itself will resolve to type.

Examples can also be ingested for compliance testing by the various respective implementations.

[innovate-invent]

I don't think I actually addressed your question. In the site build chain have it convert the grammar file and yaml (or whatever) to either rst or md using templates, then render the entire site. If it is just as easy to skip the intermediate rst/md then I would go for that, but otherwise this is a decent stopgap.

Include a github workflow that triggers on tag of the main branch, that then triggers the site repo workflow to rebuild: https://github.community/t/triggering-by-other-repository/16163/12

I like the idea of separating the functions from the spec, but I am not sure I would want each spec topic to be on a different page. I want to be able to ctrl-f items in the spec.

Functions should be all listed on the same page, per topic (string manipulation, object manipulation, type conversion, ..). It is important to leave opportunities for users to discover related functions just by scrolling.

[springcomp]

I like the idea of separating the functions from the spec, but I am not sure I would want each spec topic to be on a different page. I want to be able to ctrl-f items in the spec.

Oh I was talking about separating source files, but the target is a single page as before, of course. To simplify the automation tools. But then, I like the suggestion to create categories of functions. That fact that function files are physically separate in the source and could be included in more that one page will make it easier.

[innovate-invent]

Gotcha. Are you saying you want to break up the GRAMMAR file then?

I think having a single GRAMMAR file is ideal due to the referential nature of ABNF. I want to be able to ctrl-f references in the file without having to look through other files. Ideally I would like to include parsing the ABNF to include tag links for all references to allow easily jumping around the spec in a single webpage. The way I have set up the current GRAMMAR file should easily allow for this.

[innovate-invent]

Also, if you can give me another..week? I will then have time to create all of the templates and parsing logic I have been mentioning

[springcomp]

Gotcha. Are you saying you want to break up the GRAMMAR file then?

I like the single grammar file as it is self-sufficient. I admit I do not have a good answer for the portions of the spec that reproduce and highlight portions of the ABNF grammar. I guess we can just duplicate those parts as is the case now. Keep it like it is now.

Sure take your time. I'll wait for your contributions whenever you are ready.

[innovate-invent]

I am going to close this given the move to the GRAMMAR file and function yml files.

I have an abnf validator in the works, it might not be done before we ask the community for feedback.