search

jupyter / jupyter_markdown

Documentation and tests related to Jupyter's Markdown syntax
BSD 3-Clause "New" or "Revised" License
8 stars 10 forks source link

Test against all available versions of the spec (including newly released versions) #13

Open mpacer opened 7 years ago

mpacer commented 7 years ago

We will want to test against multiple versions of the spec.json.

This will be important if we are to detect how our tools are evolving in comparison with the spec.

Currently the only two on commonmark's site that have spec.json endpoints are 0.27 and 0.28.

This is going to be a little trickier since it requires a few separate steps, so while I'm going to tag this as sprint friendly, I would not recommend that someone tries to do this without direct guidance from a maintainer.

mpacer commented 7 years ago

Doing this right involves at least three steps (which can be tackled sequentially and may be worthwhile splitting off into their own issues):

1) automatically identify which versions of the spec exist i) it seems that the canonical place to obtain the spec.json from is from spec website: http://spec.commonmark.org/ or the github repo that holds the source for the spec site: https://github.com/commonmark/CommonMark-site ii) releases from commonmark/cmark (the canonical commonmark implementation) are not versions of the spec iii) releases from commonmark/CommonMark (the origin of the code generating the spec) does not seem to be canonical, as there are multiple versions released (0.21.0 and 0.21.1) that map onto one canonical version of the spec (0.21) iv) as of this writing (2017_W33_3_1216) the releases with spec.jsons are 0.13–0.28 2) create an interface to (optionally) pull down any subset of the released spec versions & runs the code on those versions (we should do this at least once and it should be reproducible) 3) build a default pipeline for grabbing only the most recent version of the spec (ideally the one defined by the redirect at http://spec.commonmark.org/current), creating the canonical notebook set (see #14 #11), analysing them, &c.