Md processing (codeblock tests / index generator)

ryyppy commented 5 years ago

There are two goals of this PR:

[x] Generate a proper search index for the Belt API
[x] Parse all codeblocks with certain meta attributes (e.g. example) and run it through bsc

The parsing is done via unified.

Fixes #15

vercel[bot] commented 5 years ago

This pull request is automatically deployed with Now. To access deployments, click Details below or on the icon next to each push.

Latest deployment for this branch: https://reasonmlorg-git-md-processing.reason-association.now.sh

ryyppy commented 5 years ago

As a sidenote, there is are multiple reasons why this is not written in ReasonML:

We are using unified to parse the AST of the markdown, which is very tedious to type since each plugin has different ways to manipulate the end result.
BuckleScript is set up to output ES6. This means we would need dependencies such as @std/esm and run node via node -r esm scripts/myscript.bs.js just to make it parse... not a fan of this

ryyppy commented 5 years ago

Test script is done, index file generation is (at least infra wise) also done.

After merging this, I will use the index data to build a nice search component.

Here is a taste on how the test tool feels like:

$ node scripts/test-belt-examples.js pages/belt_docs/array.mdx
Testing example in 'pages/belt_docs/array.mdx' on line 21...
Testing example in 'pages/belt_docs/array.mdx' on line 118...
Testing example in 'pages/belt_docs/array.mdx' on line 134...
Testing example in 'pages/belt_docs/array.mdx' on line 146...
Testing example in 'pages/belt_docs/array.mdx' on line 160...
Testing example in 'pages/belt_docs/array.mdx' on line 187...
Testing example in 'pages/belt_docs/array.mdx' on line 205...
Testing example in 'pages/belt_docs/array.mdx' on line 237...
Testing example in 'pages/belt_docs/array.mdx' on line 269...
Testing example in 'pages/belt_docs/array.mdx' on line 291...
Testing example in 'pages/belt_docs/array.mdx' on line 303...
Testing example in 'pages/belt_docs/array.mdx' on line 319...
Testing example in 'pages/belt_docs/array.mdx' on line 335...
Testing example in 'pages/belt_docs/array.mdx' on line 353...
Testing example in 'pages/belt_docs/array.mdx' on line 373...
Testing example in 'pages/belt_docs/array.mdx' on line 402...
Testing example in 'pages/belt_docs/array.mdx' on line 438...
Testing example in 'pages/belt_docs/array.mdx' on line 470...
Testing example in 'pages/belt_docs/array.mdx' on line 501...
Testing example in 'pages/belt_docs/array.mdx' on line 521...
Testing example in 'pages/belt_docs/array.mdx' on line 543...
Testing example in 'pages/belt_docs/array.mdx' on line 564...
Testing example in 'pages/belt_docs/array.mdx' on line 584...
Testing example in 'pages/belt_docs/array.mdx' on line 604...
Testing example in 'pages/belt_docs/array.mdx' on line 634...
Testing example in 'pages/belt_docs/array.mdx' on line 665...
Testing example in 'pages/belt_docs/array.mdx' on line 705...
Testing example in 'pages/belt_docs/array.mdx' on line 727...
Testing example in 'pages/belt_docs/array.mdx' on line 747...
Testing example in 'pages/belt_docs/array.mdx' on line 767...
Testing example in 'pages/belt_docs/array.mdx' on line 787...
Testing example in 'pages/belt_docs/array.mdx' on line 809...
Testing example in 'pages/belt_docs/array.mdx' on line 831...
Testing example in 'pages/belt_docs/array.mdx' on line 857...
Testing example in 'pages/belt_docs/array.mdx' on line 886...
Testing example in 'pages/belt_docs/array.mdx' on line 911...
Testing example in 'pages/belt_docs/array.mdx' on line 927...

-------Results for 'pages/belt_docs/array.mdx'----------
pages/belt_docs/array.mdx:21 => ok
pages/belt_docs/array.mdx:118 => failed
pages/belt_docs/array.mdx:134 => failed
pages/belt_docs/array.mdx:146 => failed
pages/belt_docs/array.mdx:160 => ok
pages/belt_docs/array.mdx:187 => failed
pages/belt_docs/array.mdx:205 => failed
pages/belt_docs/array.mdx:237 => failed
pages/belt_docs/array.mdx:269 => ok
pages/belt_docs/array.mdx:291 => failed
pages/belt_docs/array.mdx:303 => failed
pages/belt_docs/array.mdx:319 => failed
pages/belt_docs/array.mdx:335 => failed
pages/belt_docs/array.mdx:353 => failed
pages/belt_docs/array.mdx:373 => failed
pages/belt_docs/array.mdx:402 => failed
pages/belt_docs/array.mdx:438 => failed
pages/belt_docs/array.mdx:470 => failed
pages/belt_docs/array.mdx:501 => failed
pages/belt_docs/array.mdx:521 => failed
pages/belt_docs/array.mdx:543 => failed
pages/belt_docs/array.mdx:564 => failed
pages/belt_docs/array.mdx:584 => failed
pages/belt_docs/array.mdx:604 => failed
pages/belt_docs/array.mdx:634 => failed
pages/belt_docs/array.mdx:665 => failed
pages/belt_docs/array.mdx:705 => failed
pages/belt_docs/array.mdx:727 => failed
pages/belt_docs/array.mdx:747 => failed
pages/belt_docs/array.mdx:767 => failed
pages/belt_docs/array.mdx:787 => failed
pages/belt_docs/array.mdx:809 => failed
pages/belt_docs/array.mdx:831 => failed
pages/belt_docs/array.mdx:857 => failed
pages/belt_docs/array.mdx:886 => failed
pages/belt_docs/array.mdx:911 => failed
pages/belt_docs/array.mdx:927 => failed
-----------
Summary:
Total Examples: 37
Failed: 34
Success: 3

Tip: You can also run tests just for specific files / globs:
`node scripts/test-belt-examples.js "pages/belt_docs/array.mdx"`

-----------
Error Preview:

  We've found a bug for you!
  pages/belt_docs/array.mdx: 121:10-23

  1 │ let arr = [|10, 11, 12, 13, 14|];
  2 │
  3 │ let () = reverseInPlace(arr);
  4 │
  5 │ arr == [|14, 13, 12, 11, 10|];

  The value reverseInPlace can't be found

Notice how the file path points exactly to the line inside the array.mdx file. I love it!

ryyppy commented 5 years ago

For the bsc -i -e command, we were required to update to version 5.2.0. Make sure to yarn install after merging / pulling.

ryyppy commented 5 years ago

Also here an example on how index data (parsed from the result.mdx file) looks like:

    {
        "headers": [
            "t",
            "getExn",
            "mapWithDefaultU",
            "mapWithDefault",
            "mapU",
            "map",
            "flatMapU",
            "flatMap",
            "getWithDefault",
            "isOk",
            "isError",
            "eqU",
            "eq",
            "cmpU",
            "cmp"
        ],
        "href": "belt_docs/result",
        "moduleName": "Result"
    },

rescript-association / reasonml.org

Md processing (codeblock tests / index generator) #28