Support for literate programming

[samoht]

It would be nice if we could have first-class support for literate programming in dune. The idea would be to tag certain files as "file which needs to be pre-processed" to generate a file in a way very similar to what is done for the Reason support.

And example is here where the ocaml code blocks in a markdown file can be extracted into a self-contained .ml file which can compiled and linked like a normal OCaml file.

A bonus would be to support setting various pre-processor (using different aliases I guess), for instance to set-up cram-tests (e.g. check that :: and 1> outputs are correct). This is related to #622 and #771

[samoht]

An update:

• https://github.com/samoht/mdx is now able to pre-process markdown file and generate a valid OCaml file (while keeping locations): ocamlc -pp "mdx ppx" -impl README.md just works
• There is a new branch in merlin which allows to pass -pp "foo" in the FLG arguments. That would be nice if dune would add this automatically when certain files are flagged. This allows to have merlin working in markdown code blocks, which is nice.

[ghost]

To support that we just need to generalize reason support. We can have a way to setup a scheme for source files:

(source_file_scheme
 (implementation
  (extension .mdx)
  (preprocess (mdx %{input-file}))
 (interface
  (extension .mdxi)
  (preprocess (mdxi %{input-file}))))

Then dune would automatically recognize .mdx files as implementation file and .mdxi files as interface files.

[samoht]

Note that the file extension will remain .md to stay compatible with GitHub preview/editors.

[rizo]

@samoht This looks very cool!

Does it make sense to have support for odoc too? Although markdown is a very popular format (and the one supported by GitHub), the main documentation format we use for all our packages is still ocamldoc. It would be very useful to have this feature in the mld files, for example.

[ghost]

@rizo there should be no need to do anything special to have support for odoc since odoc takes cmt files as input

[samoht]

What are mld files? Just md files with a different extension? :-)

[ghost]

Ah, I misunderstood the question. @samoht, mld files are additional odoc files that use the ocamldoc syntax. For instance you may have an index.mld file.

[rizo]

@samoht mld files are documentation pages that are now supported by odoc and dune. They are like mli files containing a docstring and can be parsed and converted to HTML by odoc. Library authors can use them to write tutorials or customise the index.mld page for their packages as @diml suggested.

So to clarify my question. If I have an mld or mli file, do you think it would be possible to find the OCaml code blocks (defined with {[ ... ]}) and execute those with mdx? From what it looks mdx currently directly depends on markdown syntax.

I would be very interested in contributing support for odoc.

EDIT: Here're some WIP docs for mld files: https://ocaml.github.io/odoc/odoc/using-odoc.html#doc-pages

[samoht]

So to clarify my question. If I have an mld or mli file, do you think it would be possible to find the OCaml code blocks (defined with {[ ... ]}) and execute those with mdx? From what it looks mdx currently directly depends on markdown syntax.

Yes that would be possible. Currently mdx is just looking at code blocks with triple back-ticks. It should be pretty easy to provide an other frontend/binary doing the same kind of transformations (pp and test). Feel free to open an issue in mdx and I'll be happy to guide you to add the odoc support :-)

[rizo]

Excellent, thank you! :-) Opened an issue as you suggested: https://github.com/samoht/mdx/issues/17.

[rizo]

Update: I started working on the code block extraction logic in odoc (https://github.com/ocaml/odoc/pull/303). This will allow us to annotate and extract code blocks found in mli and mld files for execution and promotion with dune.

I wrote a document about this feature, it explains how the dune integration could work. The design is still in flux, but if you have any ideas or suggestions for this feature, please let me know.

[rgrinberg]

We've added first class support for dialects. I think literal programming should fall under that. If dialects are in any way deficient, please raise separate issues.