VS Code doc hosting integration

[davidanthoff]

I've been thinking a fair bit about a more comprehensive documentation viewer inside VS Code, and I think Publish.jl seems to be the ideal starting point for that.

For the design I have in mind, how difficult would it be to add another output option to Publish.jl (maybe named "vscode") that does the following:

• Write a JSON file with the TOC, and potentially any other "meta" info like additional CSS style sheets etc that are needed. But even just starting with just the TOC would be great.
• Writes out every content page as a markdown file. I guess for pages that are originally authored in MD that wouldn't require much at all, but then things like notebooks or literate content stuff, it would also convert that into a markdown file. Again, though, we could start with just including MD files (which I guess would be essentially copying them into the output folder?).
• Also writes all "assets" (images etc.) into the output folder, so that the relative paths in the markdown files that point to these all work.

My guess is that this might be fairly straightforward?

If Publish.jl supported that, here is how I would take it from there:

• we are currently working on a setup where each tagged package version in the registry gets analyzed in the cloud by us and then we store lots of information about that package. Ideally we would just start to also run Publish.jl as part of this for every package/version, and store the output I described above in the cloud along the other info we already cache there.
• The Julia extension in VS Code would then download this information on demand and have an integrated documentation browser. What the browser shows would depend on the active Julia environment, essentially it would simply show all the documentation for all the packages that are currently loaded in the active Julia environment.

[MichaelHatherly]

Yes, this is definitely part of the goals for the package, so I think it should be possible to make it work.

A JSON+MD output would be a viable option I think. There's the need to process and "expand" the {cell} syntax so we can't just dump the user's markdown files directly, but CommonMark is meant to be a round-tripable markdown parser/writer so any loss of information/structure will just be a bug to be fixed. The TOC should be simple enough to export as JSON. Asset files, whether user-generated or {cell}-generated shouldn't be a problem.

[davidanthoff]

Cool, that sounds very promising!

Do you think we could start with a simple copy-version, even if it doesn't resolve {cell} and other things for now? At the moment I would be quite interested in just getting a very rough prototype going that works end-to-end. So I think just having a version that a) writes the JSON index file, and copies the markdown files over would be good enough for me to start with the rest of the pipeline. We could even ignore assets for now. But I could then start to work on the VS Code integration stuff, and simultaneously we could then improve the situation over here so that it does the necessary transformations etc.

Do you think you could maybe start with that functionality? I think if you could just start with a very rough JSON write and file copy story, that would give me probably enough starting points that I could then down the road improve and refine things. But right now I wouldn't know where to start, really :)

[MichaelHatherly]

I think if we can spec what's needed in the toc.json file then the rest should be relatively straight forward to cook up in a basic form.

My understanding is that we would want the nested structure of the output markdown pages included in the project as well as page metadata such as title. There's also some project level metadata we can include as well that comes from the Project.toml [publish] section such as keywords, authors, etc.

Another thing to consider is what markdown parser will be used on vscode side for viewing, the default one I assume? Would be good to know what subset of CommonMark.jl it actually supports.

[davidanthoff]

Yeah, exactly!

I think for the JSON, we literally could start with just the TOC hierarchy, something like this:

{
  "name": "VegaLite.jl",
  "page": "foo.md",
  "children": [
    {
       "name": "Introduction",
       "page": "intro.md",
    },
    {
      "name": "Something else",
      "children": [
        {
          "name": "Sub 1",
          "page": "something.md",
          "section": "somesection"
         }
      ]
    }
  ]
}

Essentially just one element per node in the tree, and then for each node a name and a page field, and an optional section field if it links to some sub section on that page. And then an optional children field, to hold any sub nodes.

For the rendering we're going to use markdown-it, we are using that already for our document browser implementation that we are going to ship very soon (Mon or so). markdown-it is pretty flexible, so I guess in general there will be a question whether it makes more sense to output a markdown flavor that can be used without modifications in markdown-it from Publish.jl, or whether to customize markdown-it to read whatever the default Publish.jl output is. I'd say for a first pass we can just ignore that? I think right now I just want to get to a point where it shows something, and then we can iterate to sort out details?

[davidanthoff]

@MichaelHatherly Bump, just wondering whether you might have some cycles to push this forward? I think if we were to move just to the next step here on the generation side, I might be able to implement enough of it on the VS Code side of things that we could showcase this as one of the new features in the VS Code extension at this year's Juliacon, which would be neat. But I think I would need some help on this side of things :)

[MichaelHatherly]

Yeah, I'll see if I can hack something together today, it's unlikely to be very pretty for now, and will likely need heavy refactoring at a later stage, but this has been waiting for quite a while now 🤣

[davidanthoff]

Tada, this is very cool! I left some specific comments on the PR, but have one generic question: how difficult do you think it would be to generate this output (i.e. toc.json and the corresponding markdown files) for the docs of Julia itself? If we had that, we could include all of the base docs from day one on and then docs for packages that use Publish.jl. Having base ready from the beginning would be a lot of docs, which would be nice.