Create external documentation plugin

[picklejuicedev]

Description

Given the current doc system (i.e. kedro build-docs) will be discontinued with version 0.19, I would suggest a plugin should fill its place.

Context

I think ducmentation is a first class citizen, enabling both non-technical stakeholders and new-comers to a team to be included. Also, when documenting or explaining a project, flaws in both implementation and usage become very apparent and can be fixed.

Possible Implementation

In order to aid discussion, I played around with a prototype static site as an example for project documentation. I used the Kedro spaceflight tutorial project as a base, and you can explore the finished documentation here. Given that the Kedro framework defines much of the information needed for project documentation, I think it would be pretty straight forward to create a plugin that would:

• create the basic documentation structure
• fill in details about pipeline, nodes, data and parameters automatically
• insert an interactive Kedro-Viz graph
• provide empty templates for additional notes to write
• example how to publish as a static website, for example to GitHub Pages

Possible Alternatives

The above example has been implemented using the mkdocs framework. Clearly there are more alternatives that could be used. Also maybe the README.md is perfectly sufficient for what the users need?

[astrojuanlu]

I think this is an excellent idea @picklejuicedev. There could be 2 essential bits of functionality here:

• A command to boostrap a somewhat comprehensive documentation project, with the ideas that you've proposed. Super useful because creating this structure is tedious, and the plugin could provide a good starting point, in line with the Kedro philosophy. What's more, one could think of several "docs starters", similar to how kedro-starters work for kedro new.
• A command to rebuild the docs after the user has made edits to such bootstrapped project. Maybe not so useful, since it's a one liner in Sphinx (cd docs && make html or even sphinx-build docs/source docs/build/html) and for MkDocs it's mkdocs build.

[picklejuicedev]

Thanks for the comments, @astrojuanlu.

I agree it would start with bootstrapping the general structure, not just to remove the tedious work, but also create consitency in docs for different projects. Indeed the publishing command is optional, however I think it would also need a sort of update command as I would imagine that dynamically pulled information from data schemas etc may change over time and need to be regenerated.

I haven't been following the forum lately - do you think this would actually be a feature that would be popular in the community? Not sure there would be many active users - which is why I left it for the time being.

[astrojuanlu]

I haven't been following the forum lately - do you think this would actually be a feature that would be popular in the community? Not sure there would be many active users - which is why I left it for the time being.

This is something that we are trying to figure out as well. We initially deprecated the command because our telemetry indicated that it was not widely used, but then we got signals from the community that it was indeed very much loved. We are evaluating the best course of action in gh-2149. There's more discussion in gh-2381.

[picklejuicedev]

There is also #2072 that covers this discussion about what framework is actually recommended. Also an interesting thead with a user here how to render kedro-viz to a standalone static website. I hacked that in my prototype, but I could see this as a possible idea. Extend kedro-viz to render project details and then be able to push a static version to a site?

[astrojuanlu]

Also an interesting thead with a user here how to render kedro-viz to a standalone static website.

Indeed, we got asked about this in our last user interview (more specifically: a Sphinx plugin).

[antonymilne]

Another data point from Slack on the deprecation of kedro-docs:

I'm not particularly qualified to opine on this matter, but it would be nice for the docs no0bs such as I to have a simple way to render Kedro project docs, leaving some room for theme choices, etc.

[astrojuanlu]

I was supportive of this task but I think the most realistic thing to do is to say that we will not build this. The time of the Kedro core team is better spent improving the framework itself, so this "create a plugin" request is legitimate but something we prefer the community ownership of. Maybe there can be kedro-sphinx and kedro-mkdocs plugins and we see which one "wins" 😉

I'm closing this as "not planned".

[astrojuanlu]

It was highlighted in @NeroOkwa's research that this could indeed be a nice addition. The original "possible implementation" is a good starting point https://github.com/kedro-org/kedro/issues/2075#issue-1467831903

Still not a priority, but I'm reopening.