Docs engine request

[casperdcl]

Implementation request for website team (if you're from CML or Documentation teams and want to discuss specifics please see #55 instead).

• Can we have a new doc subdomain & page (the "Doc(s)" link in the hero/header should go to https://doc.cml.dev which should have a similar layout to https://dvc.org/doc)
• In addition, we need subpages. All pages should have a ToC with a nested folder structure corresponding to layout in the cml-website repo.

Blank pages corresponding to folders:

• doc.cml.dev/ CML Documentation
  • /start/ Getting Started
  • /start/github/ Getting Started with GitHub <-- sub-page
  • /ref/ Command Reference
  • /ref/#send-comment cml-send-comment <-- in-page <h2>s which show up in the ToC
  • ... etc ...

Each page can be blank for now (I'll fill in content) :pray: @rogermparent @julieg18

[julieg18]

Awesome! I'm glad we're planning to place documentation on to cml.dev 😄 A couple questions about this:

• Is there a reason why we're planning to add documentation on https://doc.cml.dev instead of https://cml.dev/docs? And if this will be on a subdomain, with documentation code be inside this repo or a separate repo?
• When you say similar layout, are we planning to basically copy dvc.orgs design, or are we going to need some new designs for the documentation?
• When it comes to creating the pages, are we planning to do it the same way dvc.org does it? Aka, created markdown pages, converting them to nodes, create pages with those nodes, etc.

[jorgeorpinel]

@julieg18 I think a lot of this is up to us (well, you guys) to decide. Quick comments:

• I'd make the subdomain question a separate issue or put it aside for now if it complicates things.
• I think that straight-up copying the docs engine from dvc.org would make sense. And it could help us determine how best to encapsulate it for other product websites in the future (in fact we see dvc.org as a side open-source product almost).

[casperdcl]

doc.cml.dev instead of cml.dev/doc~s~?

this is a pending discussion actually, but hopefully that's mostly the docs team's problem and won't affect the bulk of the web eng request.

And if this will be on a subdomain, with documentation code be inside this repo or a separate repo?

Same repo - personally I often tend to set up my hosting configs/engines to mirror X.domain.com/ and domain.com/X/ so I was hoping that it would be easy to set up the custom engine here to push a doc/ folder in this repo to either public URL format (subdmain or subfolder).

are we planning to basically copy dvc.orgs design

Copy the effective functionality, and do whatever is easiest for design for now (long term plan is to keep design the same, but if for now just having a basic design is easiest then go ahead)

When it comes to creating the pages, are we planning to do it the same way dvc.org does it? Aka, created markdown pages, converting them to nodes, create pages with those nodes, etc.

Yes plz. I don't know if you want to discuss with @shcheklein other higher-level issues of potentially switching to gh-pages (Jekyll) and our own Jekyll theme (should make web eng much easier in the long term?)

[casperdcl]

straight-up copying the docs engine from dvc.org

definitely something to ask @shcheklein about - rather than copy it's probably best to extract into separate repo and then have both websites depend on it. At that stage again I'd say gh-pages + custom Jekyll theme is a better idea than maintaining our own engine (equally powerful/customisable with *.html|js|scss and no headache of maintaining an entire engine).

[rogermparent]

straight-up copying the docs engine from dvc.org

dvc.org is very unique as a Gatsby website, and to an extent that unique architecture is a weakness as much as (if not more than) a strength- many parts of the more intricate functionality are a bit fragile, and very unwieldy to fix or add to.
To @casperdcl's point, using and contributing to an existing docs solution is, at least in my opinion, the clear path forward as it saves us development resources, gives us features, and allows us to contribute to the wider community by improving or writing plugins/guides/content for an open source docs project.

Whether that solution be a Gatsby plugin or GH Pages + Jekyll, especially since the doc writing experience between the two is nearly identical, depends on what we want out of the docs user experience compared to what GHP+Jekyll solutions can offer us.

If we want plain ol' HTML text docs with a sidebar and search like 95+% of docs do (for good reason, don't get me wrong) then a Jekyll GHA site could really simplify things. On the other hand, if we want more functionality on top of that like interactive custom components or integrating with the rest of the site, the cost/benefit starts to shift more in Gatsby's favor.

Should we decide to go with GHP+Jekyll docs, we can try to integrate it with the existing site, which would take on a "marketing" role as opposed to the GHP docs. It could be that having the sites use the same colors and link to each other is enough integration, but doing so will require us to maintain a separate Jekyll-compatible design alongside our React one.

in summary:

• I agree that we should find an established existing docs engine instead of completely maintaining our own
• We should compare and contrast the cost/benefits of docs solutions both within Gatsby and outside, like GHP+Jekyll.

edit: also worth noting Gatsby can be used on GH Pages, though I'm not confident that we'd gain much overall compared to our current setup, particularly where we're already very invested in Heroku hosting.

[shcheklein]

My take and context:

• I didn't like existing engines - they were hard to customize to create that level of UX I wanted. I'm almost sure they are still. And I hope we'll be evolving our docs even further which will require even more customizations, branding, etc.
• It's fine to start with one of those for CML - I don't mind. I would try to use something more or less compatible in nature with what we have (MD files with frontmatter, we should have preview envs, ideally branding, etc, etc) - so that we can take the structure when it's needed and migrate to our engine.
• It's fine to take the existing one and copy paste it (documenting changes that we had to make to get it compatible with CML.dev)

Can we try to prototype something with Jekyll to see how much stuff we'll have to write (it might turn out that pretty much the same amount):

• we need branding (website feels the same. no ads)
• we need preview envs
• we need CI/CD things like - link checks, formatting, restyle
• we'll most likely want ability to switch modes (GH/GL) with tabs, other dynamic elements
• custom syntax highlighters
• custom logic to link pages (w/o us manually inserting links every time)
• navigation that makes sense (e.g. I open a page I want side bar to make it visible in it, scroll to the right position, etc)
• anchors for subsections (and eventually we will want to control this - it comes handy like in the troubleshooting case with DVC)

[casperdcl]

So summary:

1. somebody build a gh-pages proof of concept? I could do this but don't know if I have the time. All of the required features can be implemented with Jekyll (liquid + JS + custom templates + plugins), except for preview envs which can use e.g. https://www.netlify.com/github-pages-vs-netlify/ or we can just use GHA to static build and push to some artefact host (or both GHA & Netlify).
2. copy-paste dvc.org docs engine into cml.dev, but keeping track/documenting changes carefully, to help with long-term plan of abstracting engine to a separate repo

If there are no takers for (1) then we should get started on (2) soon because we need docs!

[shcheklein]

Okay, I can find some resources (at least try). But quick questions before - why is it p0? :) To me still content is most valuable part - we can iterate on the content within dvc.org for a while, right?

p0 - usually means that something is completely broken, it severely blocks someone from doing anything, etc.

[casperdcl]

Well this is an issue on the cml-website repo so I assumed p0 was the most appropriate tag since it's more important than the other issues on this particular repo atm? Feel free to downgrade if you disagree.

[shcheklein]

@casperdcl no, I'm fine, you decide with the team on this. I just shared what it means in other repos.

[rogermparent]

I think this is easily at least p1, p0 may be a bit much for the entire docs engine but at least making a plan to move forward with is p0-worthy.

If there are no takers for (1) then we should get started on (2) soon because we need docs!

I don't think there's takers for 1, without a prebuilt Jekyll solution that takes care of all the functionality we want, which given this list seems unlikely, we'll want to move forward with 2, extending the Gatsby site, at least for now.

We should be able to reuse quite a bit of DVC's docs engine logic, and it's worth investigating, but I can't guarantee it'll translate easily or well and will require some research.

@julieg18 would you be able to do some research into dvc.org's docs engine, specifically oriented toward implementing it on other sites? Off the top of my head, I believe they're based on gatsby-plugin-remark and use this technique for using custom components (unlike the blog, which uses a different way) and sidebar logic is scattered in a few util files and components.
There's a lot of extra config that has to be applied to get everything together so I suspect some files will have to be blended together and it won't be truly "drop-in".

[julieg18]

@julieg18 would you be able to do some research into dvc.org's docs engine, specifically oriented toward implementing it on other sites?

Sure! When you say "dvc's docs engine" are you referring to the way that dvc.org renders the documentation pages? Taking the markdown, adding or converting certain elements, creating nodes, using nodes to create pages, etc?

[shcheklein]

I'm assigning this for myself for now. Let me try to get someone do this first step fast. Then we can think about generalizing, etc.

[rogermparent]

Sure! When you say "dvc's docs engine" are you referring to the way that dvc.org renders the documentation pages? Taking the markdown, adding or converting certain elements, creating nodes, using nodes to create pages, etc?

Yep! That and the sidebar, which is equally important to the content. Some specific features we need from the sidebar:

• manually defined order
• n-level nesting (usually 2, probably no more than 3)
• overridable labels
• the ability to change the bullet to an image on certain special items (usually home and other Iterative projects)

[julieg18]

Are we good to close this issue since the docs engine is now implemented?

[casperdcl]

fixed by #60