feat: legacy html generator

ovflowd commented 3 months ago

Description

This PR introduces the Legacy HTML generator, which aims to generate an almost 1:1 replica of the existing https://nodejs.org/api/ documentation.

This PR introduces the fully-fledged legacy-HTML generator, with all the related utilities and tools to make complete API docs in HTML.

Diversions from the original version:

Some of the outputted HTML in the "view source" might be different
The styles got a bit updated due to CSS conflicts with Shiki (Syntax Highlighter)
A new syntax highlighter is being used (same as the Node.js Website), meaning token colors will differ.
Sidebar Navigation is generated on-demand instead of relying on nodejs/node/doc/api/index.md
- This means some of the entries are a bit different, although I believe they are fine.
Copy button for CodeBoxes on the top instead of the bottom.
Stability Index position is slightly below now (below the History Table and Source Link)

Things to be done:

None. We're feature complete!

Future generators

json generator, being able to generate .json files for API docs
react generator, the redesign of API docs generator.

Validation

Clone, run npm install, then run the command against Node.js API docs, for example:

npx api-docs-tooling -i "/path/to/node/doc/api/*.md" -t legacy-html -o /tmp/out

You can just open the desired files on your browser.

Preview

This is deployed temporarily on Netlify (not auto deployed on commits, I'm manually deploying it): https://node-api-docs-legacy-preview.netlify.app/

ovflowd commented 3 months ago

cc @nodejs/collaborators and @nodejs/documentation as this PR changes a few aspects on the generated HTML as mentioned above.

targos commented 3 months ago

Dropped support of all.html

Can this be avoided?

ovflowd commented 3 months ago

Dropped support of all.html

Can this be avoided?

Is there any concrete reason why? Afaik, nothing officially gets referenced to all.html. We can still write an all.json, tho,

But yes, if this is a blocker, I can do it differently, but I'm just wondering if we could avoid an all.html as that would add some extra complexity. (and running time of the tooling, as pretty much we're going through everything twice)

targos commented 3 months ago

Well I'm using it a lot (to CTRL+F because we don't have a good docs search feature), so I expect it to be the case for other people. This is a disruptive change that I would not like to see before the docs redesign.

ovflowd commented 3 months ago

Well I'm using it a lot (to CTRL+F because we don't have a good docs search feature), so I expect it to be the case for other people. This is a disruptive change that I would not like to see before the docs redesign.

Gotcha, yes, the docs redesign aims to solve this. To be honest, the https://nodejs.org search box already includes all the API data, and it is continuously updated. (Also, the search box API data indexing will be done on the nodejs.org repository, so technically, it is decoupled from the actual API docs, or at least so it will be for the initial implementation.

For local dev environments, I believe it can source the results locally, but I'm not sure if they will match since Orama (our search provider) has some extra processing on their cloud.

I understand this is a pain point, but I do feel hesitant on an all.html, but I believe I can make it happen.

targos commented 3 months ago

To be honest, the https://nodejs.org/ search box already includes all the API data

It only works for the latest LTS version (AFAIK)

ovflowd commented 3 months ago

To be honest, the nodejs.org search box already includes all the API data

It only works for the latest LTS version (AFAIK)

Correct, it indexes the LTS version. That's a good point. @micheleriva, what are the alternatives here? Can we make it for API doc generation so the whole database can be built locally based on metadata we ingest? i.e., is it a static database stored in a static file and loaded by the Orama client during bootstrap (client side)

ovflowd commented 3 months ago

@targos I've added a legacy-html-all generator which extends on the legacy-html generator for generating the API docs + all.html

ovflowd commented 3 months ago

cc @nodejs/web-infra, can we have a 2nd thorough review and verify that the pages match?

I still need to update the code that generates the CodeBoxes and the Stability overview. I'll do that later tomorrow or today.

micheleriva commented 3 months ago

Correct, it indexes the LTS version. That's a good point. @micheleriva, what are the alternatives here? Can we make it for API doc generation so the whole database can be built locally based on metadata we ingest? i.e., is it a static database stored in a static file and loaded by the Orama client during bootstrap (client side)

@ovflowd sorry for my late reply. Yes, we can definitely do this. Let me just point out that the search results may slightly differ from Orama Cloud (more performant, more accurate results) and Orama OSS. I hope that's not a dealbreaker.

ovflowd commented 3 months ago

Correct, it indexes the LTS version. That's a good point. @micheleriva, what are the alternatives here? Can we make it for API doc generation so the whole database can be built locally based on metadata we ingest? i.e., is it a static database stored in a static file and loaded by the Orama client during bootstrap (client side)

@ovflowd sorry for my late reply. Yes, we can definitely do this. Let me just point out that the search results may slightly differ from Orama Cloud (more performant, more accurate results) and Orama OSS. I hope that's not a dealbreaker.

I think that's a fair trade-off, at least within directly the Node.js API docs pages. Are there docs out there on how we can use this?

micheleriva commented 3 months ago

I think that's a fair trade-off, at least within directly the Node.js API docs pages. Are there docs out there on how we can use this?

We have our docs at https://docs.orama.com/open-source, and you would probably need to use that plugin: https://docs.orama.com/open-source/plugins/plugin-data-persistence.

We can give you support probably starting next week or so to write down a reference architecture/implementation 🙏

ovflowd commented 3 months ago

This PR is quite massive (would it have been possible/feasible to break this up—at least smaller chunks into a feature branch that was then wholesale merged?),

I think I could move the bug fixes to a different branch, yes, and the unrelated changes. I'll do this when this branch is feature-complete. Although the PR is open, I believe it should be a draft PR.

ovflowd commented 3 months ago

I think that's a fair trade-off, at least within directly the Node.js API docs pages. Are there docs out there on how we can use this?

We have our docs at docs.orama.com/open-source, and you would probably need to use that plugin: docs.orama.com/open-source/plugins/plugin-data-persistence.

We can give you support probably starting next week or so to write down a reference architecture/implementation 🙏

Awesome! cc @canerakdas or @AugustinMauroy can any of you open an issue to keep track of the search box work for this repo? For all's worth, it should be another generator to generate the Orama local database.

The search component itself would be either the same from the Node.js website or on the react generator here in the future; Butb the database generation should be a generator.

ovflowd commented 3 months ago

I've rebased the branch and am going to keep working on the features.

ovflowd commented 3 months ago

@AugustinMauroy tests are failing, I assume because I changed the way how stability indexes are built. Can you look into it?

ovflowd commented 3 months ago

cc @nodejs/web-infra another review please 🙏 I changed the HTML creation to be mostly

ovflowd commented 3 months ago

FYI: After the next review, I once again will need to decouple changes. Apologies for the prolonged review, implementing this is time consuming, and I want to ensure things work as expected.

ovflowd commented 3 months ago

FYI @nodejs/web-infra I split the changes: https://github.com/nodejs/api-docs-tooling/pull/66, waiting for review to getting it merged back here.

ovflowd commented 3 months ago

Alrighty, I believe we are very close to have this generator done. I believe this PR is pretty much ready to be reviewed, since the last missing piece is having the CJS/ESM toggles, which I'm going to do soon whenever I have time :)

ovflowd commented 3 months ago

cc @nodejs/web-infra please add another round of reviews 🙏

canerakdas commented 3 months ago

@ovflowd I compared the output of the legacy HTML generator with the current version for a few pages, not sure if some of them need fixing 🙇

Jump links don't exist/work

There are changes in the name on the left menu. For example;

- Assertion testing
+ Assert
- C/C++ addons with Node-API
+ Node-API
- Child processes
+ Child Process

There are differences in the listing in the left menu; hr should be after "Usage and example."
The sequential links on the /errors page do not seem to be generated correctly;

This issue is likely related to the above, but there are also problems with the type references on the following pages:

/url

<a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object) | [`<string>`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#String_type" class="type">&lt;Object&gt;</a>

/os

<a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Number_type) (in MHz" class="type">&lt;number&gt;</a>

This issue also appears on other pages

Except for these minor issues, it looks content-wise 1-1 identical to the current pages ✨

ovflowd commented 3 months ago

There are changes in the name on the left menu. For example;
- Assertion testing
+ Assert
- C/C++ addons with Node-API
+ Node-API
- Child processes
+ Child Process
There are differences in the listing in the left menu; hr should be after "Usage and example."

Have you read the PR description? 😅 -- navigation generation is done automatically from now on, instead from reading the manually defined navigation file that exists on /doc/api :P, so yes, it will be different, and that's intentional.

ovflowd commented 3 months ago

Jump links don't exist/work

What are jump links?

ovflowd commented 3 months ago

The sequential links on the /errors page do not seem to be generated correctly;

This issue is likely related to the above, but there are also problems with the type references on the following pages:
/url
<a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object) | [`<string>`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#String_type" class="type">&lt;Object&gt;</a>
/os
<a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Number_type) (in MHz" class="type">&lt;number&gt;</a> 
This issue also appears on other pages
Except for these minor issues, it looks content-wise 1-1 identical to the current pages ✨

Fixed! Thanks for noticing it!

canerakdas commented 3 months ago

Jump links don't exist/work

What are jump links?

Anchors with a unique id located to the right of the headings;

ovflowd commented 3 months ago

Jump links don't exist/work

What are jump links?

Anchors with a unique id located to the right of the headings;

Oh they were working before 🤔

ovflowd commented 3 months ago

cc @nodejs/collaborators @nodejs/build this PR is finally ready, and all the features from the current legacy API docs template are built-in here.

Please, if possible test/run locally and check if the templates, and API docs look in a way y'all are satisfied with :)

ovflowd commented 3 months ago

cc @nodejs/web-infra for a final code review 🙏

targos commented 3 months ago

More than the code, I'd like to review the output. Is there a preview somewhere or do I have to build it locally?

ovflowd commented 3 months ago

Unfortunately need to build locally. I'm not sure where I could upload a preview here. I recall you had a Netlify account or something? If you build you can simply upload the out/ folder

ovflowd commented 3 months ago

(Im not sure if I can deploy static files to Vercel)

cclauss commented 3 months ago

Do we have a good input file that we can test with?

ovflowd commented 3 months ago

Do we have a good input file that we can test with?

If you want to generate against all docs:

node bin/cli.mjs -i "/path/to/node/doc/api/*.md" -o /tmp/out -t legacy-html-all

ovflowd commented 3 months ago

@targos here's a deploy using Node v22.7.0 as a binary and targeting this commit from nodejs/node for the doc files: https://node-api-docs-legacy-preview.netlify.app/

targos commented 3 months ago

Little inconsistency with the index page's title (above is the new version):

targos commented 3 months ago

View on a single page option is missing (screenshot is from current docs), although the all.html page exists:

targos commented 3 months ago

Every method has a "History" table always

I don't like that, because it seems that the history falls back to "Introduced in: v0.10.0", which is probably wrong in most cases.

ovflowd commented 3 months ago

Every method has a "History" table always

I don't like that, because it seems that the history falls back to "Introduced in: v0.10.0", which is probably wrong in most cases.

Could you give examples?

ovflowd commented 3 months ago

FYI @targos I've made a redeploy with a hot fix regarding the CJS/MJS pre elements.

targos commented 3 months ago

Is it possible to have a light theme for code blocks when the page is in light mode?

Current:

New:

ovflowd commented 3 months ago

Little inconsistency with the index page's title (above is the new version):

Fixed

View on a single page option is missing (screenshot is from current docs), although the all.html page exists:

Fixed

ovflowd commented 3 months ago

Is it possible to have a light theme for code blocks when the page is in light mode?

Current:

New:

I could check on Shiki docs. I believe they have support for dark/light theme modes.

ovflowd commented 3 months ago

Fixes deployed on https://node-api-docs-legacy-preview.netlify.app/

targos commented 3 months ago

Sidebar Navigation is generated on-demand instead of relying on nodejs/node/doc/api/index.md This means some of the entries are a bit different, although I believe they are fine.

I think that's fine (IIUC we can update the page headings to change the navigation text), except: Now "Usage and example" is sorted with the other API docs pages, which doesn't make sense. I guess we're also lucky that "About" is sorted first ^^

targos commented 3 months ago

Consequence of the light mode issue:

ovflowd commented 3 months ago

Sidebar Navigation is generated on-demand instead of relying on nodejs/node/doc/api/index.md This means some of the entries are a bit different, although I believe they are fine.

I think that's fine (IIUC we can update the page headings to change the navigation text), except: Now "Usage and example" is sorted with the other API docs pages, which doesn't make sense. I guess we're also lucky that "About" is sorted first ^^

Correct. Although, the reasoning behind this change is to not depend on the existence of an index.md to source from -- this tooling tries to be agnostic as much as possible.

In the future, with the new generator (redesign of API docs) the sidebar will be grouped by module type (Module, Class, Globals, etc) as it was for a short while on https://nodejs.dev

ovflowd commented 3 months ago

Now "Usage and example" is sorted with the other API docs pages, which doesn't make sense.

Agreed. Although the synopsis page never made sense to me, and even less now with the Node.js Website Learn section. I think that page could be removed IMO.

targos commented 3 months ago

Every method has a "History" table always

I don't like that, because it seems that the history falls back to "Introduced in: v0.10.0", which is probably wrong in most cases.

Could you give examples?

Forget it, it's actually part of the markdown source code. I just happened to click on three pages that had the same version by chance.

ovflowd commented 3 months ago

Is it possible to have a light theme for code blocks when the page is in light mode? Current: New:

I could check on Shiki docs. I believe they have support for dark/light theme modes.

Fixed.

Consequence of the light mode issue:

Fixed.

Re-deployed on same URL with Codebox fixes. (Including some margin between CJS and Copy Button)

nodejs / api-docs-tooling