search

mdn / content

The content behind MDN Web Docs
https://developer.mozilla.org
Other
9.08k stars 22.45k forks source link

(Preparing to) move NSS docs off MDN #2669

Closed chrisdavidmills closed 2 years ago

chrisdavidmills commented 3 years ago

We need to help the Mozilla NSS team move their documentation off MDN. To be clear, we are talking about everything underneath https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS.

The goal is to get to the stage where all of the documentation under /NSS is available as static, standalone HTML files that are free from KumaScript macros, have links fixed up as much as is possible, and can be used in the new destination (which we are hoping will be the Firefox source docs Sphinx instance)

The tasks required are:

    • [x] Fix all the flaws generated on the NSS docs (this is a list of errors found for each document, mainly broken links).
    • [x] Most of these are automatically fixable, so we should run the commands to do this.
    • [ ] Scrape all of the docs under the /NSS directory from the MDN site, and get rid of all the MDN-specific scaffolding to just leave a minimal standalone HTML document for each one (this includes removing any navigation menus). Present them in a directory structure the same as the existing directory structure in the content repo.
    • [ ] Come up with a list of manual fixes that are needed for each document. Make these fixes for each document to get them ready for use in the new system (I was thinking that a team comprising of me, another MDN writer, and Niklas from the NSS team, could do this work)
    • [ ] Once the content is prepared, the NSS team would need to do final sorting of the links inside the resource:
    • Change all internal links that link to other NSS docs like this — /en-US/docs/Mozilla/Projects/NSS/xxx > /new/relative/path/xxx
    • Change all internal links that link to other docs on MDN not inside the NSS section like so — /en-US/docs/Mozilla/xxx > https://developer.mozilla.org/en-US/docs/Mozilla/xxx
    • [ ] On the MDN side, our engineers would need to put hard redirects in place from the old locations of the pages on MDN, to the new locations of the pages.
hamishwillee commented 3 years ago

@chrisdavidmills Decided to be helpful on point 2 - bunch of links fixed on #2960. A good task to cool my brain down on.

There are still quite a few broken links showing up. Specifically everything to the project NSPR and also a few archived docs like https://developer.mozilla.org/en-US/docs/Archive/Security/Introduction_to_SSL

FYI, not an automatic task :-(

chrisdavidmills commented 3 years ago

@chrisdavidmills Decided to be helpful on point 2 - bunch of links fixed on #2960. A good task to cool my brain down on.

There are still quite a few broken links showing up. Specifically everything to the project NSPR and also a few archived docs like https://developer.mozilla.org/en-US/docs/Archive/Security/Introduction_to_SSL

FYI, not an automatic task :-(

Yeah, unfortunately not, by the looks of things. I'm assuming in doing this work you are just working through the folder structure alphabetically? Do you think you can create a simple list in here that shows which directories have had their broken URLs fixed so far, and by what PR? Just so others that might work on it can easily see at a glance what else there is to be done?

hamishwillee commented 3 years ago

Possibly, but if you can tell me how to fix the NSPR links and confirm that we can ignore the archive links that would be a smaller task. This pretty much fixes everything else that can be fixed due to links being "accidentally broken"

hamishwillee commented 3 years ago

@chrisdavidmills OK, so I did a bunch more fixes in #3134. Below is a list of what remains to be fixed. Not completely exhaustive, but enough to get a good picture on high level things that remain - a bunch of them could be bulk fixed if we knew the right targets.

Note also that many of these are actually archived and just showing up as broken link. See https://github.com/mdn/yari/issues/2675. This applies to much of the "/en-US/docs/Mozilla/Projects/NSPR" links

Broken Links /en-US/docs/Mozilla/Projects/NSS/Reference/NSS_cryptographic_module/Data_types 👀 line 20:15 /en-US/docs/Mozilla/Projects/NSS/Reference/NSS_cryptographic_module/Non-FIPS_mode_of_operation 👀 line 21:15

/en-US/docs/Using_CXX_in_Mozilla_code line 121:106

/en-US/docs/Mozilla/Developer_guide/Build_Instructions/CVS_Tags line 106:198 INVALID (archived): /en-US/docs/Archive/Using_SSH_to_connect_to_CVS line 19:502 INVALID (archived): /en-US/docs/Archive/Mozilla/Creating_a_Mozilla_extension/Tinderbox line 75:76

INVALID (archived): /en-US/docs/Archive/Security/Introduction_to_Public-Key_Cryptography line 30:19 INVALID (archived): /en-US/docs/Archive/Security/Introduction_to_SSL line 32:19 INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR/Reference line 122:19 INVALID (archived): /en-US/docs/Archive/Mozilla/JavaScript_crypto line 129:19

INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR line 28:57 /en-US/docs/Mozilla/Projects/NSPR INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR line 28:326 INVALID (archived): /en-US/docs/Mozilla/Security/x509_Certificates line 47:15

INVALID (archived): /en-US/docs/Mozilla/Security/x509_Certificates line 46:415 INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR line 72:1481

INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR line 14:338

INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR 👀 line 25:272 INVALID (archived): /en-US/docs/Archive/Security/Introduction_to_Public-Key_Cryptography line 141:15 INVALID (archived): /en-US/docs/Archive/Security/Introduction_to_SSL line 142:15 INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR/Reference line 147:109

Lots broken under the /Projects/NSS/Reference - just subset listed

The point here to see that many of these are common

/en-US/NSC_CancelFunction line 27:19

/en-US/NSC_CloseAllSessions line 27:109

/en-US/NSC_DigestInit line 28:19

/en-US/NSC_DigestFinal line 30:105

/en-US/NSC_DigestInit line 28:19

/en-US/NSC_SetAttributeValue 👀 line 32:19

/en-US/NSS/CERTIssuerAndSN 👀 line 19:260

No flaws reported:

chrisdavidmills commented 3 years ago

Thanks @hamishwillee , that's super helpful.

I will see if I can find some time to do a few of these. Also feel free to to keep picking bits off ;-)

hamishwillee commented 3 years ago

Great. Will do. FYI, I found the NSPR link - it exists, but is archived: https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSPR

Quite a few of these might also be fixed by creating one or two little docs - e.g. aCK_SESSION_HANDLE is clearly a copy of https://www.cryptsoft.com/pkcs11doc/v100/pkcs11__all_8h.html#aCK_SESSION_HANDLE

hamishwillee commented 3 years ago

Yet more done in #3290.

This is what is left. I think that I'm at the end of what I can do

Broken Links /en-US/docs/Mozilla/Projects/NSS/Reference/NSS_cryptographic_module/Data_types 👀 line 20:15 /en-US/docs/Using_CXX_in_Mozilla_code line 121:106

/en-US/docs/Mozilla/Developer_guide/Build_Instructions/CVS_Tags line 106:198 INVALID (archived): /en-US/docs/Archive/Using_SSH_to_connect_to_CVS line 19:502 INVALID (archived): /en-US/docs/Archive/Mozilla/Creating_a_Mozilla_extension/Tinderbox line 75:76

INVALID (archived): /en-US/docs/Archive/Security/Introduction_to_Public-Key_Cryptography line 30:19 INVALID (archived): /en-US/docs/Archive/Security/Introduction_to_SSL line 32:19 INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR/Reference line 122:19 INVALID (archived): /en-US/docs/Archive/Mozilla/JavaScript_crypto line 129:19

/en-US/NSC_CancelFunction line 27:19

/en-US/NSC_CloseAllSessions line 27:109

/en-US/NSC_DigestInit line 28:19

/en-US/NSC_DigestFinal line 30:105

/en-US/NSC_DigestInit line 28:19

/en-US/NSC_SetAttributeValue 👀 line 32:19

/en-US/NSS/CERTIssuerAndSN line 19:260

chrisdavidmills commented 3 years ago

@escattone a friendly ping on this one Ryan. @hamishwillee has done a whole bunch of work to fix broken links in these docs, so the next stage is to scrape the rendered NSS docs off the site into a "pure" HTML form that the NSS team can use.

Also pinging @beurdouche for an update as to the latest progress on this.

beurdouche commented 3 years ago

Thanks a lot, this is very helpful! :)

Regarding the links, if you can point the Non-FIPS mode to the page you suggested, it would be great https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/PKCS11_Functions

I think that there is no need to worry about the rest of the links. This list is useful and I can fix/delete/rewrite those after we move off of MDN 👍

hamishwillee commented 3 years ago

Hi @beurdouche , the link you refer to will be fixed by https://github.com/mdn/content/pull/3513.

I am not promising on my life that all links are fixed - but certainly the main docs have all been gone through and any flaws shown by the automatic tooling have been fixed up. The ones marked as Invalid above show up as broken if you build locally but work on the main site (i.e. the scaped version will work)

chrisdavidmills commented 3 years ago

Hi @beurdouche ! It looks like we've finished doing link fixes on this content, so it is ready to scrape and get into a rendered out HTML format for you folks to use.

In which case, the next stage is to create a scraper to scrape the content off of MDN. We will probably create a simple script to do this, and then hand it over to you folks so you can do the conversion and tweak it as needed.

My question here is — what do you want the HTML to look like, exactly? Do you need a full page for each, including doctype, simple document head, plus body, or do you just need the body content for each? Or something else?

dergoegge commented 3 years ago

Hi @chrisdavidmills, thanks for all the work so far! I can answer this for @beurdouche.

Just the content would be the best for us. We basically just need the files the way they are on github right now but with the macros expanded. (Easiest example would be index.html)

chrisdavidmills commented 3 years ago

Hi @chrisdavidmills, thanks for all the work so far! I can answer this for @beurdouche.

Just the content would be the best for us. We basically just need the files the way they are on github right now but with the macros expanded. (Easiest example would be index.html)

Do you want them with the YAML front matter included?

dergoegge commented 3 years ago

Good question. Currently the conversion script uses the YAML to get the title but I am guessing that the scraped HTML will have the titles in it? If that is the case then we don't need the YAML.

chrisdavidmills commented 3 years ago

Good question. Currently the conversion script uses the YAML to get the title but I am guessing that the scraped HTML will have the titles in it? If that is the case then we don't need the YAML.

That's correct — the scraped page would include the <h1>, as wll as all the KS macros being rendered out.

So assuming that you don't have any usage for the tags, slug, etc, then that sounds ok.

My esteemed colleague @escattone and his team would be helping with the actual scraping code here, so I'll pass this over to him. Ryan, the one thought I still have here is that we'd probably need to scrape them and provide the scraped content in a folder structure that matches the existing folder structure in the source, to help @dergoegge put them in the same kind of structure that they are currently on MDN, in their new home. This would in turn help us to put redirects to the new locations in place much more easily. Sound reasonable?

I was thinking you could start by scraping a few docs, and sharing them with Niklas to check if tey are ok for his purposes.

escattone commented 3 years ago

@chrisdavidmills All sounds reasonable! Let me get cracking on the scraper this week, share an example with @dergoegge, and we can iterate from there.

chrisdavidmills commented 3 years ago

@escattone thanks, much appreciated.

escattone commented 3 years ago

@dergoegge @beurdouche I created an initial set of "distilled" NSS docs in https://github.com/mdn/nss-docs. By "distilled", I mean I extracted <main> and <title> from the MDN version of each doc, and re-packaged in a fresh, minimal <html> package (so basically stripped out the MDN header, footer, and sidebar). Does that work for you all? I'm happy to adjust as you wish. Cheers.

dergoegge commented 3 years ago

@escattone Could you change it so the files only contain the inner html of <article class="main-page-content"> on each page? Besides that it looks good to me so far👍

escattone commented 3 years ago

@dergoegge I've updated https://github.com/mdn/nss-docs to reflect your requested change. Hope that works for you.

mathewhodson commented 3 years ago

It looks looks like the work to add the docs to the source tree is at https://bugzilla.mozilla.org/show_bug.cgi?id=1709817

escattone commented 3 years ago

Thanks for the update @mathewhodson.

hamishwillee commented 3 years ago

The NSS files have been deleted from MDN. Any reason we can't close this? Speak now, or I'll close next time I come around.

I appreciate there may be more "work" being done to publish by the NSS team, but that is out of scope for the people working on content,.

escattone commented 3 years ago

@hamishwillee 👍 I agree. I think we can close this.

hamishwillee commented 2 years ago

@Rumyra @schalkneethling I am reopening because this has still not been removed and Ryan has left the building.

The plan here was to delete the NSS tree, which is broken and which the NSS team were taking over. We did so in https://github.com/mdn/content/pull/8215 but it was reverted because they were not ready. Ryan was liasing to find a time but not as far as I could see actually pushing.

I would like to propose deletion of this tree within the month "come what may" as they've have more than enough time - we can put the source in archive if desired. Is there someone we can notify in that team? If so, who?

EDIT: Note, closing. Seems this came up 4 days ago and is being addressed in #12471