fix broken links for autogenerated documentation pages

turbomam commented 1 year ago

Background

Prefix/URL definitions in the schema

MIxS asserts it's own prefix as 'MIXS' in terms.yaml and provides the expansion of https://w3id.org/mixs/terms/

prefixes:
  linkml: https://w3id.org/linkml/
  mixs.vocab: https://w3id.org/mixs/vocab/
  MIXS: https://w3id.org/mixs/terms/

w3id mixs namespace

The 'mixs' namespace has been reserved in the w3id system

Options +FollowSymLinks
RewriteEngine on

# vocab/ end point
RewriteRule ^vocab$ https://genomicsstandardsconsortium.github.io/mixs/$1
RewriteRule ^vocab\/(.*)$ https://genomicsstandardsconsortium.github.io/mixs/$1 [R=301,L]

---snip---

# Schema elements use Github Pages
# Rewrite Base URL
RewriteRule ^(.*)$ https://genomicsstandardsconsortium.github.io/mixs/$1 [R=302,L]

The base of the RewriteRule, https://genomicsstandardsconsortium.github.io/mixs, does not agree with the 'terms' part of the expanded default prefix, https://w3id.org/mixs/terms/
There's a case mismatch between the schema's default prefix (MIXS) and the w3id namespace ('mixs')
We often refer to this resource as MIxS. Is that codified anywhere?

Documentation build process overview:

Makefile rule generated/docs/index.md uses LinkML gen-doc to convert the schema into Markdown pages. Those are saved locally in the generated/docs path. They are not pushed to GitHub, so you won't see them in the web interface.
generated/docs/introduction/%.md is invoked when generated/docs/introduction/background.md is requested. It copies some static Markdown files from static_md/ into generated/docs/
mkdocs_html/index.html uses mkdocs to convert the Markdown files to HTML. Those aren't pushed either.
gh_docs deploys the HTML files to GitHub pages.

Syntax of schema documentation web site URLs

The path to the MIxS GitHub repo is https://github.com/GenomicsStandardsConsortium/mixs. The path to the GitHub pages site is https://genomicsstandardsconsortium.github.io/mixs/, as far as GitHub is concerned. GitHub isn't aware that we are using w3id redirection to map http://w3id.org/mixs to https://genomicsstandardsconsortium.github.io/mixs/.

When we mention our schema documentation website to the public, we should always use the shorter http://w3id.org/mixs URL.

mkdocs.yml asserts that the base URL of each autogenerated documentation pages should be the GitHub pages path mentioned above, https://genomicsstandardsconsortium.github.io/mixs/

Numerical term identifiers as used as the right-hand portion of each term page's URLs. Numerical identifiers haven't been assigned to the EnvironmentalPackages, Checklists and CombinationClasses in the current release, so those pages end in the textual name of those elements.

Numerical identifiers have been assigned to EnvironmentalPackages, Checklists and CombinationClasses in the schemasheets branch, but that branch still require work before it can become the next release.

Manually maintained URL assertions in the GitHub repo

The About section in the upper right of the MIxS repo's home page recommends this documentation link: https://genomicsstandardsconsortium.github.io/mixs/. That should probably be replaced with a w3id-based link.

See also MIxS issues labeled w3id and documentation:

turbomam commented 1 year ago

actions:

change the schema's MIXS prefix expansion to
- MIXS: https://w3id.org/mixs/
change the default_prefix in all of the schema's YAML files to 'MIXS'
change the link in the repo's About section to https://w3id.org/mixs
disable the schema documentation web site's search feature
- the search index file is too large to be committed to the GitHub repo, so the publication process never completes

turbomam commented 1 year ago

The homepage for the MIxS autogenerated documentation can be reached through https://w3id.org/mixs or https://genomicsstandardsconsortium.github.io/mixs/

One can return to the home page by clicking the "person reading a book" button in the upper left, or the Reference link on most or all other pages

examples of reported broken links:

https://genomicsstandardsconsortium.github.io/mixs/Soil.md (from https://genomicsstandardsconsortium.github.io/mixs/#packages)
https://genomicsstandardsconsortium.github.io/mixs/soil.md does not work either
https://genomicsstandardsconsortium.github.io/mixs/terms/0000018 (via the URI section of https://genomicsstandardsconsortium.github.io/mixs/0000018/)
https://genomicsstandardsconsortium.github.io/mixs/terms/ChecklistClass, from @JKoehorst in #518
- 'terms' shouldn't be a part of the URL, but https://genomicsstandardsconsortium.github.io/mixs/ChecklistClass doesn't work now either
https://w3id.org/mixs/terms/0000050 from @cpauvert in #390
- ok without 'terms': https://w3id.org/mixs/0000050
- @cpauvert also mentioned or https://w3id.org/mixs/vocab/sequencing_field but that does work now
- @ramonawalls has suggested that 'gensc' was supposed to be part of the URLs, but it don't think that's ever true

turbomam commented 1 year ago

related problems:

building the documentation on a Mac gives rise to case conflicts
that's not an issue when building the documentation on a different platform, like @turbomam' Ubuntu workstation
the case conflict shouldn't be a problem when building in a GitHub action, but that approach fails because a search index file is too large to include in a GH repo
have to wait a few seconds for "Initializing search"... but under what circumstances?
search results aren't ranked very well and are influenced by package-specific slot usage. When searching for "depth", the depth term doesn't appear at all, although soil_depth does appear at the top.

turbomam commented 1 year ago

Might be able to solve the over-sized search index file by reading https://squidfunk.github.io/mkdocs-material/setup/setting-up-site-search/

turbomam commented 1 year ago

I have eliminated the use of the mixs.vocab prefix

If it's really important to restore it, I would ask someone else to

reflect how long it took to get this single namespace working
reread the Background section of this issue
document the conceptual differences between the MIXS and mixs.vocab namespaces in an issue
document which schema elements should use which prefix (either exhaustively, or by superclass) in an issue
in a branch, add the necessary prefix definitions, prefix assignments and redirection rules to the schema files
if the w3id redirection rules need to be modified, create an issue about that and practice with https://htaccess.madewithlove.com/
after approval from GSC/MIxS stakeholders, make edits to https://github.com/perma-id/w3id.org/blob/master/mixs/.htaccess. It can take several hours before the w3id team merges your PR.

turbomam commented 1 year ago

Note that I made edits directly to the YAML files in model/schema, so they would be overwritten if anybody ran gsctools/mixs_converter.py again. I presume it would be possible to edit that script to generate YAML files like those included in this PR, but I think finishing the scheasheets implementation is better investment, That will allow for declaring the prefixes and their expansions outside code.

turbomam commented 1 year ago

Note: some of the broken example URLs above included the .md extension. Did that ever work? It doesn't now, and that makes sense because .md files aren't/never were being served.

https://genomicsstandardsconsortium.github.io/mixs/Soil/ is the correct path to learn about the MIxS Soil package/extension. When a user requests that page, the server knows to return the index.html page within that path, i.e. https://genomicsstandardsconsortium.github.io/mixs/Soil/index.html

turbomam commented 1 year ago

@JKoehorst's

518

accurately points out that https://genomicsstandardsconsortium.github.io/mixs/terms/ChecklistClass does not resolve.

That's because there is no ChecklistClass in the release model

ChecklistClass is defined in the schemasheets branch but that's going to require some work before it can replace the current release. I can provide assistance with the technical implementation, but I don't think I should be making the judgement calls regarding things like

terms with multiple different ids
ids that point to multiple different classes
terms for which other basic definitional metadata vary between packages
the lack of valid and invalid data for checking the schema

ramonawalls commented 1 year ago

@turbomam Thank you so much for providing such extensive background and explanations of this issue. I am very grateful.

I would like to suggest (and work on, with your help) a few changes regarding the namespace.

@ramonawalls has suggested that 'gensc' was supposed to be part of the URLs, but it don't think that's ever true

The decision to include "gensc" in the namespace was decided over several discussions, and documented in issue #233. The reason to include it is that GSC has other projects besides MIxS (e.g., genomic observatories). Even though those projects aren't to the point of us publishing documentation for them yet, we want to allow for the possibility. The plan was for the MIxS namespace to be https://w3id.org/gensc/mixs/. I think that should be easy enough to do through w3id.

'terms' shouldn't be a part of the URL

Per issue #233, we had decided that all GSC terms, whether part of MIxS or not, would be under the same namespace. I am open to reconsidering this choice if it causes problems with using LinkML. We have no imminent plans to release any terms outside MIxS, so it should be fine to just put all mixs terms under the https://w3id.org/gensc/mixs/ namespace.

The homepage for the MIxS autogenerated documentation can be reached through https://w3id.org/mixs or https://genomicsstandardsconsortium.github.io/mixs/

This brings up another issue, which is the need to change our github organization name. GSC is spelled wrong (it should be "genomic" not "genomics". I have been unable to change it in the past, but I think I can get Volker to make the change. While we are changing it, I would like to suggest that we change the org name to "gensc" to be consistent with our website and easier to type. I will create a separate issue.

turbomam commented 1 year ago

OK,I apologize for jumping to conclusions.

I can help with most of these tasks if technical requirements are written out.

@cmungall do prefer for these additional changes to be made before of after your upcoming presentations?

turbomam commented 1 year ago

@ramonawalls' requests above are all doable, but have to be implemented in at least three places, all in coordination (and preferably with an understanding of the timing)

Org name

the name of the organization that "owns" the MIxS repo is GenomicsStandardsConsortium
what should the new GitHub org name be?
- here are the org owners who have permission to change the name:
- @cmungall
- @folker
- @genomicstandardsconsortium
- @lschriml
- @only1chunts
- @pyilmaz
- @ramonawalls
do we understand the potential costs of changing the org name?
- https://docs.github.com/en/organizations/managing-organization-settings/renaming-an-organization

w3id prefix and redirection rules

gensc is already registered, so we own the path https:/w3id.org/gensc, but that currently gives a "Internal Server Error" with Chrome on Ubuntu 22. Is it supposed to redirect to https://gensc.org/? The madewithlove htaccess tester says that it is being rewritten as https://gensc.org/w3id.org/gensc
the path https:/w3id.org/gensc/mixs/0000010, which presumably should redirect to to the geo_loc_name instead redirects to https://gensc.org/w3id.org/gensc/mixs/0000010
here are the current redirection rules:

Options +FollowSymLinks
RewriteEngine on

<!-- gensc/mixs ednpoint route -->
RewriteRule ^mixs$ https://genomicsstandardsconsortium.github.io/mixs/$1 [R=302,L]

<!-- TODO: gensc/terms endpoint -->

<!-- Rewrite Base URL -->
RewriteRule ^(.*)$ https://gensc.org/$1 [R=302,L]

I'm not great with htaccess, but I recommend these instead:

Options +FollowSymLinks
RewriteEngine on

RewriteRule ^gensc/mixs/(.*)$ https://genomicsstandardsconsortium.github.io/mixs/$1/ [R=301,L]
RewriteRule ^gensc(.*)$ https://gensc.org/$1 [R=301,L]

LinkML prefix expansion

If we decided to go with the rewrite rules above, then the MIXS prefix should be expanded in LinkML as

prefixes:
  MIXS: https://w3id.org/gensc/mixs/terms/

genomicstandardsconsortium commented 1 year ago

I have updated the owner/manager roles. Lynn

From: Mark A. Miller @.> Sent: Monday, February 13, 2023 10:35 AM To: GenomicsStandardsConsortium/mixs @.> Cc: genomicstandardsconsortium @.>; Mention @.> Subject: Re: [GenomicsStandardsConsortium/mixs] fix broken links for autogenerated documentation pages (Issue #530)

Org name