Closed eecavanna closed 3 months ago
turbomam
commented
9 months ago Thanks!
docs/ is git-ignored, but you can crate it locally with make squeaky-clean all test gendoc
you can view the locally generated documentation with make serve. serve and gendoc are included in testdoc
eecavanna
commented
9 months ago Here's the npm package I plan to use to do a one-time check:
https://www.npmjs.com/package/markdown-link-check
Note: This is not lychee. It's a different tool I just came across.
eecavanna
commented
9 months ago Here are all the broken links found via a quick and dirty check with that tool (will do the check more carefully another time):
Here's the command I ran on my local machine, from the docs directory, in order to generate that list (which I think added some notes to above):
docker run -v ${PWD}:/tmp:ro --rm -i ghcr.io/tcort/markdown-link-check:stable /tmp/index.mdWhile running $ make squeaky-clean all test gendoc on my local machine (at commit d1593296fbb0e296e86fa4f8c8d3be7fe63b2b01, which is the tip of the main branch and is the commit tagged as v9.2.0), I got the following error message:
# ...
ALL_SCHEMAS = ['src/schema/nmdc.yaml']
INFO:root:Skipping graphql as not in inclusion list: ['jsonldcontext', 'jsonschema', 'owl', 'python', 'rdf']
INFO:root:Generating: jsonldcontext
INFO:root: SCHEMA: src/schema/nmdc.yaml
INFO:root: PARENT=project/jsonld
Traceback (most recent call last):
File "/root/.cache/pypoetry/virtualenvs/nmdc-schema-VsnhxLU2-py3.9/bin/gen-project", line 8, in <module>
sys.exit(cli())
File "/root/.cache/pypoetry/virtualenvs/nmdc-schema-VsnhxLU2-py3.9/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/root/.cache/pypoetry/virtualenvs/nmdc-schema-VsnhxLU2-py3.9/lib/python3.9/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/root/.cache/pypoetry/virtualenvs/nmdc-schema-VsnhxLU2-py3.9/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/root/.cache/pypoetry/virtualenvs/nmdc-schema-VsnhxLU2-py3.9/lib/python3.9/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/root/.cache/pypoetry/virtualenvs/nmdc-schema-VsnhxLU2-py3.9/lib/python3.9/site-packages/linkml/generators/projectgen.py", line 253, in cli
gen.generate(yamlfile, project_config)
File "/root/.cache/pypoetry/virtualenvs/nmdc-schema-VsnhxLU2-py3.9/lib/python3.9/site-packages/linkml/generators/projectgen.py", line 144, in generate
gen = gen_cls(local_path, **all_gen_args)
File "<string>", line 31, in __init__
File "/root/.cache/pypoetry/virtualenvs/nmdc-schema-VsnhxLU2-py3.9/lib/python3.9/site-packages/linkml/generators/jsonldcontextgen.py", line 54, in __post_init__
super().__post_init__()
File "/root/.cache/pypoetry/virtualenvs/nmdc-schema-VsnhxLU2-py3.9/lib/python3.9/site-packages/linkml/utils/generator.py", line 198, in __post_init__
self._initialize_using_schemaloader(schema)
File "/root/.cache/pypoetry/virtualenvs/nmdc-schema-VsnhxLU2-py3.9/lib/python3.9/site-packages/linkml/utils/generator.py", line 243, in _initialize_using_schemaloader
loader.resolve()
File "/root/.cache/pypoetry/virtualenvs/nmdc-schema-VsnhxLU2-py3.9/lib/python3.9/site-packages/linkml/utils/schemaloader.py", line 220, in resolve
self.raise_value_error(
File "/root/.cache/pypoetry/virtualenvs/nmdc-schema-VsnhxLU2-py3.9/lib/python3.9/site-packages/linkml/utils/schemaloader.py", line 995, in raise_value_error
SchemaLoader.raise_value_errors(error, loc_str)
File "/root/.cache/pypoetry/virtualenvs/nmdc-schema-VsnhxLU2-py3.9/lib/python3.9/site-packages/linkml/utils/schemaloader.py", line 1005, in raise_value_errors
raise ValueError(f'{TypedNode.yaml_loc(loc_str, suffix="")} {error}')
ValueError: File "nmdc.yaml", line 435, col 9 Class "Biosample" - unknown slot: "host_disease_stat"
make: *** [Makefile:102: gen-project] Error 1I then ran $ poetry install to update the project dependencies, then I re-ran the above $ make ... command. I got the same error message.
eecavanna
commented
9 months ago In contrast, I was able to run $ make squeaky-clean all test gendoc on commit a4896ca44e35a9b40dfe3ae3b6c39fe3c1bae614 (which is about 10 days old). I then used lychee (command shown below) to check the links in the generated docs folder.
Lychee command (run from root folder of repository):
$ docker run --init -it --rm -w /input -v $(pwd):/input lycheeverse/lychee /input/docsHere's a summary of the test results (as of that commit):
9821 Total ✅ 8574 OK 🚫 1244 Errors (HTTP:1242|Timeouts:2) 💤 1 ExcludedThe detailed results are below.
eecavanna
commented
9 months ago Hi @turbomam, does $ make squeaky-clean all test gendoc work for you when you run it on the tip of the main branch (i.e. commit d1593296fbb0e296e86fa4f8c8d3be7fe63b2b01)?
It works for me when I run it on commit https://github.com/microbiomedata/nmdc-schema/commit/a4896ca44e35a9b40dfe3ae3b6c39fe3c1bae614 (a commit from 10 days ago), but not on the tip of the main branch. When I run it on the latter, I get this error (full stack trace is in this comment from earlier today):
ValueError: File "nmdc.yaml", line 435, col 9 Class "Biosample" - unknown slot: "host_disease_stat"In the meantime, I ran Lychee on the commit from 10 days ago and posted the results in my previous comment. If that report is sufficient, I'll close this ticket and then open one about doing this automatically via GitHub Actions.
eecavanna
commented
9 months ago Note: @turbomam suggested via Slack that I add mixs-yaml-clean as the first target of the make command. I did that and, as a result, the make command ran to completion, generating docs based upon the tip of the main branch.
With the newly-generated docs, here is a summary of the link check results:
🔍 9838 Total ✅ 8572 OK 🚫 1244 Errors (HTTP:1223|Timeouts:21) 💤 1 ExcludedThe numbers are approximately the same as before (i.e. above). I'll leave the large, detailed report out of this comment. Interested parties can generate it themselves by running the following commands in the root folder of the repo:
# Generate docs.
make mixs-yaml-clean squeaky-clean all test gendoc
# Perform link check on generated docs.
docker run --init -it --rm -w /docs -v $(pwd)/docs:/docs lycheeverse/lychee /docsHere's a link to a JSON-formatted version of the full report, which I generated by running:
# Note: I map two volumes so I can (a) store the output file in the repo root,
# while also being able to (b) keep the paths in the lychee output concise.
docker run --init -it --rm -w /docs \
-v $(pwd):/src \
-v $(pwd)/docs:/docs \
lycheeverse/lychee --format json --output /src/lychee.report.json /docsJSON-formatted report: lychee.report.json
I ran the contents of that JSON-formatted report through jq and sort -u (using the command shown below) to get a list of the unique broken hyperlinks.
$ cat lychee.report.json | ~/Downloads/jq-macos-arm64 -r '.fail_map | .[] | .[].url' | sort -u@turbomam, here's that list of the unique broken hyperlinks: lychee.unique-urls.txt
Note: The contents of that text file used to be displayed in this comment. I have since removed that displayed list in order to de-clutter the GitHub Issue, given that there is another long list of URLs in a subsequent comment. The attached file contains the exact list of URLs that used to be in this comment (the file was created by copy/pasting the list from this comment into an empty file).
turbomam
commented
9 months ago Thanks @eecavanna. @sujaypatil96 and I just reviewed this. We think that the link checker is giving some false positives. For the trues positives there are probably several failure points that we'll look into.
turbomam
commented
9 months ago can we please configure those redirected links to be considered valid?
turbomam
commented
9 months ago https://gensc.org/mixs is indeed 404. It was introduced into src/docs/Metadata_Documentation_Overview.md by Bill 2 years ago. @sujaypatil96 or I could ask what the GSC considers an appropriate replacement URL. I assume https://genomicsstandardsconsortium.github.io/mixs/#extensions
While we're on the subject
Extensions
Learn more about all of the [<u>17 MIxS environmental packages</u>](https://gensc.org/mixs)
We need to revise this file!
@cmungall @mslarae13 you may or may not be interested in this whole thread, but I would appreciate your thoughts about overhauling src/docs/Metadata_Documentation_Overview.md
eecavanna
commented
9 months ago
- http://purl.obolibrary.org/obo/CHMO_0002773 resolves to https://ontobee.org/ontology/CHMO?iri=http://purl.obolibrary.org/obo/CHMO_0002773
- https://doi.org/10.1007/978-1-61779-986-0_28 resolves to https://link.springer.com/protocol/10.1007/978-1-61779-986-0_28
can we please configure those redirected links to be considered valid?
One thing I noticed about those 2 URLs, specifically, is that they each involve multiple redirects (2 redirects/3 URLs for one; and 3 redirects/4 URLs for the other). The lychee documentation says the default number of redirects it allows is 5. I didn't override that when I ran it. Since 2 =< 5 and 3 =< 5, I don't expect that to be the root cause of the false positive, but it is something we can pay closer attention to/experiment with on subsequent runs. The relevant lychee CLI option is --max-redirects 5. If it comes down to it, we can exclude those specific URLs from the check using the --exclude ... CLI option (or another one of the "URL exclusion" options lychee provides).
I checked the raw error log above (from a little bit older commit) and saw the following:
✗ [ERR] https://doi.org/10.1007/978-1-61779-986-0_28 | Failed: Too many redirects: error following redirect for url (https://idp.springer.com/transit?redirect_uri=https%3A%2F%2Flink.springer.com%2Fprotocol%2F10.1007%2F978-1-61779-986-0_28&code=87ae14e5-ba52-4f33-ba8e-99d4bb3eea39): too many redirectsSo, the number of redirects it encountered when checking that URL was too high for it at the time. I wonder whether either (a) the actual number of redirects being applied to that URL on the Internet changed between then and now; or (b) Lychee didn't actually use 5 as its default value of "allowed" redirects, contrary to its documentation. Either way, we can explicitly include the --max-redirects 5 (or some other quantity) next time we do a check.
mslarae13
commented
9 months ago @turbomam
One quick note on your comment above
The count is now 23, not 17
Make sure, I think it's 17 extensions provided by NMDC. We don't provide all 23.
mslarae13
commented
9 months ago I do agree on a overhaul / review of the documentation
eecavanna
commented
9 months ago Hi @turbomam,
I checked out the latest commit on the main branch (9afca37b1a946438254e4c44a1f4ddca3593c5c3), re-generated the docs, and re-ran lychee on them to check the links. This time, I explicitly included the --max-redirects 5 CLI option.
make mixs-yaml-clean squeaky-clean all test gendoc# Note: I put `9afca3` in the name of the output file so I could correlate it with the Git commit.
docker run --init -it --rm -w /docs \
-v $(pwd):/src \
-v $(pwd)/docs:/docs \
lycheeverse/lychee --max-redirects 5 --format json --output /src/lychee.9afca3.report.json /docsNote: I map the
/docsfolder as its own volume (even though the container would have been able to access it via the other volume) so I can keep the file paths in the report more concise.
Here's the raw JSON file: lychee.9afca3.report.json
cat lychee.9afca3.report.json | ~/Downloads/jq-macos-arm64 -r '.fail_map | .[] | .[].url' | sort -u > lychee.9afca3.unique-urls.txtHere's the resulting text file (57 lines of URLs + 1 blank line): lychee.9afca3.unique-urls.txt
Here's the contents of that text file (for quick reference):
file:///images/NMDC_metadata_img1.png
file:///images/NMDC_metadata_img2.png
file:///images/NMDC_metadata_img3.png
file:///images/NMDC_metadata_img4.png
file:///images/NMDC_metadata_img5.png
file:///images/NMDC_metadata_img6.png
http://3.236.215.220:7200/sparql
http://dl.ppdm.org/dl/690
http://himedialabs.com/TD/PT158.pdf
http://identifiers.org/chembl.compound/
http://insdc.org/country.html
http://petrowiki/
http://purl.obolibrary.org/obo/ENVO
http://purl.obolibrary.org/obo/EO_0007147
http://purl.obolibrary.org/obo/FBcv_0003024
http://purl.org/dc/terms/description
http://purl.org/dc/terms/isPartOf
http://rs/
http://www.cropontology.org/ontology/CO_715/Crop%20Research
http://www.fao.org/nr/land/sols/soil/wrb-soil-maps/reference-groups
http://www.geosamples.org/getigsn
https://acsess.onlinelibrary.wiley.com/doi/full/10.2136/sssaj2009.0389?casa_token=bm0pYIUdNMgAAAAA%3AOWVRR0STHaOe-afTcTdxn5m1hM8n2ltM0wY-b1iYpYdD9dhwppk5j3LvC2IO5yhOIvyLVeQz4NZRCZo
https://docs.google.com/document/d/1oNlMNQySuCoEeqhf1Qou8D-BV5bE76TkjrJLya8Ehw4/edit
https://docs.google.com/spreadsheets/d/1INlBo5eoqn2efn4H2P2i8rwRBtnbDVTqXrochJEAPko/edit#gid=0
https://doi.org/10.1007/978-1-61779-986-0_28
https://doi.org/10.1007/BF01056090
https://doi.org/10.1007/s00216-006-0322-8
https://doi.org/10.1080/07352680902776556
https://doi.org/10.1111/j.1654-109X.2011.01154.x
https://doi.org/10.2134/agronmonogr9.2.c32
https://doi.org/10.2136/sssabookser5.3.c16
https://en/
https://gensc.org/mixs
https://github.com/microbiomedata/nmdc-metadata
https://github.com/microbiomedata/nmdc-metadata/issues/184
https://github.com/microbiomedata/nmdc-metadata/issues/195
https://github.com/microbiomedata/nmdc-metadata/pull/297
https://graphdb-dev.microbiomedata.org/
https://link.springer.com/content/pdf/10.1007/BF02796489.pdf
https://microbiomedata.github.io/nmdc-metadata/#classes
https://microbiomedata.github.io/nmdc-metadata/docs/GeneProduct
https://microbiomedata.github.io/nmdc-metadata/docs/OmicsProcessing
https://microbiomedata.github.io/nmdc-metadata/docs/OrthologyGroup
https://mobio.com/media/wysiwyg/pdfs/protocols/12888.pdf
https://w3id.org/linkml/DateOrDatetime
https://w3id.org/linkml/base
https://w3id.org/linkml/pattern
https://w3id.org/linkml/repr
https://w3id.org/linkml/typeof
https://w3id.org/linkml/uri
https://w3id.org/mixs/0000341
https://w3id.org/mixs/investigation_type
https://w3id.org/nmdc/activity_set
https://w3id.org/nmdc/instrument_name
https://w3id.org/nmdc/nmdc
https://w3id.org/nmdc/nucleic_acid_concentration
https://www.sciencedirect.com/science/article/abs/pii/0038071787900526
ssarrafan
commented
8 months ago Appears to be active. Moving to next sprint. @eecavanna let me know if this should be in the backlog.
eecavanna
commented
8 months ago Thanks. I'm not actively working on it anymore.
Here's a summary of what I've done: I used a third-party tool (lychee) to generate a list of what the tool considers to be broken links. @turbomam found some false positives in the list. I then re-generated the list in an attempt to eliminate those false positives. I documented what I did so people could do it themselves in the future. One thing I haven't done is verify the list is accurate (beyond spot checking a couple URLs, which were, indeed, broken).
We didn't have a specific "definition of done" on this ticket.
Hi @turbomam, I think I participated in this effort because I had past experience with lychee; as opposed to because I have been involved with schema documentation in general. Given that I documented (here) the process I used to check the documentation for broken links (using lychee), are you OK with me reassigning the ticket to you (to go through the list and, if necessary, regenerate the list in the future)?
mslarae13
commented
8 months ago It looks like the next step is review the links that were found to be broken & remove or replace them.
Yes? If so, I think I or Bea can help with that.
bmeluch
commented
8 months ago Here's the contents of that text file (for quick reference):
I went through Eric's list of links and recorded the errors - many actually worked (e.g. they link to a real page with a paywall-blocked paper, or automatically redirect to a functional page even if the original URL isn't anything) What happens next? broken_link_check.txt
ssarrafan
commented
8 months ago Thanks for the update @bmeluch @mslarae13 @turbomam what are the next steps?
mslarae13
commented
7 months ago @turbomam See Bea's above comment. What are the next steps to get the links that were broken updated?
turbomam
commented
7 months ago Thanks @mslarae13 and @bmeluch. This is very valuable.
@eecavanna could you, @sujaypatil96 and I look at this together and come up with policies and or link-checker configurations that better handle the categories of responses that @bmeluch described?
eecavanna
commented
7 months ago Thanks for going through the links and generating that report, @bmeluch!
Hi @turbomam, I can take care of the link-checker configuration once we are at that point.
Here's what I suggest as next steps (based on my understanding of people's backgrounds):
Are you OK moving forward with that plan?
ssarrafan
commented
7 months ago Appears to be active. Will move to new sprint. Please remove from sprint if not actively working on this.
mslarae13
commented
7 months ago @eecavanna @turbomam @bmeluch @sujaypatil96 This was kept in the "in review" column for this spring (ends today). Did it get worked on? Should it be put back into todo / in progress for next sprint? Will people have time to work on the steps Eric identified above?
cmungall
commented
7 months ago @eecavanna https://github.com/marketplace/actions/baler-bad-link-reporter is an action layered on lychee that seems to have some nice functions
ssarrafan
commented
7 months ago Moving to next sprint so I can close out this spring. Please add to backlog if not active.
ssarrafan
commented
6 months ago Doesn't appear to be active. Removing from sprint. @eecavanna @mslarae13 @bmeluch @turbomam
bmeluch
commented
6 months ago @eecavanna @turbomam @sujaypatil96 not sure if you need me for the rest of the steps Eric outlined, but please let me know if I can help with any of it!
eecavanna
commented
6 months ago @eecavanna https://github.com/marketplace/actions/baler-bad-link-reporter is an action layered on lychee that seems to have some nice functions
Thanks, @cmungall. I wasn't familiar with that Action. I like that it can be used to automatically create an Issue about the links it considers to be broken.
I am familiar with the lycheeverse/lychee-action Action currently in use in the NMDC_documentation repo. It does not automatically create Issues, but its documentation does reference a second Action that it can be paired with in order to automatically create issues.
I am in support of having either of those Actions (i.e. baler-bad-link-reporter, or lycheeverse/lychee-action + https://github.com/peter-evans/create-issue-from-file) set up in this repository.
I'll create a separate Issue for setting one of those up.
eecavanna
commented
6 months ago I created a separate issue for setting up a GitHub Actions workflow that checks for broken links and creates an Issue for any that it finds. Props to @cmungall for identifying a GHA action that works out-of-the-box for that.
Here's a comment where I show the GHA action "in action" in a demo repository: https://github.com/microbiomedata/nmdc-schema/issues/1780#issuecomment-1949596732
eecavanna
commented
6 months ago Earlier today, we configured the nmdc-schema repository to have a GitHub Action that runs this link checker for us when one of the repository's maintainers manually clicks this "Run workflow" button in the GitHub UI:
If it finds any broken links, it automatically creates a GitHub Issue containing a list of those broken links. For example, here's the GitHub Issue that it created when it was run this morning: https://github.com/microbiomedata/nmdc-schema/issues/1825
@turbomam has proposed a plan here for working through the list of broken links. One part of the plan is to understand why each link is being reported as broken (in practice, maybe we'll be able to group them into "buckets"—conceptually—and then characterize each bucket). I think the report @bmeluch created a couple months ago will be directly applicable to that characterization work (thanks, @bmeluch!).
eecavanna
commented
3 months ago This issue is about identifying the broken links.
They have been identified. They are listed here: https://github.com/microbiomedata/nmdc-schema/issues/2025
Here's an issue about fixing them: https://github.com/microbiomedata/nmdc-schema/issues/1826
Closing this issue about identifying them.
This task was suggested by @turbomam, who said all the hyperlinks are defined in Markdown files in the
docsdirectory.I plan to use https://github.com/lycheeverse/lychee to check the hyperlinks.
Related: https://github.com/microbiomedata/nmdc-schema/issues/1313