`berkeley-schema-fy24`: `make squeaky-clean all` fails at `gendoc` command

[eecavanna]

I checked out the tip of the main branch of the berkeley-schema-fy24 fork (this commit) and — with no local changes present — spun up the Docker-based development environment and ran $ make squeaky-clean all.

The operation failed with the following error:

poetry run gen-doc -d docs --template-directory src/doc-templates --include src/schema/deprecated.yaml src/schema/nmdc.yaml
Usage: gen-doc [OPTIONS] YAMLFILE
Try 'gen-doc --help' for help.

Error: No such option: --include
make: *** [Makefile:167: gendoc] Error 2

Here's the full transcript:

$ docker compose exec app bash
WARN[0000] /Users/(...)/berkeley-schema-fy24/docker-compose.yml: `version` is obsolete

    Welcome to this nmdc-schema development container,
    which is running Debian GNU/Linux 12 (bookworm).

root@88fea4e9137f:/nmdc-schema# make squeaky-clean all
Error: open local/gold-study-ids.yaml: no such file or directory
rm -rf project
rm -rf tmp
rm -rf docs/*.md
rm -rf docs/*.html
rm -rf nmdc_schema/nmdc_schema_accepting_legacy_ids*
rm -rf examples/output
rm -rf \
    OmicsProcessing.rq \
    local/mongo_as_nmdc_database.ttl \
    local/mongo_as_nmdc_database_cuire_repaired.ttl \
    local/mongo_as_nmdc_database_rdf_safe.yaml \
    local/mongo_as_nmdc_database_validation.log \
    local/mongo_as_unvalidated_nmdc_database.yaml
#rm -rf local/mixs_regen/mixs_subset_modified.yaml # triggers complete regeneration
rm -rf local/mixs_regen/mixs_subset.yaml
rm -rf local/mixs_regen/mixs_subset_modified.yaml.bak
mkdir -p local/mixs_regen
touch local/mixs_regen/.gitkeep
rm -rf nmdc_schema/*.json
rm -rf nmdc_schema/*.tsv
rm -rf nmdc_schema/*.yaml
mkdir project
poetry run gen-linkml \
    --format yaml \
    --mergeimports \
    --metadata \
    --no-materialize-attributes \
    --no-materialize-patterns \
    --useuris \
    --output nmdc_schema/nmdc_schema_accepting_legacy_ids.yaml src/schema/nmdc.yaml
nmdc_schema/nmdc_schema_accepting_legacy_ids.yaml
grep "^'" assets/yq-for-nmdc_schema_accepting_legacy_ids.txt | while IFS= read -r line ; do echo $line ; eval yq -i $line nmdc_schema/nmdc_schema_accepting_legacy_ids.yaml ; done
'(.classes[] | select(.name == "Biosample") | .slot_usage.id.pattern) = ".*"'
'(.classes[] | select(.name == "Biosample") | .slot_usage.part_of.pattern) = ".*"'
'(.classes[] | select(.name == "Biosample") | .slot_usage.id.structured_pattern.syntax) = ".*"'
'(.classes[] | select(.name == "DataObject") | .slot_usage.id.pattern) = ".*"'
'(.classes[] | select(.name == "DataObject") | .slot_usage.id.structured_pattern.syntax) = ".*"'
'(.classes[] | select(.name == "MagsAnalysisActivity") | .slot_usage.id.pattern) = ".*"'
'(.classes[] | select(.name == "MagsAnalysisActivity") | .slot_usage.id.structured_pattern.syntax) = ".*"'
'(.classes[] | select(.name == "MetabolomicsAnalysisActivity") | .slot_usage.id.pattern) = ".*"'
'(.classes[] | select(.name == "MetabolomicsAnalysisActivity") | .slot_usage.id.structured_pattern.syntax) = ".*"'
'(.classes[] | select(.name == "MetagenomeAnnotationActivity") | .slot_usage.id.pattern) = ".*"'
'(.classes[] | select(.name == "MetagenomeAnnotationActivity") | .slot_usage.id.structured_pattern.syntax) = ".*"'
'(.classes[] | select(.name == "MetagenomeAssembly") | .slot_usage.id.pattern) = ".*"'
'(.classes[] | select(.name == "MetagenomeAssembly") | .slot_usage.id.structured_pattern.syntax) = ".*"'
'(.classes[] | select(.name == "MetagenomeSequencingActivity") | .slot_usage.id.pattern) = ".*"'
'(.classes[] | select(.name == "MetagenomeSequencingActivity") | .slot_usage.id.structured_pattern.syntax) = ".*"'
'(.classes[] | select(.name == "MetaproteomicsAnalysisActivity") | .slot_usage.id.pattern) = ".*"'
'(.classes[] | select(.name == "MetaproteomicsAnalysisActivity") | .slot_usage.id.structured_pattern.syntax) = ".*"'
'(.classes[] | select(.name == "MetatranscriptomeActivity") | .slot_usage.id.pattern) = ".*"'
'(.classes[] | select(.name == "MetatranscriptomeActivity") | .slot_usage.id.structured_pattern.syntax) = ".*"'
'(.classes[] | select(.name == "MetatranscriptomeAnnotationActivity") | .slot_usage.id.pattern) = ".*"'
'(.classes[] | select(.name == "MetatranscriptomeAnnotationActivity") | .slot_usage.id.structured_pattern.syntax) = ".*"'
'(.classes[] | select(.name == "MetatranscriptomeAssembly") | .slot_usage.id.pattern) = ".*"'
'(.classes[] | select(.name == "MetatranscriptomeAssembly") | .slot_usage.id.structured_pattern.syntax) = ".*"'
'(.classes[] | select(.name == "NomAnalysisActivity") | .slot_usage.id.pattern) = ".*"'
'(.classes[] | select(.name == "NomAnalysisActivity") | .slot_usage.id.structured_pattern.syntax) = ".*"'
'(.classes[] | select(.name == "OmicsProcessing") | .slot_usage.id.pattern) = ".*"'
'(.classes[] | select(.name == "OmicsProcessing") | .slot_usage.part_of.pattern) = ".*"'
'(.classes[] | select(.name == "OmicsProcessing") | .slot_usage.has_input.pattern) = ".*"'
'(.classes[] | select(.name == "OmicsProcessing") | .slot_usage.has_output.pattern) = ".*"'
'(.classes[] | select(.name == "OmicsProcessing") | .slot_usage.id.structured_pattern.syntax) = ".*"'
'(.classes[] | select(.name == "ReadBasedTaxonomyAnalysisActivity") | .slot_usage.id.pattern) = ".*"'
'(.classes[] | select(.name == "ReadBasedTaxonomyAnalysisActivity") | .slot_usage.id.structured_pattern.syntax) = ".*"'
'(.classes[] | select(.name == "ReadQcAnalysisActivity") | .slot_usage.id.pattern) = ".*"'
'(.classes[] | select(.name == "ReadQcAnalysisActivity") | .slot_usage.id.structured_pattern.syntax) = ".*"'
'(.classes[] | select(.name == "Study") | .slot_usage.id.pattern) = ".*"'
'(.classes[] | select(.name == "Study") | .slot_usage.id.structured_pattern.syntax) = ".*"'
poetry run gen-linkml \
    --format yaml \
    --mergeimports \
    --metadata \
    --no-materialize-attributes \
    --materialize-patterns \
    --useuris \
    --output nmdc_schema/nmdc_schema_accepting_legacy_ids.yaml.temp nmdc_schema/nmdc_schema_accepting_legacy_ids.yaml
nmdc_schema/nmdc_schema_accepting_legacy_ids.yaml.temp
mv nmdc_schema/nmdc_schema_accepting_legacy_ids.yaml.temp nmdc_schema/nmdc_schema_accepting_legacy_ids.yaml
poetry run gen-json-schema \
    --closed nmdc_schema/nmdc_schema_accepting_legacy_ids.yaml > nmdc_schema/nmdc_schema_accepting_legacy_ids.schema.json
poetry run gen-python --log_level ERROR --validate nmdc_schema/nmdc_schema_accepting_legacy_ids.yaml > nmdc_schema/nmdc_schema_accepting_legacy_ids.py # todo doesn't honor --log_level
poetry run test-more-tolerant-schema
id: xxx
type: nmdc:Biosample
associated_studies:
- gold:Gs0114663
env_broad_scale:
  type: nmdc:ControlledIdentifiedTermValue
  term:
    id: ENVO:00010483
    type: nmdc:OntologyClass
    name: freshwater environment
env_local_scale:
  type: nmdc:ControlledIdentifiedTermValue
  term:
    id: ENVO:00010483
    type: nmdc:OntologyClass
    name: freshwater environment
env_medium:
  type: nmdc:ControlledIdentifiedTermValue
  term:
    id: ENVO:00010483
    type: nmdc:OntologyClass
    name: freshwater environment

# keep these in sync between PROJECT_FOLDERS and the includes/excludes for gen-project and test-schema
poetry run gen-project \
    --exclude excel \
    --exclude graphql \
    --exclude jsonld \
    --exclude markdown \
    --exclude proto \
    --exclude shacl \
    --exclude shex \
    --exclude sqlddl \
    --include jsonldcontext \
    --include jsonschema \
    --include owl \
    --include python \
    --include rdf \
    -d project src/schema/nmdc.yaml && mv project/*.py nmdc_schema
WARNING:root:ignoring equals_string=plate as unable to tell if literal
WARNING:root:ignoring equals_string=plate as unable to tell if literal
WARNING:root:ignoring equals_string=plate as unable to tell if literal
WARNING:root:ignoring equals_string=plate as unable to tell if literal
WARNING:root:ignoring equals_string=dataset_doi as unable to tell if literal
WARNING:root:ignoring equals_string=award_doi as unable to tell if literal
WARNING:root:Null expr in: [None, None] for http://www.w3.org/2002/07/owl#unionOf None
WARNING:root:Null expr in: [None] for http://www.w3.org/2002/07/owl#intersectionOf None
WARNING:root:ignoring equals_string=pass as unable to tell if literal
WARNING:root:Multiple owl types {rdflib.term.URIRef('http://www.w3.org/2000/01/rdf-schema#Literal'), rdflib.term.URIRef('http://www.w3.org/2002/07/owl#Thing')}
WARNING:root:Multiple owl types {rdflib.term.URIRef('http://www.w3.org/2000/01/rdf-schema#Literal'), rdflib.term.URIRef('http://www.w3.org/2002/07/owl#Thing')}
WARNING:root:ignoring equals_string=retention_index as unable to tell if literal
WARNING:root:ignoring equals_string=gas_chromatography as unable to tell if literal
WARNING:root:ignoring equals_string=liquid_chromatography as unable to tell if literal
WARNING:root:ignoring equals_string=gas_chromatography as unable to tell if literal
WARNING:root:Null expr in: [None, None] for http://www.w3.org/2002/07/owl#unionOf None
WARNING:root:Null expr in: [None] for http://www.w3.org/2002/07/owl#intersectionOf None
WARNING:root:Multiple owl types {rdflib.term.URIRef('http://www.w3.org/2000/01/rdf-schema#Literal'), rdflib.term.URIRef('http://www.w3.org/2002/07/owl#Thing')}
WARNING:root:Multiple owl types {rdflib.term.URIRef('http://www.w3.org/2000/01/rdf-schema#Literal'), rdflib.term.URIRef('http://www.w3.org/2002/07/owl#Thing')}
WARNING:root:Multiple owl types {rdflib.term.URIRef('http://www.w3.org/2000/01/rdf-schema#Literal'), rdflib.term.URIRef('http://www.w3.org/2002/07/owl#Thing')}
cp project/jsonschema/nmdc.schema.json  nmdc_schema
# added copying of images and renaming of TEMP.md
cp src/docs/*md docs ; \
cp -r src/docs/images docs ; \
poetry run gen-doc -d docs --template-directory src/doc-templates --include src/schema/deprecated.yaml src/schema/nmdc.yaml
Usage: gen-doc [OPTIONS] YAMLFILE
Try 'gen-doc --help' for help.

Error: No such option: --include
make: *** [Makefile:167: gendoc] Error 2
root@88fea4e9137f:/nmdc-schema#

Here's the problematic target (Makefile lines 165-171 are shown here):

gendoc: $(DOCDIR)
    # added copying of images and renaming of TEMP.md
    cp $(SRC)/docs/*md $(DOCDIR) ; \
    cp -r $(SRC)/docs/images $(DOCDIR) ; \
    $(RUN) gen-doc -d $(DOCDIR) --template-directory $(SRC)/$(TEMPLATEDIR) --include src/schema/deprecated.yaml $(SOURCE_SCHEMA_PATH)
    mkdir -p $(DOCDIR)/javascripts
    $(RUN) cp $(SRC)/scripts/*.js $(DOCDIR)/javascripts/

Source: https://github.com/microbiomedata/berkeley-schema-fy24/blob/37388fef9573ae46d86fb5b7962e55b2e0eb80bd/Makefile#L165-L171

[eecavanna]

In case it helps: Here's what $ git status shows immediately after the error occurs:

root@88fea4e9137f:/nmdc-schema# git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    deleted:    nmdc_schema/gold-to-mixs.sssom.tsv
    modified:   nmdc_schema/nmdc.py
    deleted:    nmdc_schema/nmdc_materialized_patterns.schema.json
    deleted:    nmdc_schema/nmdc_materialized_patterns.yaml
    modified:   nmdc_schema/nmdc_schema_accepting_legacy_ids.py
    deleted:    nmdc_schema/nmdc_schema_merged.yaml

no changes added to commit (use "git add" and/or "git commit -a")

[ssarrafan]

@eecavanna @turbomam I'm going to move this to the new sprint since the issue was just created 2 days ago and the sprint ends today.

[kheal]

@eecavanna I too saw this error locally. I fixed it by sychronizing my environment with the poetry lockfile using $ poetry install. Have you tried that?

[eecavanna]

Thanks, @kheal! No, I haven't. I forgot to do that. I will try that and report back.

[sierra-moxon]

+1 @kheal - there was an upgrade to linkml to support the deprecation process we discussed.

[eecavanna]

That did the trick, @kheal — thanks again!

For future reference, here's what I ran:

• On my host machine:

  docker compose up --detach
  docker compose exec app bash

• Then, within the app container:

  poetry install
  make squeaky-clean all

I will close this issue. I consider this user error on my part (I haven't gotten into the habit of running poetry install on a regular basis yet).