save the content in `local/from_schemasheets.yaml` and retire makefile target that builds it

turbomam commented 2 weeks ago

We may want to give it a new path and new name

pkalita-lbl commented 2 weeks ago

On 2024-10-24, Montana, Mark and I decided to remove the use of schemasheets to build a "starting point" schema (to which the subsequent shuttling and modification steps are applied). Enough team members are now familiar enough with editing LinkML YAML files directly that the TSV-to-YAML step via schemasheets is an unnecessary complication.

This task is to:

Move the current output of the schemasheets process (local/from_schemasheets.yaml) to src/nmdc_submission_schema/schema/nmdc_submission_schema_base.yaml
Update any Makefile targets which utilize local/from_schemasheets.yaml and point them to src/nmdc_submission_schema/schema/nmdc_submission_schema_base.yaml
Remove the schemasheets directory
Remove the schemasheets dependency from pyproject.toml
Remove Makefile targets which use schemasheets CLIs and read from/write to the schemasheets directory

turbomam commented 2 weeks ago

I have generated src/nmdc_submission_schema/schema/nmdc_submission_schema_base. and have linked a PR possible followup actions:

[ ] update schema version, possibly using dynamic versioning
[ ] review prefixes
[x] remove LinkML types from types and add import statement.
- [ ] https://github.com/microbiomedata/submission-schema/blob/main/schemasheets/tsv_in/types.tsv
- [ ] improve sheets_and_friends so that dependent types (unit, decimal degree, and language code) are imported from nmdc-schema, as dependent classes and enums are.
  - I'm planning on replacing the whitespaces in types and subset names with underscores in the upstream nmdc-schema soon)
[ ] enums
- [ ] remove 'placeholder PV descr' descriptions
- [x] make annotations flatter
- [ ] remove sntc_name annotations
- [ ] more grooming might be helpful
[ ] slots:
- [ ] some may not have ranks, although those may be added later during the modification phase?
[ ] classes:
- [ ] add aliases for the interfaces, such as the verbatim MIxS class names

turbomam commented 2 weeks ago

May actually need to run schemas genrated later in the build process though soemthing like this

src/nmdc_submission_schema/schema/nmdc_submission_schema_base.yaml: local/from_schemasheets.yaml
    cat $< | yq eval '(.. | select(has("from_schema")) | .from_schema) style="" | del(.. | select(has("from_schema")).from_schema)' |\
    yq eval '.classes[] |= select(has("annotations")).annotations |= map_values(.value)' |\
    yq eval '.prefixes |= map_values(.prefix_reference)' |\
    yq eval '.settings |= map_values(.setting_value)'  |\
    yq eval '.slots[] |= select(has("annotations")).annotations |= map_values(.value)' |\
    yq eval 'del(.classes.[].name)' |\
    yq eval 'del(.classes.[].slot_usage.[].name)'  |\
    yq eval 'del(.enums.[].name)'  |\
    yq eval 'del(.enums.[].permissible_values.[].text)' |\
    yq eval 'del(.slots.[].domain)'  |\
    yq eval 'del(.slots.[].name)' |\
    yq eval 'del(.source_file)'  |\
    yq eval 'del(.subsets.[].name)' | cat > $@

turbomam commented 2 weeks ago

poetry run do_shuttle \
                        --config_tsv  sheets_and_friends/tsv_in/import_slots_regardless.tsv \
                        --recipient_model src/nmdc_submission_schema/schema/nmdc_submission_schema_base.yaml \
                        --yaml_output local/with_shuttles.yaml.raw

Traceback (most recent call last):
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/bin/do_shuttle", line 8, in <module>
    sys.exit(do_shuttle())
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/sheets_and_friends/shuttle.py", line 44, in do_shuttle
    shuttle.prep_views_dict()
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/sheets_and_friends/shuttle.py", line 93, in prep_views_dict
    temp = SchemaView(k)
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/linkml_runtime/utils/schemaview.py", line 138, in __init__
    schema = load_schema_wrap(schema)
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/linkml_runtime/utils/schemaview.py", line 70, in load_schema_wrap
    schema = yaml_loader.load(path, target_class=SchemaDefinition, **kwargs)
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/linkml_runtime/loaders/loader_root.py", line 71, in load
    results = self.load_any(*args, **kwargs)
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/linkml_runtime/loaders/yaml_loader.py", line 41, in load_any
    return self._construct_target_class(data_as_dict, target_class)
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/linkml_runtime/loaders/loader_root.py", line 132, in _construct_target_class
    return target_class(**data_as_dict)
  File "<string>", line 66, in __init__
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/linkml_runtime/linkml_model/meta.py", line 589, in __post_init__
    self._normalize_inlined_as_dict(slot_name="enums", slot_type=EnumDefinition, key_name="name", keyed=True)
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/linkml_runtime/utils/yamlutils.py", line 100, in _normalize_inlined_as_dict
    self._normalize_inlined(slot_name, slot_type, key_name, keyed, False)
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/linkml_runtime/utils/yamlutils.py", line 205, in _normalize_inlined
    form_1({k: v})
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/linkml_runtime/utils/yamlutils.py", line 153, in form_1
    order_up(key, slot_type(**as_dict(raw_obj)))
  File "<string>", line 64, in __init__
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/linkml_runtime/linkml_model/meta.py", line 976, in __post_init__
    self._normalize_inlined_as_dict(slot_name="permissible_values", slot_type=PermissibleValue, key_name="text", keyed=True)
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/linkml_runtime/utils/yamlutils.py", line 100, in _normalize_inlined_as_dict
    self._normalize_inlined(slot_name, slot_type, key_name, keyed, False)
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/linkml_runtime/utils/yamlutils.py", line 205, in _normalize_inlined
    form_1({k: v})
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/linkml_runtime/utils/yamlutils.py", line 153, in form_1
    order_up(key, slot_type(**as_dict(raw_obj)))
  File "<string>", line 43, in __init__
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/linkml_runtime/linkml_model/meta.py", line 3370, in __post_init__
    self._normalize_inlined_as_dict(slot_name="structured_aliases", slot_type=StructuredAlias, key_name="literal_form", keyed=False)
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/linkml_runtime/utils/yamlutils.py", line 100, in _normalize_inlined_as_dict
    self._normalize_inlined(slot_name, slot_type, key_name, keyed, False)
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/linkml_runtime/utils/yamlutils.py", line 205, in _normalize_inlined
    form_1({k: v})
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/linkml_runtime/utils/yamlutils.py", line 153, in form_1
    order_up(key, slot_type(**as_dict(raw_obj)))
  File "<string>", line 40, in __init__
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/linkml_runtime/linkml_model/meta.py", line 1250, in __post_init__
    super().__post_init__(**kwargs)
  File "/Users/MAM/Documents/gitrepos/submission-schema/.venv/lib/python3.9/site-packages/linkml_runtime/utils/yamlutils.py", line 48, in __post_init__
    raise ValueError('\n'.join(messages))
ValueError:  Unknown argument: contexts = ['https://gold.jgi.doe.gov/']

turbomam commented 2 weeks ago

runs to completion in GH action and on my laptop, after poetry update

microbiomedata / submission-schema

save the content in `local/from_schemasheets.yaml` and retire makefile target that builds it #229