OMOP test case: generalize-tsv

matentzn commented 2 years ago

Fixes #74

Blocked by:

cmungall commented 1 year ago

can we close this?

matentzn commented 1 year ago

I forgot to make a fork, sorry. If you see this PR on the 1st January 2023 just close it. I still hope to do some work here and GitHub is my extended memory. :P

matentzn commented 1 year ago

Environment:

poetry run pip list | grep linkml
The currently activated Python version 3.8.3 is not supported by the project (>=3.9,<4.0.0).
Trying to find and use a compatible version. 
linkml                        1.3.14
linkml-dataops                0.1.0
linkml-runtime                1.3.7

Given this slot:

concept_list:
    description: Container slot for CONCEPT.
    from_schema: https://w3id.org/omop_vocabulary
    multivalued: true
    range: CONCEPT

and this class:

CONCEPT_CONTAINER:
    from_schema: https://w3id.org/omop_vocabulary
    slots:
    - concept_list

I get this error:

Using python3.9 (3.9.13)
Loading from tests/resources/tsvs/CONCEPT.csv
Traceback (most recent call last):
  File "/Users/matentzn/Library/Caches/pypoetry/virtualenvs/schema-automator-iL9MyQRf-py3.9/bin/linkml-convert", line 8, in <module>
    sys.exit(cli())
  File "/Users/matentzn/Library/Caches/pypoetry/virtualenvs/schema-automator-iL9MyQRf-py3.9/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/matentzn/Library/Caches/pypoetry/virtualenvs/schema-automator-iL9MyQRf-py3.9/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/matentzn/Library/Caches/pypoetry/virtualenvs/schema-automator-iL9MyQRf-py3.9/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/matentzn/Library/Caches/pypoetry/virtualenvs/schema-automator-iL9MyQRf-py3.9/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/matentzn/Library/Caches/pypoetry/virtualenvs/schema-automator-iL9MyQRf-py3.9/lib/python3.9/site-packages/linkml/utils/converter.py", line 140, in cli
    obj = loader.load(source=input, target_class=py_target_class, **inargs)
  File "/Users/matentzn/Library/Caches/pypoetry/virtualenvs/schema-automator-iL9MyQRf-py3.9/lib/python3.9/site-packages/linkml_runtime/loaders/csv_loader.py", line 44, in load
    return JSONLoader().loads(json.dumps({index_slot: objs}), target_class=target_class)
  File "/Users/matentzn/Library/Caches/pypoetry/virtualenvs/schema-automator-iL9MyQRf-py3.9/lib/python3.9/site-packages/linkml_runtime/loaders/loader_root.py", line 126, in loads
    return self.load(source, target_class, metadata=metadata)
  File "/Users/matentzn/Library/Caches/pypoetry/virtualenvs/schema-automator-iL9MyQRf-py3.9/lib/python3.9/site-packages/linkml_runtime/loaders/loader_root.py", line 85, in load
    results = self.load_any(*args, **kwargs)
  File "/Users/matentzn/Library/Caches/pypoetry/virtualenvs/schema-automator-iL9MyQRf-py3.9/lib/python3.9/site-packages/linkml_runtime/loaders/json_loader.py", line 28, in load_any
    return self.load_source(source, loader, target_class,
  File "/Users/matentzn/Library/Caches/pypoetry/virtualenvs/schema-automator-iL9MyQRf-py3.9/lib/python3.9/site-packages/linkml_runtime/loaders/loader_root.py", line 66, in load_source
    return target_class(**data_as_dict)
  File "<string>", line 13, in __init__
  File "test", line 292, in __post_init__
  File "/Users/matentzn/Library/Caches/pypoetry/virtualenvs/schema-automator-iL9MyQRf-py3.9/lib/python3.9/site-packages/linkml_runtime/utils/yamlutils.py", line 48, in __post_init__
    raise ValueError('\n'.join(messages))
ValueError:  Unknown argument: concept_list = [{'concept_id': 1146945, 'concept_name':
make[1]: *** [tests/outputs/omop_data.ttl] Error 1
make: *** [test-omop] Error 2

When running:

poetry run linkml-convert -t rdf -s tests/outputs/omop.yaml \
        -m tests/outputs/omop/omop.py \
        -S concept_list \
        -C CONCEPT tests/resources/tsvs/CONCEPT.csv \
        --output tests/outputs/omop_data.ttl

matentzn commented 1 year ago

From @sierra-moxon on slack (not to lose)

likely concept_list needs an inlined_as_list: true addition, and your container class CONCEPT_CONTAINER needs a tree_root: true addition. I added both of those in the generated yaml itself (to try and test this), and I still get the error (these are likely changes for schema_automator/linkml-runtime itself). But I think that's because linkml-convert needs to somehow know that your CONCEPT.csv is the content of concept_list - that is where I am in debugging.

I'm finding this difficult to debug as its being called from a Makefile and involves several software packages (linkml, linkml-runtime) upstream -- I started a test here: https://github.com/linkml/linkml/blob/omop/tests/test_utils/test_converter.py

linkml / schema-automator

OMOP test case: generalize-tsv #98