Test that the MIMIC-IV ETL runs on the MIMIC-IV demo dataset

[tompollard]

As discussed in https://github.com/Medical-Event-Data-Standard/meds_etl/issues/1, we would like to add some tests to confirm that the MIMIC ETL runs as expected.

This pull request:

• Adds a step to the Github workflow that wgets the MIMIC-IV demo dataset from PhysioNet and puts it at: tests/data/mimic-iv-demo
• Adds instructions for downloading the MIMIC-IV demo dataset to tests/data/mimic-iv-demo locally
• If the dataset exists when pytest is run, checks that the meds_etl_mimic script successfully outputs files to a temporary destination folder at tests/data/mimic-iv-demo/build/

[tompollard]

Sorry for the churn! The test now runs:

• I added a few missing dependencies to pyproject.toml: jsonschema, meds, typing_extensions
• The behaviour of importlib.resources changed between Python 3.9 and 3.10. There are several possible fixes, but the easiest one is just to drop testing on < Python 3.10.

[tompollard]

My main request is that we want the test to at minimum try to load the resulting files with huggingface Datasets.

Thanks Ethan. I added a couple of simple tests as suggested.

[tompollard]

Hmm, looks like the build was broken by a recent update in meds:

https://github.com/Medical-Event-Data-Standard/meds/commit/e93f63a2f9642123c49a31ecffcdb84d877dc54a

So successful test?

[EthanSteinberg]

Good catch and sorry about that! I pushed an update to the meds package which should fix this.

https://pypi.org/project/meds/0.1.1/

Now the pyproject.toml file needs to be updated to require 0.1.1 instead of 0.1

Also, we shouldn't use subprocess here since it results in terrible error messages.

We should just import main from the corresponding module.

[tompollard]

Now the pyproject.toml file needs to be updated to require 0.1.1 instead of 0.1

Thanks, pull request updated and tests are passing again.

Also, we shouldn't use subprocess here since it results in terrible error messages. We should just import main from the corresponding module.

I think this change belongs in a new pull request, but if you let me know how you'd like to implement it then I can add it here. The ETL needs refactoring beforehand really.

[EthanSteinberg]

I think this change belongs in a new pull request, but if you let me know how you'd like to implement it then I can add it here. The ETL needs refactoring beforehand really.

Yep, let's just merge this as is.

[tompollard]

Thanks for reviewing!