CDCgov / cfa-viral-lineage-model

Apache License 2.0
10 stars 0 forks source link

First-draft infrastructure for evaluation #10

Closed thanasibakis closed 3 months ago

thanasibakis commented 3 months ago

This PR's goal is to take a first stab at how our package will provide model evaluation functionality.

Key changes

eval subpackage

The package now has an eval subpackage. The idea here is that this will contain modules for the various classes of metrics we will want (e.g. on proportions directly, on samples, on lineage domination times). I have a simple MAE on proportions in the eval.proportions submodule to start us out.

demo

The exploration/demo workspace has two model-fitting scripts now. The idea here is that the outputs of each are collected into some output folder, and pitted against each other. exploration/demo/evaluate.py does this, but hardcodes CSV paths for now.

Other changes

afmagee42 commented 3 months ago

I'm encountering a few issues that are outside the scope of this PR, but which are blocking me from checking that the code runs in my newly-cloned repo. I suggest that this PR is merged as soon as @swo approves, and I will make a new branch to see about cleaning those up.

swo commented 3 months ago

I'm blocked right away:

linmod-py3.10➜  demo git:(first-metric) ../../data/load_metadata.py > metadata.csv
Will load from https://data.nextstrain.org/files/ncov/open/metadata.tsv.zst

Downloading...Traceback (most recent call last):
  File "/home/ulp7/proj/cfa-viral-lineage-model/exploration/demo/../../data/load_metadata.py", line 108, in <module>
    data = load_metadata(
  File "/home/ulp7/proj/cfa-viral-lineage-model/exploration/demo/../../data/load_metadata.py", line 58, in load_metadata
    with urlopen(url) as response, save_path.open("wb") as out_file:
  File "/usr/lib/python3.10/pathlib.py", line 1119, in open
    return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: '.cache/data.nextstrain.org/files/ncov/open/metadata.tsv'
afmagee42 commented 3 months ago

@swo see #12