Closed thanasibakis closed 3 months ago
I'm encountering a few issues that are outside the scope of this PR, but which are blocking me from checking that the code runs in my newly-cloned repo. I suggest that this PR is merged as soon as @swo approves, and I will make a new branch to see about cleaning those up.
I'm blocked right away:
linmod-py3.10➜ demo git:(first-metric) ../../data/load_metadata.py > metadata.csv
Will load from https://data.nextstrain.org/files/ncov/open/metadata.tsv.zst
Downloading...Traceback (most recent call last):
File "/home/ulp7/proj/cfa-viral-lineage-model/exploration/demo/../../data/load_metadata.py", line 108, in <module>
data = load_metadata(
File "/home/ulp7/proj/cfa-viral-lineage-model/exploration/demo/../../data/load_metadata.py", line 58, in load_metadata
with urlopen(url) as response, save_path.open("wb") as out_file:
File "/usr/lib/python3.10/pathlib.py", line 1119, in open
return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: '.cache/data.nextstrain.org/files/ncov/open/metadata.tsv'
@swo see #12
This PR's goal is to take a first stab at how our package will provide model evaluation functionality.
Key changes
eval
subpackageThe package now has an
eval
subpackage. The idea here is that this will contain modules for the various classes of metrics we will want (e.g. on proportions directly, on samples, on lineage domination times). I have a simple MAE on proportions in theeval.proportions
submodule to start us out.demo
The
exploration/demo
workspace has two model-fitting scripts now. The idea here is that the outputs of each are collected into some output folder, and pitted against each other.exploration/demo/evaluate.py
does this, but hardcodes CSV paths for now.Other changes
utils
submodule continues to grow with helper functions that I think are nice to have all in one place. Happy to hear other ideas here.visualize
submodule has a new function for plotting posterior proportion trajectories over time, replacing the R script I had before.linmod/eval.py
deleted here.