bryanlaraway
commented
5 months ago Easy enough to do on the DuckDB side. Can trim the phenotype table after initial creation simply by inserting the following code snippet:
duckdb.sql("DELETE FROM phenotypes WHERE phenotype_row_number > 10")
Should be able to implement something similar on the python side.
It would be nice to have a test mode/development mode available to be able to run the pipeline from end to end with a small set of phenotypes, say 10 or 100 phenotypes per species, just to facilitate speed of testing changes to the pipeline. Thinking this may be easiest to implement at the start of the pipeline at the initial parsing of the Monarch KG, with a simple limit applied to the number of distinct phenotypes parsed for each taxon.