Open black-snow opened 1 year ago
@haganjam can you give me a rough description of what's inside the rds files? I'm not sure yet that they are the reason things are so slow - I'd have to do some profiling. Maybe it's some heavy computation instead.
Is it all just tables? If so, can't we just use CSV instead?
If computation is the bottleneck I'd have to do some digging to see where exactly and what to tweak.
MEMO: if rds is slow, maybe https://www.fstpackage.org/ can help
The rds files are a range of different file-types actually. For example, the freshwater_ecoregion_map.rds is a spatial polygon object class in R. The ..._higher_taxon_matrices.rds files are effectively igraph objects classes. So, when they are read in, igraph as a package needs to be loaded.
I used to store the ..._higher_taxon_matrices.rds as spare matrices and then converting them to igraph objects but that ended up taking up a lot of space and it was quite slow.
The equation_database and taxon_databases are just tables so they could be stored as .csv files.
Intuitively, I doubt that the .rds extension is the problem. I think it's more likely that the tests take minutes to complete because of the bdc functions that clean and harmonise the taxon names. This generally takes quite a bit of time. In the tests, if we could only run those functions once or twice, I think it would become a lot faster.
Alright, I'll invest a minute in profiling when I get to it.
bdc::bdc_query_names_taxadb
seems to be slow - 17-23s on my machine for one test case.
Apparently missing entries / an exhaustive search is the most expensive thing:
bdc::bdc_query_names_taxadb(sci_name=c('Triops granitica'), rank_name="Animalia", rank="kingdom")
/edit: let's see if there's an easy "fix" with https://github.com/brunobrr/bdc/issues/252 - otherwise I either have to dig into bdc
or refactor, so that we can inject a test double.
No quick fix - let's mock it except for one explicit test using the actual dependency maybe.
Working on it in https://github.com/haganjam/FW_invert_biomass_allometry/tree/test_doubles
Tests for clean_taxon_names
are now pretty much instantaneous.
Reading the rds files (?) is painfully slow and makes the tests take minutes to complete. Maybe we can speed things up.
I'll have to take a look at what's actually inside. Maybe we can go with, e.g., sqlite instead if it's just tabular data.