Open eecavanna opened 7 months ago
Yeah, pure-export needs a lot of work! I already have a few related issues that I will add soon. @brynnz22 and I have had some conversations about the local/mongo_as_unvalidated_nmdc_database.yaml
target.
Maybe there should be subcommands for
Or do you think that doing away with the hybrid/fallback solution is a bad idea @eecavanna ?
Do you have suggestions for breaking improvements to pure-export into small commits and then prioritizing them? I would love to work on this with you.
We could also include @mbthornton-lbl because he wrote an API-only dumper with fewer configuration options that only returns records (from multiple collections) if they have some path back to a named Study. We worked on that together to
get-study-related-records
, as defined by the tool.poetry.scripts
section in pyproject.toml
local/.env
. I would like to either integrate get-study-related-records
with pure-export
or at least to make the launching and configuration even more similar.
The
make
targetlocal/mongo_as_unvalidated_nmdc_database.yaml
runs a command calledpure-export
, passing it several CLI options, including:--client-base-url https://api.microbiomedata.org
--mongo-host localhost
--mongo-port 27777
A user could have localhost port
27777
mapped to a different Mongo database than the one used by the web server athttps://api.microbiomedata.org
. In that case, when thepure-export
command (i.e. whatever runs when that command is issued) uses a Mongo client, the Mongo database it is accessing would differ from the one it would be accessing when using HTTP.Here's a message the command showed when I ran it:
I want to emphasize the final line:
warning: FastAPI request to nmdcschema/functional_annotation_agg appears to have failed. Trying as a PyMongo query.
This implies to me that the underlying code first tried to fetch data via HTTP (which involved one Mongo database) and then was going to try via a Mongo client (which would involve a different Mongo database).