search

cellarium-ai / cellarium-cloud

Cellarium Cloud Core Library
BSD 3-Clause "New" or "Revised" License
5 stars 1 forks source link

Consensus Engine #144

Closed fedorgrab closed 6 months ago

fedorgrab commented 7 months ago

Changes:

  1. Cell Metadata models and querying. Check out how a temp table query could be executed in SQLalchemy, no raw SQL required, which seems p cool.

  2. Admin restructure. Since we now getting more and more models in admin, seems reasonable to group them together. Now admin page nav bar looks like:

    image

    Where All user related models go under Users, all ML realted models go under ML Management and all Cell Metadata models go under Cell Data management.

  3. Consensus Engine which includes ontology_aware (new method that uses postgres for cell metadata fetching), and the old method which is now called cell_type_count. Old method should be deprecated and rewritten with use of postgres as well, but not in this PR.:)

  4. In local environment SQLAlchemy will print all SQL queries

  5. Inspired by @nmalfroy 's PR (#141), adding more pydantic models to wrap internal data flows into the python data classes

nmalfroy commented 7 months ago

Meta question: how does the data get ingested into postgres? We should document that in the readme

fedorgrab commented 7 months ago

Meta question: how does the data get ingested into postgres? We should document that in the readme

About that, we have cell data lifecycle scripts, one of which is 'ingest'. I believe there should be significant refactoring there (they've been used for bq ingest). Also, this seems like it should be a separate PR to me.

nmalfroy commented 7 months ago

Meta question: how does the data get ingested into postgres? We should document that in the readme

About that, we have cell data lifecycle scripts, one of which is 'ingest'. I believe there should be significant refactoring there (they've been used for bq ingest). Also, this seems like it should be a separate PR to me.

Yep, totally agreed. Just wanted to make sure it's on the radar

fedorgrab commented 7 months ago

Meta question: how does the data get ingested into postgres? We should document that in the readme

About that, we have cell data lifecycle scripts, one of which is 'ingest'. I believe there should be significant refactoring there (they've been used for bq ingest). Also, this seems like it should be a separate PR to me.

Yep, totally agreed. Just wanted to make sure it's on the radar

Oh totally there's even an old issue for it.