search

ImagingDataCommons / idc-index

Python package to simplify access to the data available from NCI Imaging Data Commons
https://idc-index.readthedocs.io/
MIT License
11 stars 5 forks source link

Documentation of the underlying index tables #126

Open fedorov opened 2 months ago

fedorov commented 2 months ago

We need to set up a process where we could have schemas and relationships among the growing number of those smaller tables automatically reflected in our documentation, and ideally have a visual browser where users could explore those relationships - automatically generated from the schema documents.

Related thread with ideas and relevant technologies: https://discord.com/channels/909674491309850675/921073327009853451/1283795006477565983

fedorov commented 2 months ago

Some NCI components use this for describing the model: https://github.com/CBIIT/c3dc-model

vkt1414 commented 2 months ago

@fedorov in the past, I was discussing this with Deepa about having a relationship diagram.

https://github.com/drawdb-io/drawdb was the tool I found. Its pretty good in my opinion. If you find it good as well, I can help with this issue.

fedorov commented 2 months ago

The other tool mentioned in the thread above - Mermaid - seemed like a nice solution:

drawdb looks sleek, but I think the question is what is next once you modeled it there? I don't want to create yet another manual task for anyone.

On the other hand, we can automatically generate Mermaid code directly from the Parquet files (column name + data type). We could then embed that Mermaid code into the docs. We could also augment idc-index-data with a mechanism to either inject descriptions of the columns directly into Parquet files metadata fields, or require a JSON schema to accompany each query. Or if we want to play nice with CRDC use Bento MDF. We could next generate Mermaid diagram code as part of the release, which could then be picked up downstream in the IDC documentation and/or idc-index documentation.

vkt1414 commented 2 months ago

Looks very cool. I'll try to learn how it works.

fedorov commented 2 months ago

If you plan to contribute your time working on this, I would prefer if we find the time to chat before you jump into getting a PR ready! ;-)

vkt1414 commented 2 months ago

Sure @fedorov! My progress can be slow, but mermaid seems exciting, so I'll see if it can stop me from sleeping early. I can meet tomorrow or Friday after 4 pm.

fedorov commented 2 months ago

Sounds good - let's coordinate on Discord! Thank you!

vkt1414 commented 2 months ago

@fedorov could I know where the column descriptions are there for the tables in idc-index?

fedorov commented 1 month ago

As discussed last week, here's the tentative plan:

  1. add Yaml or JSON file accompanying each of the tables generated by the queries in https://github.com/ImagingDataCommons/idc-index-data/tree/main/scripts and https://github.com/ImagingDataCommons/idc-index-data/tree/main/assets in idc-index-data. This existing documentation can be used to initialize column descriptions: https://github.com/ImagingDataCommons/idc-index/blob/main/docs/column_descriptions.md.
  2. while generating Parquet files with the resulting tables, inject column descriptions into the Parquet schema (see related discussion in https://www.perplexity.ai/search/i-am-distributing-sql-tables-a-yWRtF.V8SMOgIcTiZQ6Dsw.
  3. Upon generating of the release in idc-index-data, include schema files along with the parquet files as release assets.
  4. While generating documentation here in idc-index, replace https://github.com/ImagingDataCommons/idc-index/blob/main/docs/column_descriptions.md with the content automatically generated from Parquet.
  5. Generate Mermaid descriptions for rendering the relationships across tables automatically.