search

Medical-Event-Data-Standard / meds

Schema definitions and Python types for Medical Event Data Standard, a standard for medical event data such as EHR and claims data
Apache License 2.0
38 stars 3 forks source link

Do we really want to store all the label columns in every task dataframe even when the task will just have one kind of label? #42

Open mmcdermott opened 3 months ago

mmcdermott commented 3 months ago

@EthanSteinberg, should we somehow indicate that the label columns are optional or don't all need to be included? It seems silly that we'll need to make every label dataframe have columns for every kind of task when all but one will be empty.

https://github.com/Medical-Event-Data-Standard/meds/blob/5f87c2fdcce7f8bab46af6f81ef7892fdee098c1/src/meds/schema.py#L58

EthanSteinberg commented 3 months ago

I'd be open to that, but I'm not sure the best way to do it

mmcdermott commented 3 months ago

I don't think there is a clean way to support it, after googling around a bit. If/when we solve #37 we can integrate this capability in there (e.g., make it so that if you use meds utilities to re-cast and/or validate and/or read your data in the specified schemas, optional columns are enforced and/or cast to the right types if they are present but no issue is raised if they are not?

I also filed https://github.com/apache/arrow/issues/43626 but I don't know if this is in scope or something they'd want on their end.