Open bethac07 opened 2 years ago
(I am not 100% convinced this will tell us anything super useful, but since it's really easy(see note) we might as well do it . I would do #2 first though, because if we decide that ie for all our comparisons we want to use subsampled data to make things more apples to apples, we'd then have to do this again)
Note: (should be <30 lines of code- make a list of all of the compartments+channels+measurement types, open each per-batch CSV, grab total column count, and then for each "thing to check for" report len([x for x in columns if thing_to_check_for in x])
(I'm sure there are more efficient ways to do this if we had to check millions of columns times thousands of factors but this should be enough for our small purposes)
The question was raised as to whether the feature composition of each batch is the same; I think it's going to be hard to do this on a per feature level because of the random dropout of highly correlated features, but I do think there are a few metrics we can quite easily generate for each batch based just on the columns present in each CSV: