arteymix
commented
1 year ago In our current data model, raw and processed data are treated as two separate relations: ExpressionExperiment.rawExpressionDataVectors and ExpressionExperiment.processedExpressionDataVectors. In addition, their QT are marked as preferred via two different corresponding boolean fields: isPreferred and isMaskPreferred. I learned that quite painfully when I had to generate the data download endpoints for the REST API.
In cases where you don't specify the QT, the preferred raw or preferred processed vectors should be used if available. There should never be more than one preferred of each type.
Here's the thing though, sometimes vectors are removed, but the QT attached to the EE is still there since it's a denormalization, so the multi-QT badge can be spurious. I notably fixed dangling QT in 1c9214d2ba7ac810cdda1f67db85f8baa01f2387 when we replace raw vectors.
I think the multi-QT badge is only applicable to raw data.
Most of our code work under the assumption that there is one preferred processed QT (via isMaskPreferred) and most methods that do not expect the QT will retrieve it this way.
To capture your idea of a single processed QT that is by-default preferred per dataset, we would have to get rid of isMaskPreferred and make two subclasses of QuantitationType for raw and processed vectors and use those subclasses in the vectors.
The new data model would be:
class ExpressionExperiment extends BioAssaySet {
List<QuantitationType> quantitationTypes; // mixture of raw and processed QTs (denormalization
Set<RawExpressionDataVector> rawExpressionDataVectors; // groupable by QT, one is preferred
Set<ProcessedExpressionDataVector> processedExpressionDataVectors; // only one
}
class QuantitationType {
ScaleType scale
Representation representation
}
class RawQuantitationType extends QuantitationType {
boolean isPreferred // (at most one per ExpressionExperiment)
}
class ProcessedQuantitationType extends QuantitationType {
// nothing, just basic QT stuff
}
class DataVector<T extends QuantitationType> {
byte[] data
T quatitationType
}
class RawExpressionDataVector extends DataVector<RawQuantitationType> {
}
class ProcessedExpressionDataVector extends DataVector<ProcessedQuantitationType> {
}That would require an additional class column in the database to distinguish between raw and processed QT type, which is actually a pretty good thing because we are essentially guessing right now.
If we can confirm that there is only one processed QT & set of vectors per datasets, we could start deprecating isMaskPreferred right away and start simplifying some of our logic.
ppavlidis
There is confusion about the semantics of "preferred" and there might be a basic error in how this works (probably my fault)
The desired behaviour is for
preferredto just be a flag to indicate which "raw" expression data is intended to be used as the primary source for "processed" data. The only reason this exists is because some data sets (I'm looking at you, two-color microarrays) there are often multiple numeric quantitation types from GEO. GEO always (?) calls a column "VALUE" in the SOFT file to point us at it.Then, when we make processed data, we really don't need to care if the processed data has the preferred QT flag. There is only one "expression matrix" for the processed data, so there is no need to distinguish among QTs.
I'm a little confused about what is going on. I don't know if all of the following are violated, but the following behaviour is needed:
preferredflag for any function (i.e. DEA)preferredflag on the processed data QT. I believe that is always what triggers it now. It would still be reasonable to show admins a flag if there are >1 preferred raw data QTpreferredflag. We can consider some refactoring of the QT class hierarchy i.e. make a QT subclass that doesn't expose this property. But I'd be fine with just ensuring that the flag is FALSE for the processed data QT.preferredflag, so that if processed data needs to be regenerated for any reason, the appropriate raw QT can be identified.I'm marking this as a bug though some of the above are enhancements. I'd say the most downstream actual bug is the thing about the multiQT badge.