Open KarlLevik opened 5 years ago
You'll need to be able to define what each entry in DC is, so extend experimenttype enum?
I've added a section above about renaming the 'image' columns in DataCollection
+ the necessary SQL.
There is some decoupling to be done here too. EnergyScan for example contains parameters from a processing output (inflection, peak, etc). Really this process should create an autoprocprogram, and populate a table elsewhere with these values. Lots of our display of edgescans (and xfe) comes from file scraping at the moment
Big Picture
As has been highlighted in the past, there are a lot of columns in
EnergyScan
andXFEFluorescenceSpectrum
that also exist inDataCollection
. Thinking about it,EnergyScan
andXFEFluorescenceSpectrum
really are types ofDataCollection
, so should therefore perhaps be merged intoDataCollection
.Additionally, there are some columns in the
DataCollection
table that refer to "images". I think we should rename these so the table make more sense to disciplines that don't use images.I hereby propose to do this work. Let's keep the name
DataCollection
and merge the columns from the two other tables into this. We will also need aDataCollection.experimentType
enum to say whether the data is from an XRF spectrum, energy scan or other type of data collection as currently enumerated in theDataCollectionGroup.experimentType
enum.Columns moving to DataCollectionFileAttachment
We can use the
DataCollectionFileAttachment
table to store the following columns inEnergyScan
:... and also these columns in
XFEFluorescenceSpectrum
:... and these columns in
DataCollection
:(This will probably require some extra values in the
fileType
enum inDataCollectionFileAttachment
.)Columns moving to DataCollection
We can use the following mapping:
Renaming image columns in DataCollection
SQL (DML + DDL)
See the work-in-progress for the SQL statements needed for this issue here - please run them in the order given:
DataCollection
s tables.Appendix: Wide tables with many NULLs
I know that having lots of columns that we know will be
NULL
for many of the rows feels instinctively wrong as we could have normalised and broken the table up into a parent table with multiple child tables. (E.g. in the case ofDataCollection
, maybe this could be done based onexperimentType
.)So I want to assure you that at least from a database storage optimisation point of view, wide tables with lots of
NULL
s is fine. In MariaDB and MySQL, assuming we use the default storage engine, InnoDB, and eitherDYNAMIC
(now default in MariaDB) orCOMPACT
row format, the following is true:Source: https://dev.mysql.com/doc/refman/8.0/en/innodb-row-format.html#innodb-row-format-compact
So as long as the columns are
NULL
-able to start with, we're not wasting any space at all by populating them withNULL
values. Whereas with a normalised approach we would need anINT
for the primary key of each child table + anotherINT
for the foreign key pointing to the parent table + indexes and other overhead that comes with extra tables.I'm happy to provide a test case as evidence in support of this, if anyone is interested. E.g. we can create two
DataCollection
tables in a dev database: One where a lot of columns are populated withNULL
values and another where the same columns do not exist. Then we can compare the difference in size of the tables on disk, and hopefully observe that the difference is very small.