search

ispyb / ispyb-database-modeling

4 stars 3 forks source link

Table for auto processing analysis #10

Open stufisher opened 7 years ago

stufisher commented 7 years ago

Diamond is now running xtraige on each autoprocessed data set, we would like to propose a table to store these results:

AutoProcProgramAnalysis
-------------------------------
autoprocprogramanalysisid primary key auto incremnet
autoprocprogramid int foreign key
severity int (0-2)
message varchar(200)
description text

Example: Severity: 0 == good, 1 == alert, 2 == bad Message

"The overall completeness in low-resolution shells is less than 90%"

Description:

The following table shows the completeness of the data to 5.0 A. Poor low-resolution completeness often leads to map distortions and other difficulties, and is typically caused by problems with the crystal orientation during data collection, overexposure of frames, interference with the beamstop, or omission of reflections by data-processing software.

| Resolution range | N(obs)/N(possible) | Completeness |

| 31.4594 - 10.6518 | [53/151] | 0.351 | | 10.6518 - 8.5098 | [46/134] | 0.343 | | 8.5098 - 7.4504 | [55/131] | 0.420 | | 7.4504 - 6.7766 | [55/125] | 0.440 | | 6.7766 - 6.2951 | [56/128] | 0.438 | | 6.2951 - 5.9265 | [51/126] | 0.405 | | 5.9265 - 5.6315 | [57/123] | 0.463 | | 5.6315 - 5.3876 | [54/122] | 0.443 | | 5.3876 - 5.1811 | [42/119] | 0.353 | | 5.1811 - 5.0031 | [51/130] | 0.392 |

@karllevik can weigh in with sql statements again

antolinos commented 7 years ago

Hi @stufisher,

I had a brief discussion with Olof about this and we wonder if we could make it more generic. It seems to us that AutoProcProgramAnalysis will work only for xtraige (that by the way it is also run at ESRF) and it makes it not very scalable.

Our question is: could we make a new generic table like AutoProcProgramStatistic, for instance, with the following columns:

AutoProcProgramStatistic
---------------------------------
autoProcProgramStatisticId
autoProcProgramId
key
value

Then someone can attach n statistics to an AutoProcProgram but not necessarily severity, message or description. Otherwise, every new program to be executed with different output parameters would need a new table on ISPyB...

What do you think?

stufisher commented 7 years ago

Sorry i think i forgot to explain the use case. We would populate one row in this table for each analysis result from any analysis program. I.e. xtraige might make 10 rows.

We think severity is the most important part of this table so you can specify whether the analysis result is good or bad.

I'm not quite clear how message and description are xtraige specific? Any program could output a message and a description?

I guess message could become summary?

rjgildea commented 7 years ago

message/description could be shortComments/comments or shortText/text in line with some other tables?

antolinos commented 7 years ago

Ok. I understand, we thought severity was a specific output from xtriage but what you are suggesting here is a sort of generic notifications table, right?

drnasmith commented 5 years ago

So think this has evolved into an AutoProcProgramMessage table mentioned under the notifications discussion #3 and discussed on Video conference

AutoProcProgramMessage
----------------------------------
autoProcProgramMessageId - Primary Key
autoProcProgramId - Foreign Key
recordTimeStamp - timestamp
severity - enum('ERROR', 'WARNING', 'INFO')
message - varchar(200)
description - text