Jj fix acq storing import

[sonarcloud[bot]]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

[azazellochg]

Hi, could you give a summary of what is being fixed here? Did we have dose information wrong during import?

[azazellochg]

Okay, after more reading I see that we had initialDose = 0 for all tilt images.

[JorMaister]

Sure, @azazellochg!

• We were reported about a problem with Dynamo subtomogram averaging, and seemed to be related with some missing attributes of the acquisition when importing TS, and thus the propagation of that incomplete data through the pipeline.
• These were attributes angleMin, angleMax, and step (angular step). They're now always present, no matter the angular information data source.
• At the same time, the dose was reviewed, being filled correctly, including the initial dose. The accumulated dose was ok I think.
• Going back to the acquisition, the data model has 3 acquisition levels: Set of sets level (set of TS), set level (TS), and element level (TI). As the sets of TS are potentially heterogeneous due to the exclusion of the views, DW, etc, how each of these acquisitions is filled needs to be reviewed for each TS alignment protocol, as it was done in aretomo. As a reference line, when coding, the Set of set acq should be considered not very representative as it won't represent all the elements because of the potential heterogeneity (but necessary for the project opening). Hence, the acq requested should be in general the one corresponding to the set (TS acq) expecting the PPPT entering protocols, in which I'd recommend getting the one from each TI, as the rotation angle may have been refined individually (imod, eman) and if the dose is desired to be introduced with the exact values instead of a mean (mdoc case, non-homogeneus dose per tilt).

[azazellochg]

Brilliant, thank you @JorMaister . I've run an import on my movies and acq looks correct! The only thing I noticed is: [ FAILED ] TestTomoImportTsFromMdoc.test_mdocFileCheckerRealOk I don't know if that's expected to fail..

[JorMaister]

Tests are passing on my side, @azazellochg. What's the error you got?

[azazellochg]

[ FAILED ] TestTomoImportTsFromMdoc.test_mdocFileCheckerRealOk

Traceback (most recent call last): File "/public/EM/Scipion/miniconda3/envs/scipion3/lib/python3.8/unittest/case.py", line 60, in testPartExecutor yield File "/public/EM/Scipion/miniconda3/envs/scipion3/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/public/EM/Scipion/miniconda3/envs/scipion3/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/public/EM/Scipion/scipion-dev/scipion-em-plugins/scipion-em-tomo/tomo/tests/test_import.py", line 970, in test_mdocFileCheckerRealOk self._checkMDocParsingErrorMsg(mdocList, expectedErrorKeyWordList) File "/public/EM/Scipion/scipion-dev/scipion-em-plugins/scipion-em-tomo/tomo/tests/test_import.py", line 1034, in _checkMDocParsingErrorMsg self.assertTrue(not errorMsg, "There are errors unexpected when validating the Mdocs: %s" % errorMsg) File "/public/EM/Scipion/miniconda3/envs/scipion3/lib/python3.8/unittest/case.py", line 765, in assertTrue raise self.failureException(msg) AssertionError: False is not true : There are errors unexpected when validating the Mdocs: /public/EM/Scipion/scipion-dev/data/tests/tomo-em/ts_tsM_and_mdocs/mdocs/realFromEmpiar/complete/TS-G8-box7-015.mrc.mdoc is missing:

• Dose values. Related mdoc labels are: ExposureDose or FrameDosesAndNumber or (DoseRate and ExposureTime) or (MinMaxMean and CountsPerElectron)

[JorMaister]

It seems a validation error, like if the mdoc does not contain the data required to calculate the dose... Is the corresponding test dataset up to date?

[azazellochg]

You were right, I have removed the offending mdoc and the manifest, after regeneration that it's all working!