Scan-o-Matic / scanomatic-standalone

GNU General Public License v3.0

0 stars 0 forks source link

Validate feature extraction #144

Closed e-larsson closed 2 years ago

e-larsson commented 2 years ago

Part of #141

[x] Feature extraction can be run and the results can be opened in QC.
[x] Improved readability of the submit buttons

Part of the task was to validate that the results are the same as in the old project, and they are not. They seem to be different right from the initial project compilation step so not really sure it is due to the feature extraction.

Here is a summary of what I found

2 - Compile Project

With local fixture

Produces

*.project.compilation
*.project.compilation.instructions

Differences

`*.project.compilation`

Skipped index 0 image
Grayscale has slight differences in coordinates and decimals in values
Orientation marks slightly different
FixturePlateModel coordinates slightly different

Errors

2022-01-28 08:21:55 -- ERROR **Compile Effector** Could not output analysis to file /somprojects/PayamMartin19Oct16Scanner2/PayamMartin19Oct16Scanner2_0000_30.2968.tiff

3 - Analysis

With dynamic positioning

Produces

grid_plate___*.npy
grid_size___*.npy
time_data.npy
image_*_data.npy

Differences

`grid_plate___*.npy`

Same shape but slightly different values by single digits

`image_*_data.npy`

The file with the last index is missing, due to missing index 0 in compilation probably
The numbers are different by about factor 2

`time_data.npy`

Shape is off by one, first element is missing, due to missing index 0 in compilation probably

4 - Feature extraction

Run with overwriting QC

Produces

curves_raw.npy
curves_smooth.npy
normalized_phenotypes.npy
phenotype_params.json
phenotypes.Absolute.plate_*.csv
phenotypes.extraction.instructions
phenotypes_filter.npy
phenotypes_meta_vector_raw.npy
phenotypes_raw.npy
phenotypes_reference_offsets.npy
phenotype_vectors_raw.npy
phenotype_times.npy

Differences

`curves_raw.npy` and `curves_smooth.npy`

Shape is off by one, probably same issue as earlier
Numbers are off by about factor 2

`phenotypes.Absolute.plate_*.csv`

5 more columns in new version
Numbers are off

`phenotypes_filter.npy`

The filters are different on a few (18 out of 1536) positions for one of the plates

`phenotypes_raw.npy`

New format so difficult to compare

`phenotype_vectors_raw.npy`

Old one is empty

`phenotype_times.npy`

First element is missing

local-minimum commented 2 years ago

*.project.compilation: Skipped index 0 image

Please make an issue out of this and place it in modernize SoM project and top priority.

local-minimum commented 2 years ago

Grayscale has slight differences in coordinates and decimals in values Orientation marks slightly different FixturePlateModel coordinates slightly different

Several detection methods are not deterministic so slight differences are expected, but if one plots the grayscales next to each other they ought to be extremely similar in shape and more or less lie on top of each other.

The original image is something like 4800x6000 pixels and the coordinates errors for orientation marks and fixture plate model should as a relative error be very small.

local-minimum commented 2 years ago

2022-01-28 08:21:55 -- ERROR Compile Effector Could not output analysis to file /somprojects/PayamMartin19Oct16Scanner2/PayamMartin19Oct16Scanner2_0000_30.2968.tiff

This is really weird error, I suppose it output the result of the compilation somewhere? Probably make an issue for this because it would be very bad if it attempts to overwrite images.

local-minimum commented 2 years ago

image_*_data.npy

The file with the last index is missing, due to missing index 0 in compilation probably
The numbers are different by about factor 2

I agree that it's probably the missing image 0, so if you compare e.g. old index 3 with new index 2 are the numbers still that off?

local-minimum commented 2 years ago

phenotypes_filter.npy
Old one can't be opened

@skymandr this shouldn't be created anymore should it?

skymandr commented 2 years ago

phenotypes_filter.npy
Old one can't be opened
@skymandr this shouldn't be created anymore should it?

The one I fixed and deprecated was phenotype_params.npy. Don't think this one came up as problematic in QC. But yeah, looks like it might need conversion.

skymandr commented 2 years ago

phenotypes_filter.npy
Old one can't be opened
@skymandr this shouldn't be created anymore should it?
The one I fixed and deprecated was phenotype_params.npy. Don't think this one came up as problematic in QC. But yeah, looks like it might need conversion.

I can at least load it manually:

In [3]: np.load('../Projects/PM2/PayamMartin19Oct16Scanner2/analysis/phenotypes_filter.npy', allow_pickle=True)
Out[3]: 
array([{<Phenotypes.InitialValue: 12>: array([[0, 0, 0, ..., 0, 0, 0],
...