Data relationships and properties to enforce

[vreuter]

• [ ] A trace ID (if we've gotten it to be unique) should only have supports from exactly 1 regional barcode frame
• [ ] The set of trace IDs should be a subset of [0, #(ROIs)), again assuming global uniqueness, as each trace is done within the regional spot corresponding to an initial ROI
• [ ] #135
• [ ] #134
• [x] #99
• [ ] #129
• [ ] Row count (excluding header) of*_rois.csv and frame count for the experiment should multiply to give the number of rows (excluding header) of the *_dc_rois.csv file.
• [ ] #82
• [ ] difference between each coordinate min/max should match (at least when using intensity-based spot detection) the value for the roi_image_size defined in the config file. For example, z_max - z_min should give 16 for any ROI when the ROI image size is defined in the config as something like (16, 32, 32) represented as YAML array.
• [ ] For each trace ID in the traces table, the number of rows with that trace ID should decline in multiples of the number of hybridisations / "rounds" / "frames" / "probes" used in the experiment. This is because each row corresponds to an individual FISH spot (ostensibly), each of which corresponds to a combination of three things: (field of view, regional barcode, and timepoint/frame/probe/round). From each regional barcode spot (found in a single field of view), a record / measurement will be emitted for each of the "frames" (or "rounds" / hybridisations) in the experiment. Therefore, the fundamental unit by which the row count in the traces file may change is a regional spot / barcode, corresponds to a change of row count equal to the number of imaging round in the experiment.
• [ ] Relatedly, the maximum number of rows / records for each trace ID in the traces file at the end of looptrace is the product of three values: (number of fields of view, number of regional barcodes, and number of hybridisation rounds). This is because the trace ID counting starts fresh in each field of view, so a single trace ID may be present in each position, and every regional barcode imaging round in the experiment may support a trace with this ID in every single field of view, and then for each such case, a row/record in the traces file will be generated for each of the individual FISH hybridisation rounds / frames / timepoints.
• [ ] In the traces file, the number of rows/records for a particular trace ID should monotonically decrease in the trace ID. This is because the trace ID numbering counts upwards, and as the regional spots are "used up" in building the traces, there are fewer and fewer left to use, and the higher number trace IDs will correspond to regional barcodes in fewer and fewer fields of view.
• [ ] #103
• [ ] The values for coarse drift correction shift should all be multiples of the downsampling factor in the config file.
• [ ] The values for fine drift should never exceed coarse drift downsampling factor
• [ ] Files inside zarr dataset should reflect the (t, c, z, y, x) dimensionality
• [ ] Status label of {unused, used} should bifurcate the pool of bead ROIs for each (FOV, frame) pair; see #108
• [ ] #128
• [ ] filtered files should always have line count no more than unfiltered counterpart
• [ ] nuclei labels should be in [0, 1, ...N_nuclei], since 0 = no label per cellpose.CellposeModel.eval docstring
• [ ] spot ROI size should be independent of the downsampling factor

[vreuter]

This could (and maybe should) be implemented in the post-hoc analysis repository.

[vreuter]

Note that we can use the .zattrs file in each position FOV and the .zarray file in the 0 folder subarray to validate various dimensional properties of the dataset, as well perhaps as the layout of data in terms of structure on the filesystem.

[vreuter]

% % cat ../.zattrs
{
    "multiscale": {
        "multiscales": [
            {
                "axes": [
                    "t",
                    "c",
                    "z",
                    "y",
                    "x"
                ],
                "datasets": [
                    {
                        "path": "0"
                    }
                ],
                "name": "seq_images_raw_decon_P0002.zarr.zarr",
                "version": "0.3"
            }
        ]
    }
}%

[vreuter]

% cat .zarray
{
    "chunks": [
        1,
        1,
        1,
        2044,
        2048
    ],
    "compressor": {
        "blocksize": 0,
        "clevel": 5,
        "cname": "zstd",
        "id": "blosc",
        "shuffle": 2
    },
    "dtype": "<u2",
    "fill_value": 0,
    "filters": null,
    "order": "C",
    "shape": [
        2,
        2,
        2,
        2044,
        2048
    ],
    "zarr_format": 2
}