Dataset schema changes for versioning, viewer settings

[stuarteberg]

As discussed in today's meeting, we'll be changing the backend's specification of dataset properties:

• neuroglancer settings are always specified in exactly the same way neuroglancer does it, in a JSON object named neuroglancer. We should be able to copy/paste directly from a neuroglancer window into our dataset specification.
• Datasets will have global settings and per-version settings
• Each dataset version will have a few version-specific properties, such as title, version, public, roles, neuroglancer.

@DocSavage will post an example below. I'll also post an issue on the clio_website repo for @tingzhao.

[DocSavage]

@tingzhao Here is an example of a dataset JSON object:

{
    "vnc-test": {
        "title": "vnc-test",
        "description": "VNC Test Dataset",
        "mainLayer": "grayscale",
        "neuroglancer": {
            "crossSectionScale": 100,
            "position": [
                23056.5,
                29733.5,
                41138.5
            ],
            "projectionScale": 2600,
            "dimensions": {
                "x": [
                    8e-09,
                    "m"
                ],
                "y": [
                    8e-09,
                    "m"
                ],
                "z": [
                    8e-09,
                    "m"
                ]
            },
            "layers": [{
                "blend": "additive",
                "name": "grayscale",
                "opacity": 1,
                "shader": "#uicontrol invlerp normalized(range=[182.988, 206.9835], window=[134.997, 254.97449999999998]) \\n #uicontrol int invertColormap slider(min=0, max=1, step=1, default=0) \\n #uicontrol vec3 color color(default=\"white\") \\n  float inverter(float val, int invert) {return 0.5 + ((2.0 * (-float(invert) + 0.5)) * (val - 0.5));} \\n void main() {   emitRGB(color * inverter(normalized(), invertColormap)); }",
                "shaderControls": {
                    "normalized": {
                        "range": [94,177],
                        "window": [0,254]
                    }
                },
                "source": "gs://flyem-vnc-2-26-213dba213ef26e094c16c860ae7f4be0/v3_emdata_clahe_xy/jpeg",
                "type": "image"
            }]
        },
        "versions": [
            {
                "title": "v0.1",
                "version": "dvid://05280fe90579437495affcdf5afe8065",
                "public": false,
                "roles": {
                    "segmentation-v0.1": ["mergeable"]
                },
                "neuroglancer": {
                    "layers": [
                        {
                            "name": "segmentation-v0.1",
                            "type": "segmentation",
                            "source": "dvid://https://emdata5-private.janelia.org/05280fe90579437495affcdf5afe8065/segmentation"
                        }
                    ]
                }
            },
            {
                "title": "v0.2",
                "version": "dvid://2e4ac3804a5d41fbbb4adc46ecfcc7d7",
                "public": false,
                "roles": {
                    "segmentation-v0.2": ["mergeable"]
                },
                "neuroglancer": {
                    "layers": [
                        {
                            "name": "segmentation-v0.2",
                            "source": "dvid://https://emdata5-private.janelia.org/2e4ac3804a5d41fbbb4adc46ecfcc7d7/segmentation",
                            "type": "segmentation"
                        }
                    ]
                }
            }
        ]
    }
}

Typically the above dataset object would be in a list returned by GET /v2/datasets and is now in the firestore database under the vnc-test dataset id.

[DocSavage]

In commit 7b120f7, dataset object handling loosened to allow new schema while also retaining ability to return legacy dataset objects if still in firestore.

[DocSavage]

We will also be adding a tag field to each version that can serve as a version identifier that can be used for version-specific calls to clio-store like the future merge endpoint.