Next spec updates

[constantinpape]

We will need a couple of spec updates to support spotSources etc. They will also require rewriting of the old spec because we will change the table layout a bit (and some potential other changes, see below). This means the spec for all existing projects will need to be updated, and this is quite a bit of work, so I just want to do this once! So we should collect all the changes that we want to do here, I can then create some branches with prototypes and once we are sure of the changes do all the upgrades.

Here is the list of spec changes I see so far:

• mergedGrid: add optional field metadataSource to the grid views, and change the name of all sources transformed by the grid to <sourceName>_<gridName>.
  • This will require rewriting projects with a merged grid to change the source names in the regionDisplays.
  • Related: https://github.com/mobie/mobie-viewer-fiji/issues/820, https://github.com/mobie/covid-if-project/pull/6, https://github.com/mobie/mobie.github.io/pull/87
• Change the tableData definition: make the name default.tsv mandatory for tsv tables and note it in the spec, remove it from all the tables fields in the views, rename tables->additionalTables.
  • This will require rewriting projects to rename tables in the views and remove the default table
  • Q: do we want to rename tables->additionalTables in the segmentationDisplay and regionDisplay. I'd be for it because it makes more explicit that the default table should not be in there. In this case we would also need to take care of this when rewriting projects.
  • Related: https://github.com/mobie/spatial-transcriptomics-example-project/issues/7
• Add the spots source and spotDisplay to support point-like data
  • source: just has tableData
  • display: same as segmentationDisplay, additional has spotRadius, see https://github.com/mobie/spatial-transcriptomics-example-project/issues/2
  • some other related things / to keep in mind: https://github.com/mobie/spatial-transcriptomics-example-project/issues/8
• just a note for the spec: for additional segmentation tables they need to have the columns timepoint and/or image_id if these are present in the default table, see https://github.com/mobie/spatial-transcriptomics-example-project/issues/13
• Move regionTable to sources: https://github.com/mobie/mobie-viewer-fiji/issues/818, specify tableSource in regionDisplay.

There's potentially more spec changes that could be done, but I would vote to leave these be for now and only discuss after the revision:

• Changing some names:https://github.com/mobie/mobie-viewer-fiji/issues/814

[constantinpape]

Started working on PRs for this here:

• spec updates: #89
• adapting python libraries and converter: https://github.com/mobie/mobie-utils-python/pull/85

[constantinpape]

@tischi: there is another thing we discussed a while ago (can't find the issue now): that we might not need the imageDataFormats in the project level metadata any more. Do you still need it in your current implementation of MoBIE3? If not, now is a good opportunity to get rid of it.

[tischi]

I had a look at the code and it feels that this is true, however my code currently still uses this field.

Should I first try to remove everything from the code that looks at this field and then let you know whether this works? Or you anyway produce an example project branch without that field for me to test?

[constantinpape]

Can you first try to just remove the parts of the code that use it and see if it works?

[tischi]

Seems OK => you can remove it.

[constantinpape]

@tischi I have finished the spec update script now, and we have the following branches to test:

• https://github.com/mobie/sponge-fibsem-project (segmentationDisplays)
• https://github.com/mobie/plankton-fibsem-project/pull/6 (segmentationDisplays + regionDisplays)
• https://github.com/mobie/covid-if-project/pull/8 (including the new metadataSource for the mergedGrid)
• https://github.com/mobie/spatial-transcriptomics-example-project (for the spotSource / spotDisplay)

I suggest we continue as follows: once all of these projects can be opened with the new mobie we push it to mobie-beta so that I can also open these projects (and continue working on the spatial transcriptomics data). And of course let me know if you run into any issues / think something is wrong with the spec updates.

[tischi]

regionTable => regions ? That would be more consistent with spots?!

[constantinpape]

regionTable => regions ? That would be more consistent with spots?!

Makes sense, I will update it later.

[constantinpape]

regionTable => regions ? That would be more consistent with spots?!

Done and updated in all the example projects.

[constantinpape]

@tischi: I deleted a couple comments here and moved them to https://github.com/mobie/mobie-viewer-fiji/issues/818 instead since they didn't make much sense here (and I don't want to do too many purely semantic name changes now, since all of this seems to be mostly future proofing for ome.zarr tables, which we would most likely need to revisit when we integrate those in any case.)

[tischi]

@constantinpape I think I can now open everything. Maybe not every view and there will still be bugs and to-dos, but at least the default views seem to open, more-or-less. I still have a couple of FIXME in the project, where I know it cannot work right now.

Shall I continue working on those or do you want to try out stuff on MoBIE-beta already now? In fact, how would we do this? Maybe I could change the main entry point in the Fiji UI such that it loads the mobie3 branch by default?

@Plugin(type = Command.class, menuPath = CommandConstants.MOBIE_PLUGIN_ROOT + "Open>Open MoBIE Project..." )
public class OpenMoBIEProjectCommand implements Command
{
    @Parameter ( label = "Project Location" )
    public String projectLocation = "https://github.com/mobie/platybrowser-datasets";

    @Override
    public void run()
    {
        MoBIESettings options = MoBIESettings.settings();
        options.gitProjectBranch( "mobie3" ); // <---- ?!

        try
        {
            new MoBIE( projectLocation, options );
        }
        catch ( IOException e )
        {
            e.printStackTrace();
        }
    }
}

[constantinpape]

Shall I continue working on those or do you want to try out stuff on MoBIE-beta already now?

It would be great if you can push it the changes to MoBIE beta already! (And of course still work on fixing the bugs ;)).

And it would indeed be helpful if we can change it for the time being that mobie3 is opened by default.

[tischi]

OK.

1. Should we send a mail to all possible MoBIE-beta users that this will be broken-ish for some time?
2. For the release, how do we deal with people with local projects (I think Karel at EMBL and also Rafael Camacho in Sweden). Will you have a python script that they can eventually run to update their projects to the latest spec? I think we have breaking changes and I do not yet have any mechanism to support different spec versions on the Java side.

[constantinpape]

1. Should we send a mail to all possible MoBIE-beta users that this will be broken-ish for some time?

Sounds good. Do you send it?

2. For the release, how do we deal with people with local projects (I think Karel at EMBL and also Rafael Camacho in Sweden). Will you have a python script that they can eventually run to update their projects to the latest spec? I think we have breaking changes and I do not yet have any mechanism to support different spec versions on the Java side.

Yep, there is a python script, which I have used already to update the projects we currently have. For Karel I have run an earlier version of this already and can do it again once we have fully updated. And Rafa should be able to run it since he has installed the python library.

[martinschorb]

I will also look into the conversion/update script. Let me know once I can try...

[tischi]

@martinschorb you can try from now on.

[constantinpape]

@martinschorb

I will also look into the conversion/update script. Let me know once I can try...

I will clean the conversion script up next week and ping you once it can be tried.

@tischi we have a new project that I have updated: https://github.com/mobie/zebrafish-lm-project

[constantinpape]

This is all implemented now.