Spec description

[constantinpape]

Currently the spec is only briefly described here: https://github.com/mobie/mobie#data-storage. We should expand the description and move it to mobie.github.io; I started to work on a PR: https://github.com/mobie/mobie.github.io/pull/1.

In general, I am not really sure if we need this repository (https://github.com/mobie/mobie). I think we can move everything that's here (and that we want to keep) to https://github.com/mobie/mobie.github.io / http://mobie.github.io/ so that this becomes the default landing page for all general MoBIE things.

What do you think @tischi, @K-Meech?

[tischi]

. I think we can move everything that's here (and that we want to keep) to https://github.com/mobie/mobie.github.io / http://mobie.github.io/ so that this becomes the default landing page for all general MoBIE things.

Sounds like a great idea! And less is almost always more :-)

[K-Meech]

Yes - sounds good!!

[constantinpape]

Great! I will gradually start to move things over then.

[constantinpape]

@tischi @K-Meech Ok, I started to work on the spec description and as now is the time to change things: Should we rename tables from csv to tsv? I think its a bit confusing that we are using tab separated values, but call it csv. (I know that this goes back to my suggestion from quite some time ago, but I changed my mind about this since then :D)

[K-Meech]

Yes! I think .tsv makes sense. Although, I think that both comma-separated and tab-separated work in MoBIE? Supporting both is useful, if possible.

[constantinpape]

Yes! I think .tsv makes sense. Although, I think that both comma-separated and tab-separated work in MoBIE? Supporting both is useful, if possible.

Ok, if we support both it's even more important to have different file extensions. Do comma separated values work, @tischi?

[tischi]

It SHOULD work ;-)

• https://github.com/tischi/imagej-utils/blob/747c232ed0f873fb6a1497fef7ef9249fae11dc2/src/main/java/de/embl/cba/tables/Tables.java#L229
• https://github.com/tischi/imagej-utils/blob/747c232ed0f873fb6a1497fef7ef9249fae11dc2/src/main/java/de/embl/cba/tables/Tables.java#L205

[tischi]

In practice, I find TSV more robust, because, e.g., some file and folder and gene or sample treatment drug names sometimes contain a , but I don't think they normally contain \t.

So maybe we should recommend TSV for above reasons?

[constantinpape]

So maybe we should recommend TSV for above reasons?

:+1:

[constantinpape]

I have all schema files for the new spec now: https://github.com/mobie/mobie.github.io/tree/spec/schema. There are still some small todos, see #5. The spec description is not updated yet, I am looking into auto-generating it partly from the schema.

[constantinpape]

Ok, I merged #1 now, so the schemas are on master: https://github.com/mobie/mobie.github.io/tree/master/schema I implemented partial auto generation of the spec description from the schemas, so that this is less duplicate effort and also easier to keep in synch. This is also on master already, so the spec description also gets rendered: https://mobie.github.io/specs/mobie_spec.html

[K-Meech]

Specs look great! I think there are a few things missing though (unless I missed them somewhere!):

• valueLimits: these were the limits used on 'color by column' for numeric colouring schemes like BlueWhiteRed
• resolution3dView: this was the resolution used to display that segmentation in the 3d viewer
• showImageIn3d: equivalent of showSelectedSegmentsin3D for images

They're all part of the original 'Metadata' in imagej-utils: https://github.com/tischi/imagej-utils/blob/eeea86750ed3307f204d8753675e0eeab3e9ab5c/src/main/java/de/embl/cba/bdv/utils/sources/Metadata.java

[constantinpape]

Indeed, these are still missing.

• valueLimits: these were the limits used on 'color by column' for numeric colouring schemes like BlueWhiteRed

Ok, I will add it to segmentationDisplays.

• resolution3dView: this was the resolution used to display that segmentation in the 3d viewer

How do we specify this? As the index to the Image Pyramid or as resolution in physical units? Right now it seems to be a single double: https://github.com/tischi/imagej-utils/blob/eeea86750ed3307f204d8753675e0eeab3e9ab5c/src/main/java/de/embl/cba/bdv/utils/sources/Metadata.java#L61 Also, I guess we will need it for both imageDisplays and segmentationDisplays

• showImageIn3d: equivalent of showSelectedSegmentsin3D for images

Ok, I will add it to imageDisplays.

[K-Meech]

I think resolutiond3dView was in physical units - I believe the 3d viewer displays everything as if it is isotropic, hence only the single double? And 0 meant that it would be auto-adjusted to a good default. @tischi - do you remember?

[tischi]

I think resolutiond3dView was in physical units

Physical Units! Should we use -1 for "auto"? Or, even better, we just say, if it is missing we use an automated strategy?

[constantinpape]

Physical Units!

Ok, but shouldn't it be 3 values then? Even if you assume that it's isotropic internally, for the spec it doesn't make much sense to give a physical unit with just one value.

Should we use -1 for "auto"? Or, even better, we just say, if it is missing we use an automated strategy?

Yes, I think the latter is best.

[tischi]

Ok, but shouldn't it be 3 values then?

Makes sense to have three values.

[constantinpape]

Ok, I put it all in the spec. One minor thing, I will call it showImagesIn3d, because the imageDisplays refer to a group of images.

[constantinpape]

@tischi I need to make one more small change compared to what we discussed for selectedSegmentIds: we said this should be given as sourceName-timepoint-labelId. But we often use - as part of names, so we can't use it as a separator here. I suggest to use , instead, but let me know if you prefer some other separator.

[tischi]

, is risky, if that ever ends up in some csv file it will break everything 🌋 What about -- or _ or __?

[constantinpape]

, is risky, if that ever ends up in some csv file it will break everything

Yeah, that's also true.

What about -- or _ or __?

_ is also often part of names. Double - or _ would work, but I would prefer a single character to keep the regex a bit more readable (I am adding a regex to all name fields that is preventing the use of the separator) How about ;?

[tischi]

; is an option although sometimes this also is used as a table delimiter. In the ALMF, after long considerations of such issues they use -- But I can live with ;

[constantinpape]

@tischi @K-Meech I have added the changes we discussed here and added regexps to the schema. I will stop working on the spec for now. The schemas and the description are up-to-date. I also added a short note there on how to use the schema files for validation or example generation, which I find super useful.

@tischi I have put a project with the current version here: /g/kreshuk/pape/Work/data/mobie/example-project

[tischi]

Yo, the fun starts :-)

[tischi]

Wow, we need many classes now, but it seems to fly ✈️

[constantinpape]

@tischi I am keeping track of the changes we discussed here: https://github.com/mobie/mobie.github.io/issues/5#issuecomment-804365778 Let me know once you think you have all the changes currently necessary, then I can implement them in a bulk.