Mappings implementation

[boergens]

I don't understand super well yet what we would need them for.. perhaps you can explain next time we talk :-)

Log Time

[tmbo]

From @jfrohnhofen:

Current state of the implementation:

• mappings are expected as JSONs (containing an array of arrays of integers as equivalence classes) in a mapping subfolder within a data layer folder
• they are detected and „normalized“ during dataset import and the „normalized“ version is saved in a target subfolder
• „normalization“ currently only consists of parsing and pretty-printing the mapping
• the paths to the mapping files are saved to the datasource.json / the DB
• all mappings associated with a layer can be selected / activated on the client via the console
• the client will then request the mapping and the server will deliver the gzipped JSON from its HD

Manually adding / manipulating mappings (for now):

• mappings can easily be updated by editing / replacing the JSON file in the target folder
• additional mappings can be added by creating a new valid JSON file (preferably in the target folder) and adding the mapping entry in the datasource.json
• on server restart, the datasource.json will then update the database content and the mapping will then be available to the client

Todos:

• [server] we should move mappings into the datasets target folder, since this will allow for a cleaner re-import of datasets (issue already created)
• [server] allow adding mappings via web interface or allow reimport of only mappings?
• [client] build proper ui

Switching to multilayer mappings:

• as soon as we want to support multi-stage-mappings, we need a way to „merge“ mappings (i.e. create a single mapping that acts as if two ore more mappings were applied consecutively)
• this could be done on the server (on import) or the client (on selecting the mapping)
• Georg and I discussed this some time ago and assumed the most common use case to be: Manuel creates a single large mapping and many other people then create relatively small mappings to group and combine supervoxels created by Manuel
• if our assumption is valid, we should do the merge on the client, since a merged mapping will be at least as large as the largest merged mapping
• so if we have Manuels mapping (100MB) and 100 tracer mappings (1MB each), we either need to load 100x100MB (server merge) or 100MB + 100x1MB (client merge)
• however, to make the merge on the client more efficient, we suggest to normalize all mappings on import:
• consider the following two mappings A and B: A = [[1, 2, 3, 4], [5, 6, 7], [10, 11, 12], [18, 19]] B = [[2, 12], [4, 5, 10], [5, 15]]
• B can be simplified in a two-step process: 1) map all IDs to some canonical identifier of that equivalence class (currently the minimum) —> B’ = [[1, 10], [1, 5, 10], [5, 15]] 2) merge equivalence classes with shared members (non-empty intersections) —> B’’ = [[1, 5, 10, 15]]
• the implementation of this normalization process is already implemented, but commented out, since it is not performant enough and leads to out-of-memory exception
• I already started on making the implementation more performant by providing custom JSON parsers and printers for mappings, which perform quite well and already deployed in production
• I also produced some Javascript prototype-level code, that can be used to merge mappings on the client. We might want to benchmark different approaches I chose for best performance

[boergens]

B can be simplified in a two-step process: 1) map all IDs to some canonical identifier of that equivalence class (currently the minimum) —> B’ = [[1, 10], [1, 5, 10], [5, 15]] 2) merge equivalence classes with shared members (non-empty intersections) —> B’’ = [[1, 5, 10, 15]]

I'm not sure that it is a common use case that B would have non empty intersections.. has this ever occurred? Perhaps we can hangout with Manuel and Alessandro to discuss

[jfrohnhofen]

If we are sure, that this is never going to happen, this would certainly simplify things. However, if in case it accidentally did happen then, the behavior would be kind of ill defined. Can we also assume/ensure, that the new mapping always references each equivalence class in the parent by its smallest member? Then, the no normalization would be needed at all. Also, we probably want a mechanism to ensure that a mapping can no longer be updated, once it serves as a parent to another mapping. Or is that a use case?

Best

Johannes

Johannes Frohnhofen

johannes.frohnhofen@scalableminds.com mailto:johannes.frohnhofen@scalableminds.com http://scm.io http://scm.io/

scalableminds UG (haftungsbeschränkt) & Co. KG Amtsgericht Potsdam, HRA 5753 Geschäftsführer: Tom Bocklisch, Tom Herold, Norman Rzepka, Thomas Werkmeister

Am 17.06.2016 um 10:27 schrieb Kevin Michael Boergens notifications@github.com:

B can be simplified in a two-step process: 1) map all IDs to some canonical identifier of that equivalence class (currently the minimum) —> B’ = [[1, 10], [1, 5, 10], [5, 15]] 2) merge equivalence classes with shared members (non-empty intersections) —> B’’ = [[1, 5, 10, 15]]

I'm not sure that it is a common use case that B would have non empty intersections.. has this ever occurred? Perhaps we can hangout with Manuel and Alessandro to discuss

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/scalableminds/oxalis/issues/1241#issuecomment-226830532, or mute the thread https://github.com/notifications/unsubscribe/ABn4WVXB_6D52fYkSGnunidDjue03TBQks5qMtjqgaJpZM4IjHgT.

[boergens]

I think with mappings it's safe to assume that they have been created by someone competent. But we should first discuss more what the actual use cases of mappings are

[jfrohnhofen]

@kevinbbb87 do you think there is anything left to be done here? as far as I remember, the discussion above was mostly about multi-level mappings. I suggest we start a new discussion as soon as this becomes relevant.