Help setting up DVID for neu3

[pgunn]

neu3 needs a labels_sv_meshes instance to function, but setting this up takes some doing; as requested I'm filing this in a github issue so anyone else who needs to do this could follow the paper trail.

I made an instance with the name labels_sv_meshes with the tarsupervoxels type and Extension=drc to match Janelia's configuration. Then with trying to use neu3 to create task protocols to split a body, it fails immediately; by the DVID logs I see:

​​2022/10/04 10:41:47 [wasptrace/vk0uctQJyE-010756] Started HEAD "/api/node/020cc:main/labels_sv_meshes/tarfile/56791722?u=pgunn&app=Neu3" from 158.106.193.218:50984
​​2022/10/04 10:41:47 [wasptrace/vk0uctQJyE-010756] Returning 400 in 145.009µs
​​ ERROR can't check existence of tarfile for label 56791722: data "labels_sv_meshes" is not synced with any labelmap instance (/api/node/020cc:main/labels_sv_meshes/tarfile/56791722).

I set up a sync:

sync_to_labels_config = {'sync': 'labels'}
sync_url = fiurl.build_instance_sync_url(dvid=server, repo=uuid, instance='labels_sv_meshes')
fihttp.dvid_post_data_ret_json(sync_url, json=sync_to_labels_config)

and then it fails the same way from the UI, with a new error in the DVID logs:

2022/10/04 10:47:02 [wasptrace/vk0uctQJyE-011582] Started HEAD "/api/node/020cc:main/labels_sv_meshes/tarfile/56791622?u=pgunn&app=Neu3" from 158.106.193.218:52018
​​ ERROR can't check existence of tarfile for label 56791622: not all supervoxel data available for label 56791622 (/api/node/020cc:main/labels_sv_meshes/tarfile/56791622).

If you can summarise what else I might need to do to get this working, that'd be great.

[stuarteberg]

@DocSavage I need help interpreting the error in the DVID log. Is that the message one would normally expect when using HEAD .../tarfile/<label> if there are supervoxel meshes missing from the instance?

In other words, are we looking at an "ordinary" failed HEAD check (indicating that Pat needs to populate the instance with some missing supervoxel meshes), or are we looking at some deeper error in DVID or the configuration?

@pgunn Did the DVID log indicate which HTTP response code would be returned? Was it 400 or something else?

[pgunn]

@stuarteberg I imagine it's that I need to populate the instance with missing supervoxel meshes (I don't know how to do that, and have nothing set up to do it - I imagine this is not something NeuTu does even though it seems to generate meshes on the fly for displaying things).

I get a 400.

[DocSavage]

@stuarteberg It's a "normal" error in that if you run a HEAD request, if there are any supervoxels for the given label that do not exist in the database, then it returns that error.

[pgunn]

Any advice on this?

[DocSavage]

@pgunn Now that you have the sync between your tarsupervoxels and labelmap instance, the original no sync error went away. So now with the HEAD request DVID is saying not all of the supervoxel files have been uploaded for that particular label. I believe if you were to simply do a GET request, you'd get the data and see some supervoxels aren't there. I believe this can even occur under normal situations depending on the approach/toolkit you are using to generate meshes if, for example, the supervoxel size falls below some threshold and isn't actually generated/stored but created on-the-fly if necessary. Since I'm not involved with that meshing system, I have to refer to @stuarteberg who would know how we're doing it -- and we'd have to know the specifics of your mesh service approach.

[pgunn]

Right now we don't have an approach to a mesh service or any software that makes meshes; I don't know what my options are and am aiming to do the easiest thing possible to make neu3 work. I expect no repo we have ever made has any meshes for its supervoxels/labels.

[DocSavage]

So this is the issue and it's out-of-scope for DVID itself. Tarsupervoxels is working I think but there's no service that monitors mutations in the labelmaps and then generates meshes, storing them into a tarsupervoxels instance, which is what neu3 expects. We'll follow up with you on how we do it here.