search

seung-lab / cloud-volume

Read and write Neuroglancer datasets programmatically.
https://twitter.com/thundercloudvol
BSD 3-Clause "New" or "Revised" License
131 stars 47 forks source link

How to void the cloudvolume.exceptions.EmptyVolumeException #605

Closed liuyx599 closed 9 months ago

liuyx599 commented 9 months ago

Hi, I recently generated mesh and skeleton for the segmentation results of a larger electron microscopy dataset, and I used the following commands:

Generate mesh:

igneous mesh forge  $segdir --dir mesh --mip 2 --queue $meshforge_queue --sharded
igneous execute -x $meshforge_queue
igneous mesh merge-sharded $segdir --queue $meshmerge_queue --nlod 0
igneous execute -x $meshmerge_queue

ptq status $meshforge_queue
ptq status $meshmerge_queue

Generate skeleton:

igneous skeleton forge  $segdir --mip 2 --queue $skelforge_queue --scale 4 --const 100 --sharded
igneous execute -x $skelforge_queue
igneous skeleton merge-sharded $segdir --queue $skelmerge_queue
igneous execute -x $skelmerge_queue

ptq status $skelforge_queue
ptq status $skelmerge_queue

When running the command igneous execute, the same error seems to occur in both scripts:

INFO Running SkeletonTask(cloudpath='file:///EM/autoseg',shape=[513, 513, 513],offset=[47044, 34201, 25090],mip=2,teasar_params={'scale': 4.0, 'const': 100.0, 'pdrf_exponent': 4, 'pdrf_scale': 100000, 'soma_detection_threshold': 1100.0, 'soma_acceptance_threshold': 3500.0, 'soma_invalidation_scale': 1.0, 'soma_invalidation_const': 300.0, 'max_paths': None},will_postprocess=True,info=None,object_ids=None,mask_ids=None,fix_branching=True,fix_borders=True,fix_avocados=False,fill_holes=False,dust_threshold=1000,progress=False,parallel=1,fill_missing=False,sharded=True,frag_path=None,spatial_index=True,spatial_grid_shape=[512, 512, 512],synapses=None,dust_global=False)  (id: 963ec416-ff67-4856-94e6-85644f45e71b)
ERROR SkeletonTask(cloudpath='file:///EM/autoseg',shape=[513, 513, 513],offset=[15812, 39833, 18946],mip=2,teasar_params={'scale': 4.0, 'const': 100.0, 'pdrf_exponent': 4, 'pdrf_scale': 100000, 'soma_detection_threshold': 1100.0, 'soma_acceptance_threshold': 3500.0, 'soma_invalidation_scale': 1.0, 'soma_invalidation_const': 300.0, 'max_paths': None},will_postprocess=True,info=None,object_ids=None,mask_ids=None,fix_branching=True,fix_borders=True,fix_avocados=False,fill_holes=False,dust_threshold=1000,progress=False,parallel=1,fill_missing=False,sharded=True,frag_path=None,spatial_index=True,spatial_grid_shape=[512, 512, 512],synapses=None,dust_global=False) raised Bbox([15940, 40089, 18946],[16004, 40153, 19010], dtype=int32)
 Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/taskqueue/taskqueue.py", line 375, in poll
    task.execute(*execute_args, **execute_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/igneous/tasks/skeleton.py", line 103, in execute
    all_labels = vol[ bbox.to_slices() ]
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/frontends/precomputed.py", line 551, in __getitem__
    img = self.download(requested_bbox, self.mip)
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/frontends/precomputed.py", line 731, in download
    tup = self.image.download(
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/datasource/precomputed/image/__init__.py", line 190, in download
    return rx.download_sharded(
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/datasource/precomputed/image/rx.py", line 99, in download_sharded
    img3d = decode_fn(
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/datasource/precomputed/image/rx.py", line 623, in decode
    return _decode_helper(
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/datasource/precomputed/image/rx.py", line 684, in _decode_helper
    raise EmptyVolumeException(input_bbox)
cloudvolume.exceptions.EmptyVolumeException: Bbox([15940, 40089, 18946],[16004, 40153, 19010], dtype=int32)

The staus of $skelforge_queue

Inserted: 312832
Enqueued: 312828 (100.0% left)
Completed: 4 (0.0%)
Leased: 0 (0.0% of queue)

Then, I try to access the autoseg data by CloudVolume

import cloudvolume

cv = cloudvolume.CloudVolume("file:///EM/autoseg",mip=2,cache=True")
cv.key         # '32x32x40'
cv.bounds  # Bbox([6596, 7577, 14850],[54702, 40340, 27858], dtype=int32)

cv[15940, 40089, 18946]

The same error seems to have occurred

  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/frontends/precomputed.py", line 551, in __getitem__
    img = self.download(requested_bbox, self.mip)
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/frontends/precomputed.py", line 731, in download
    tup = self.image.download(
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/datasource/precomputed/image/__init__.py", line 190, in download
    return rx.download_sharded(
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/datasource/precomputed/image/rx.py", line 99, in download_sharded
    img3d = decode_fn(
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/datasource/precomputed/image/rx.py", line 664, in decode_single_voxel
    return _decode_helper(
  File "/usr/local/lib/python3.8/dist-packages/cloudvolume/datasource/precomputed/image/rx.py", line 684, in _decode_helper
    raise EmptyVolumeException(input_bbox)
cloudvolume.exceptions.EmptyVolumeException: Bbox([15940, 40089, 18946],[16004, 40153, 19010], dtype=int32)

The cv.meta.info of the autoseg(mip=2) is 

{'chunk_sizes': [[64, 64, 64]],
   'compressed_segmentation_block_size': [8, 8, 8],
   'encoding': 'compressed_segmentation',
   'key': '32x32x40',
   'resolution': [32, 32, 40],
   'sharding': {'@type': 'neuroglancer_uint64_sharded_v1',
    'data_encoding': 'gzip',
    'hash': 'identity',
    'minishard_bits': 6,
    'minishard_index_encoding': 'gzip',
    'preshift_bits': 9,
    'shard_bits': 12},
   'size': [48106, 32763, 13008],
   'voxel_offset': [6596, 7577, 14850]},

I was observing the region Bbox([15940, 40089, 18946],[16004, 40153, 19010] on Neuroglancer and it seems to be at the edge of the entire electron microscopy dataset now. It is indeed a dark patch with no segmentation results.

In large-scale electron microscopy datasets, such as FAFB, MiCrONS, H01, etc., the vast majority of neuron segmentation results are in the middle region of the dataset, and it seems to be not uncommon to have no segmentation results in the edge region.

I'm wondering if there is another way to avoid this error, and I'm wondering if cloudvlomue, or igenous, has a relevant api parameter that defaults empty regions to 0 when accessed, which might avoid the above EmptyVolumeException.