search

seung-lab / cloud-volume

Read and write Neuroglancer datasets programmatically.
https://twitter.com/thundercloudvol
BSD 3-Clause "New" or "Revised" License
130 stars 45 forks source link

Missing IDs and segmentation faults with 2.1.1 #404

Closed schlegelp closed 3 years ago

schlegelp commented 4 years ago

Hi. I'm not entirely sure what's happening here but I ran into some strange issues fetching flywire segmentation IDs today. After some head scratching I installed an older version of cloud volume and it worked again:

With 2.1.1

>>> from cloudvolume import CloudVolume
>>> vol = CloudVolume('graphene://https://prodv1.flywire-daf.com/segmentation/1.0/fly_v31', use_https=True, progress=False)
>>> vol[35500: 35505, 19800: 19805, 5278][: , :, 0, 0]                                                                   
VolumeCutout([[0, 0, 0, 0, 0],
              [0, 0, 0, 0, 0],
              [0, 0, 0, 0, 0],
              [0, 0, 0, 0, 0],
              [0, 0, 0, 0, 0]], dtype=uint64)

With 1.20.1

>>> from cloudvolume import CloudVolume
>>> vol = CloudVolume('graphene://https://prodv1.flywire-daf.com/segmentation/1.0/fly_v31', use_https=True, progress=False)
>>> vol[35500: 35505, 19800: 19805, 5278][:, :, 0, 0]
VolumeCutout([[80365985605081915, 80365985605081915, 80365985605081915,
               80365985605064481, 80365985605064481],
              [80365985605081915, 80365985605081915, 80365985605081915,
               80365985605077988, 80365985605064481],
              [80365985605076586, 80365985605077988, 80365985605077988,
               80365985605077988, 80365985605064481],
              [80365985605076586, 80365985605076586, 80365985605077988,
               80365985605077988, 80365985605077988],
              [80365985605076586, 80365985605076586, 80365985605077988,
               80365985605077988, 80365985605077988]], dtype=uint64)

Not sure if related but with 2.1.1 I also got frequent segmentation faults when fetching the cutout.

william-silversmith commented 4 years ago

I've seen the segmentation fault issue myself from time to time, but I'm never able to reproduce it when I try to debug it. If there's a segfault it's most likely in the compressed segmentation library (https://github.com/seung-lab/compressedseg). I tried running the test suite through valgrind with leak detection and didn't see anything but maybe there are weird inputs patters or malformed files? Is there a particular download that almost always causes a segfault for you? What OS/Python version are you using?

schlegelp commented 4 years ago

Thanks for looking into this Will! Above example code caused segfaults in maybe 1 out of 4 tries with cloud-volume 2.1.1. I'm running Python 3.7.5 on OSX Mojave (10.14.16). I haven't tried much on the segfault issue (not even a simple reboot) but thought I'd mention it in case it was connected to the other issue.

schlegelp commented 4 years ago

Is this helpful?

>>> import faulthandler
>>> faulthandler.enable()
>>> _ = vol[30500: 35505, 12800: 19805, 5278][: , :, 0, 0]                                                                                                            
Fatal Python error: Segmentation fault

Thread 0x000070001819d000 (most recent call first):
  File "/Users/philipps/.local/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1839 in recv_into
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 313 in recv_into
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 328 in recv_into
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/socket.py", line 589 in readinto
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 267 in _read_status
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 306 in begin
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 1344 in getresponse
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421 in _make_request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677 in urlopen
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/adapters.py", line 449 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 646 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 533 in request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/auth/transport/requests.py", line 454 in request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 150 in wait_and_retry
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/_helpers.py", line 136 in http_request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/download.py", line 262 in consume
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 834 in _do_download
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 951 in download_to_file
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1105 in download_as_string
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/interfaces.py", line 391 in get_file
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 167 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 192 in get
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 251 in download_chunk
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 284 in process
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 23 in realupdatefn
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 180 in _consume_queue_execution
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 153 in _consume_queue
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x0000700017c9a000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 105 in do_poll
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 43 in _retry_on_intr
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 107 in poll_wait_for_socket
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 146 in wait_for_read
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 325 in recv_into
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/socket.py", line 589 in readinto
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 267 in _read_status
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 306 in begin
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 1344 in getresponse
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421 in _make_request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677 in urlopen
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/adapters.py", line 449 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 646 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 533 in request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/auth/transport/requests.py", line 454 in request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 150 in wait_and_retry
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/_helpers.py", line 136 in http_request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/download.py", line 262 in consume
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 834 in _do_download
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 951 in download_to_file
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1105 in download_as_string
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/interfaces.py", line 391 in get_file
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 167 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 192 in get
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 251 in download_chunk
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 284 in process
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 23 in realupdatefn
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 180 in _consume_queue_execution
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 153 in _consume_queue
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x0000700017797000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 105 in do_poll
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 43 in _retry_on_intr
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 107 in poll_wait_for_socket
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 146 in wait_for_read
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 325 in recv_into
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/socket.py", line 589 in readinto
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 267 in _read_status
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 306 in begin
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 1344 in getresponse
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421 in _make_request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677 in urlopen
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/adapters.py", line 449 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 646 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 533 in request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/auth/transport/requests.py", line 454 in request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 150 in wait_and_retry
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/_helpers.py", line 136 in http_request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/download.py", line 262 in consume
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 834 in _do_download
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 951 in download_to_file
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1105 in download_as_string
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/interfaces.py", line 391 in get_file
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 167 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 192 in get
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 251 in download_chunk
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 284 in process
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 23 in realupdatefn
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 180 in _consume_queue_execution
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 153 in _consume_queue
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x0000700017294000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 105 in do_poll
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 43 in _retry_on_intr
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 107 in poll_wait_for_socket
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 146 in wait_for_read
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 325 in recv_into
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/socket.py", line 589 in readinto
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 267 in _read_status
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 306 in begin
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 1344 in getresponse
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421 in _make_request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677 in urlopen
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/adapters.py", line 449 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 646 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 533 in request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/auth/transport/requests.py", line 454 in request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 150 in wait_and_retry
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/_helpers.py", line 136 in http_request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/download.py", line 262 in consume
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 834 in _do_download
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 951 in download_to_file
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1105 in download_as_string
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/interfaces.py", line 391 in get_file
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 167 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 192 in get
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 251 in download_chunk
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 284 in process
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 23 in realupdatefn
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 180 in _consume_queue_execution
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 153 in _consume_queue
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x0000700016d91000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 105 in do_poll
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 43 in _retry_on_intr
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 107 in poll_wait_for_socket
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 146 in wait_for_read
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 325 in recv_into
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/socket.py", line 589 in readinto
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 267 in _read_status
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 306 in begin
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 1344 in getresponse
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421 in _make_request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677 in urlopen
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/adapters.py", line 449 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 646 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 533 in request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/auth/transport/requests.py", line 454 in request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 150 in wait_and_retry
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/_helpers.py", line 136 in http_request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/download.py", line 262 in consume
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 834 in _do_download
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 951 in download_to_file
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1105 in download_as_string
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/interfaces.py", line 391 in get_file
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 167 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 192 in get
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 251 in download_chunk
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 284 in process
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 23 in realupdatefn
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 180 in _consume_queue_execution
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 153 in _consume_queue
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x000070001688e000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 105 in do_poll
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 43 in _retry_on_intr
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 107 in poll_wait_for_socket
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 146 in wait_for_read
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 325 in recv_into
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/socket.py", line 589 in readinto
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 267 in _read_status
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 306 in begin
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 1344 in getresponse
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421 in _make_request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677 in urlopen
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/adapters.py", line 449 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 646 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 533 in request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/auth/transport/requests.py", line 454 in request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 150 in wait_and_retry
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/_helpers.py", line 136 in http_request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/download.py", line 262 in consume
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 834 in _do_download
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 951 in download_to_file
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1105 in download_as_string
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/interfaces.py", line 391 in get_file
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 167 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 192 in get
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 251 in download_chunk
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 284 in process
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 23 in realupdatefn
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 180 in _consume_queue_execution
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 153 in _consume_queue
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x000070001638b000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 105 in do_poll
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 43 in _retry_on_intr
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 107 in poll_wait_for_socket
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 146 in wait_for_read
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 325 in recv_into
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/socket.py", line 589 in readinto
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 267 in _read_status
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 306 in begin
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 1344 in getresponse
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421 in _make_request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677 in urlopen
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/adapters.py", line 449 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 646 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 533 in request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/auth/transport/requests.py", line 454 in request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 150 in wait_and_retry
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/_helpers.py", line 136 in http_request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/download.py", line 262 in consume
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 834 in _do_download
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 951 in download_to_file
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1105 in download_as_string
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/interfaces.py", line 391 in get_file
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 167 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 192 in get
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 251 in download_chunk
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 284 in process
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 23 in realupdatefn
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 180 in _consume_queue_execution
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 153 in _consume_queue
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x0000700015e88000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 105 in do_poll
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 43 in _retry_on_intr
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 107 in poll_wait_for_socket
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 146 in wait_for_read
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 325 in recv_into
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/socket.py", line 589 in readinto
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 267 in _read_status
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 306 in begin
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 1344 in getresponse
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421 in _make_request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677 in urlopen
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/adapters.py", line 449 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 646 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 533 in request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/auth/transport/requests.py", line 454 in request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 150 in wait_and_retry
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/_helpers.py", line 136 in http_request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/download.py", line 262 in consume
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 834 in _do_download
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 951 in download_to_file
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1105 in download_as_string
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/interfaces.py", line 391 in get_file
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 167 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 192 in get
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 251 in download_chunk
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 284 in process
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 23 in realupdatefn
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 180 in _consume_queue_execution
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 153 in _consume_queue
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x0000700015985000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 105 in do_poll
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 43 in _retry_on_intr
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 107 in poll_wait_for_socket
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 146 in wait_for_read
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 325 in recv_into
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/socket.py", line 589 in readinto
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 267 in _read_status
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 306 in begin
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 1344 in getresponse
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421 in _make_request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677 in urlopen
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/adapters.py", line 449 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 646 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 533 in request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/auth/transport/requests.py", line 454 in request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 150 in wait_and_retry
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/_helpers.py", line 136 in http_request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/download.py", line 262 in consume
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 834 in _do_download
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 951 in download_to_file
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1105 in download_as_string
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/interfaces.py", line 391 in get_file
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 167 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 192 in get
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 251 in download_chunk
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 284 in process
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 23 in realupdatefn
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 180 in _consume_queue_execution
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 153 in _consume_queue
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x0000700015482000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 105 in do_poll
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 43 in _retry_on_intr
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 107 in poll_wait_for_socket
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 146 in wait_for_read
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 325 in recv_into
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/socket.py", line 589 in readinto
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 267 in _read_status
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 306 in begin
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 1344 in getresponse
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421 in _make_request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677 in urlopen
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/adapters.py", line 449 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 646 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 533 in request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/auth/transport/requests.py", line 454 in request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 150 in wait_and_retry
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/_helpers.py", line 136 in http_request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/download.py", line 262 in consume
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 834 in _do_download
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 951 in download_to_file
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1105 in download_as_string
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/interfaces.py", line 391 in get_file
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 167 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 192 in get
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 251 in download_chunk
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 284 in process
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 23 in realupdatefn
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 180 in _consume_queue_execution
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 153 in _consume_queue
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x0000700014f7f000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 105 in do_poll
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 43 in _retry_on_intr
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 107 in poll_wait_for_socket
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 146 in wait_for_read
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 325 in recv_into
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/socket.py", line 589 in readinto
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 267 in _read_status
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 306 in begin
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 1344 in getresponse
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421 in _make_request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677 in urlopen
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/adapters.py", line 449 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 646 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 533 in request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/auth/transport/requests.py", line 454 in request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 150 in wait_and_retry
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/_helpers.py", line 136 in http_request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/download.py", line 262 in consume
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 834 in _do_download
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 951 in download_to_file
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1105 in download_as_string
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/interfaces.py", line 391 in get_file
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 167 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 192 in get
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 251 in download_chunk
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 284 in process
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 23 in realupdatefn
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 180 in _consume_queue_execution
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 153 in _consume_queue
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Current thread 0x0000700014a7c000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/chunks.py", line 180 in decode_compressed_segmentation_c_ext
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/chunks.py", line 175 in decode_compressed_segmentation
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/chunks.py", line 76 in decode
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 336 in decode
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 269 in download_chunk
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 284 in process
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 23 in realupdatefn
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 180 in _consume_queue_execution
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 153 in _consume_queue
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x0000700014579000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/common.py", line 110 in shade
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 139 in process
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 286 in process
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 23 in realupdatefn
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 180 in _consume_queue_execution
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 153 in _consume_queue
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x0000700014076000 (most recent call first):
  File "/Users/philipps/.local/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1839 in recv_into
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 313 in recv_into
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/socket.py", line 589 in readinto
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 620 in _safe_read
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/response.py", line 720 in _handle_chunk
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/response.py", line 767 in read_chunked
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/response.py", line 572 in stream
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/download.py", line 216 in _write_to_stream
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/download.py", line 268 in consume
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 834 in _do_download
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 951 in download_to_file
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1105 in download_as_string
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/interfaces.py", line 391 in get_file
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 167 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 192 in get
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 251 in download_chunk
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 284 in process
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 23 in realupdatefn
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 180 in _consume_queue_execution
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 153 in _consume_queue
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x0000700013b73000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 105 in do_poll
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 43 in _retry_on_intr
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 107 in poll_wait_for_socket
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 146 in wait_for_read
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 325 in recv_into
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/socket.py", line 589 in readinto
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 267 in _read_status
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 306 in begin
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 1344 in getresponse
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421 in _make_request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677 in urlopen
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/adapters.py", line 449 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 646 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 533 in request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/auth/transport/requests.py", line 454 in request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 150 in wait_and_retry
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/_helpers.py", line 136 in http_request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/download.py", line 262 in consume
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 834 in _do_download
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 951 in download_to_file
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1105 in download_as_string
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/interfaces.py", line 391 in get_file
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 167 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 192 in get
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 251 in download_chunk
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 284 in process
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 23 in realupdatefn
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 180 in _consume_queue_execution
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 153 in _consume_queue
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x0000700013670000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 105 in do_poll
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 43 in _retry_on_intr
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 107 in poll_wait_for_socket
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 146 in wait_for_read
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 325 in recv_into
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/socket.py", line 589 in readinto
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 267 in _read_status
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 306 in begin
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 1344 in getresponse
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421 in _make_request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677 in urlopen
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/adapters.py", line 449 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 646 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 533 in request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/auth/transport/requests.py", line 454 in request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 150 in wait_and_retry
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/_helpers.py", line 136 in http_request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/download.py", line 262 in consume
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 834 in _do_download
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 951 in download_to_file
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1105 in download_as_string
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/interfaces.py", line 391 in get_file
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 167 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 192 in get
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 251 in download_chunk
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 284 in process
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 23 in realupdatefn
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 180 in _consume_queue_execution
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 153 in _consume_queue
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x000070001316d000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 105 in do_poll
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 43 in _retry_on_intr
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 107 in poll_wait_for_socket
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 146 in wait_for_read
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 325 in recv_into
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/socket.py", line 589 in readinto
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 267 in _read_status
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 306 in begin
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 1344 in getresponse
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421 in _make_request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677 in urlopen
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/adapters.py", line 449 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 646 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 533 in request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/auth/transport/requests.py", line 454 in request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 150 in wait_and_retry
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/_helpers.py", line 136 in http_request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/download.py", line 262 in consume
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 834 in _do_download
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 951 in download_to_file
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1105 in download_as_string
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/interfaces.py", line 391 in get_file
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 167 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 192 in get
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 251 in download_chunk
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 284 in process
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 23 in realupdatefn
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 180 in _consume_queue_execution
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 153 in _consume_queue
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x0000700012c6a000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 105 in do_poll
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 43 in _retry_on_intr
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 107 in poll_wait_for_socket
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 146 in wait_for_read
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 325 in recv_into
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/socket.py", line 589 in readinto
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 267 in _read_status
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 306 in begin
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 1344 in getresponse
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421 in _make_request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677 in urlopen
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/adapters.py", line 449 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 646 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 533 in request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/auth/transport/requests.py", line 454 in request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 150 in wait_and_retry
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/_helpers.py", line 136 in http_request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/download.py", line 262 in consume
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 834 in _do_download
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 951 in download_to_file
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1105 in download_as_string
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/interfaces.py", line 391 in get_file
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 167 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 192 in get
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 251 in download_chunk
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 284 in process
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 23 in realupdatefn
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 180 in _consume_queue_execution
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 153 in _consume_queue
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x0000700011ad2000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 105 in do_poll
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 43 in _retry_on_intr
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 107 in poll_wait_for_socket
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 146 in wait_for_read
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 325 in recv_into
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/socket.py", line 589 in readinto
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 267 in _read_status
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 306 in begin
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 1344 in getresponse
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421 in _make_request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677 in urlopen
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/adapters.py", line 449 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 646 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 533 in request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/auth/transport/requests.py", line 454 in request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 150 in wait_and_retry
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/_helpers.py", line 136 in http_request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/download.py", line 262 in consume
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 834 in _do_download
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 951 in download_to_file
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1105 in download_as_string
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/interfaces.py", line 391 in get_file
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 167 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 192 in get
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 251 in download_chunk
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 284 in process
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 23 in realupdatefn
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 180 in _consume_queue_execution
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 153 in _consume_queue
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007000115cf000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 105 in do_poll
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 43 in _retry_on_intr
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 107 in poll_wait_for_socket
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/util/wait.py", line 146 in wait_for_read
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 325 in recv_into
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/socket.py", line 589 in readinto
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 267 in _read_status
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 306 in begin
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/http/client.py", line 1344 in getresponse
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421 in _make_request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677 in urlopen
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/adapters.py", line 449 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 646 in send
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/sessions.py", line 533 in request
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/auth/transport/requests.py", line 454 in request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 150 in wait_and_retry
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/_helpers.py", line 136 in http_request
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/resumable_media/requests/download.py", line 262 in consume
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 834 in _do_download
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 951 in download_to_file
  File "/Users/philipps/.local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1105 in download_as_string
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/interfaces.py", line 391 in get_file
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 167 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 192 in get
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 251 in download_chunk
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 284 in process
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 23 in realupdatefn
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 180 in _consume_queue_execution
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 153 in _consume_queue
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x0000700010bc9000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/prompt_toolkit/key_binding/key_processor.py", line 397 in wait
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/prompt_toolkit/eventloop/utils.py", line 35 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x000070000fcc0000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/prompt_toolkit/key_binding/key_processor.py", line 397 in wait
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/prompt_toolkit/eventloop/utils.py", line 35 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 870 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x0000700011fd5000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 300 in wait
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 552 in wait
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/tqdm/_monitor.py", line 69 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x000070000f7bd000 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 296 in wait
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 552 in wait
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/IPython/core/history.py", line 829 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/IPython/core/history.py", line 58 in needs_sqlite
  File "</Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/decorator.py:decorator-gen-24>", line 2 in run
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x000000010b93b5c0 (most recent call first):
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 228 in wait
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 257 in __exit__
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 30 in schedule_threaded_jobs
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/scheduler.py", line 90 in schedule_jobs
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 305 in download_chunks_threaded
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 145 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/image/__init__.py", line 163 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/frontends/precomputed.py", line 571 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/frontends/graphene.py", line 192 in download
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/frontends/graphene.py", line 242 in __getitem__
  File "<ipython-input-13-e4457968cbac>", line 1 in <module>
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3331 in run_code
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3254 in run_ast_nodes
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3063 in run_cell_async
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/IPython/core/async_helpers.py", line 68 in _pseudo_sync_runner
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2886 in _run_cell
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2858 in run_cell
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/IPython/terminal/interactiveshell.py", line 554 in interact
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/IPython/terminal/interactiveshell.py", line 563 in mainloop
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/IPython/terminal/ipapp.py", line 356 in start
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/traitlets/config/application.py", line 664 in launch_instance
  File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/IPython/__init__.py", line 126 in start_ipython
  File "/Users/philipps/.pyenv/vers
william-silversmith commented 4 years ago

Interesting, I'm on MacOS Catalina and I'm unable to reproduce. The trace confirms my suspicious:

File "/Users/philipps/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/chunks.py", line 180 in decode_compressed_segmentation_c_ext

I'm new to valgrind, but would you be able to try running your script through valgrind --leak-check=full and showing me the results? That might point out the exact line that is causing the issue in the binary.

william-silversmith commented 4 years ago

I'm running it myself and the output is prodigious... if it segfaults try filtering looking for the strings "compressed_segmentation" and "compress_segmentation".

schlegelp commented 4 years ago

Hmm. Tried installing valgrind using homebrew but no luck:

configure: error: Valgrind works on Darwin 10.x, 11.x, 12.x, 13.x, 14.x, 15.x, 16.x and 17.x (Mac OS X 10.6/7/8/9/10/11 and macOS 10.12/13)

READ THIS: https://docs.brew.sh/Troubleshooting

Pointers welcome!

william-silversmith commented 4 years ago

I ran into the same problem. I ended up installing 3.16.1 from source here:

https://sourceware.org/pub/valgrind/valgrind-3.16.1.tar.bz2

./configure
make install

If you're not already familiar with it, it's essentially a C virtual machine that executes your binaries in a simulated environment. You can detect memory leaks and profile code by counting instructions. The simulator makes execution very slow though fyi.

schlegelp commented 4 years ago

Afraid the same issue:

$ ./configure
Philipps-MacBook-Pro-2:valgrind-3.16.1 philipps$ ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... ./install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether to enable maintainer-specific portions of Makefiles... no
checking whether ln -s works... yes
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking whether make supports the include directive... yes (GNU style)
checking dependency style of gcc... gcc3
checking how to run the C preprocessor... gcc -E
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking dependency style of g++... gcc3
checking for ranlib... ranlib
checking for gcc-ranlib... no
checking for a sed that does not truncate output... /usr/bin/sed
checking for ar... /usr/bin/ar
checking for gcc-ar... no
checking for perl... /usr/bin/perl
checking for gdb... /no/gdb/was/found/at/configure/time
checking dependency style of gcc... gcc3
checking for diff -u... yes
checking for gcc option to accept ISO C99... none needed
checking for a supported version of gcc... Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/c++/4.2.1
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/c++/4.2.1
ok (Apple LLVM version 10.0.1)
checking build system type... x86_64-apple-darwin18.7.0
checking host system type... x86_64-apple-darwin18.7.0
checking for a supported CPU... ok (x86_64)
checking for a 64-bit only build... no
checking for a 32-bit only build... no
checking for a supported OS... ok (darwin18.7.0)
checking for the kernel version... unsupported (18.7.0)
configure: error: Valgrind works on Darwin 10.x, 11.x, 12.x, 13.x, 14.x, 15.x, 16.x and 17.x (Mac OS X 10.6/7/8/9/10/11 and macOS 10.12/13)
william-silversmith commented 4 years ago

Jeez, sorry I guess I didn't remember what I actually did then. Maybe try using this guy's version? https://github.com/LouisBrunner/valgrind-macos

schlegelp commented 4 years ago

Thanks! After a bit of jousting with stupid Xcode I got valgrind installed. I'm still doing something wrong though:

$ valgrind --leak-check=full python3 cv_segfault_test.py 
==97765== Memcheck, a memory error detector
==97765== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==97765== Using Valgrind-3.17.0.GIT and LibVEX; rerun with -h for copyright info
==97765== Command: /Users/philipps/.pyenv/shims/python3 cv_segfault_test.py
==97765== 
Segmentation fault: 11

That does not look like what we are expecting, does?

william-silversmith commented 4 years ago

That's weird there should be significantly more output. You can try confirming by running it against a hello world c or cpp program.

On Sat, Sep 12, 2020, 4:47 AM Philipp Schlegel notifications@github.com wrote:

Thanks! After a bit of jousting with stupid Xcode I got valgrind installed. I'm still doing something wrong though:

$ valgrind --leak-check=full python3 cv_segfault_test.py ==97765== Memcheck, a memory error detector ==97765== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.==97765== Using Valgrind-3.17.0.GIT and LibVEX; rerun with -h for copyright info==97765== Command: /Users/philipps/.pyenv/shims/python3 cv_segfault_test.py==97765== Segmentation fault: 11

That does not look like what we are expecting, does?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/seung-lab/cloud-volume/issues/404#issuecomment-691447762, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATGQSOP2TYG3UBH3YNDMH3SFMYQ7ANCNFSM4RHQKKCA .

schlegelp commented 4 years ago

I tried this:

import numpy as np
x = np.random.rand(1000, 1000)
y = x ** 2

But same valgrind output (minus the segfault).

william-silversmith commented 4 years ago

I was hoping valgrind would point out the exact error line. I had this experience when working with a pure C++ program the other day. If I was able to reproduce the error, one thing I might try is saving the image to disk in compressed format and reading it into a C++ program to decompress it and see if that still segfaulted.

schlegelp commented 4 years ago

I wished I could be of more help here but I have 0 experience with C/C++ code (be it pure C/C++ or as Python extension).

Do you think the segfault issue is related to/the cause of cloud-volume returning only zeroes? Can you reproduce this behaviour?

william-silversmith commented 4 years ago

Yea, sorry I know that's a lot to ask. Many months ago, I had been able to produce a segfault when examining some region of a sharded image, but I didn't have time to investigate. Later, when I did have time I wasn't able to reproduce it. I have also been unable to reproduce with the most recent example.

Yes, since a segmentation fault refers to reading or writing memory outside of the bounds of an officially allocated array, all kinds of memory corruption behaviors are possible. For example, a pointer could be overwritten and cause the array to look at a zeroed out block of memory. The segmentation fault is an OS safeguard. In older and simpler OSes, segmentation faults didn't exist as a program had the purview of the whole of memory. Later, safeties were introduced by dividing RAM into "segments" allocated to programs. When a program allocates new memory, that can cause the number of legal segments to be extended. When a program breaches its assigned memory segment, a segmentation fault is introduced as this is an absolute sign that there is a significant error in the program and prevents it from causing chaos in the rest of the system by for example overwriting the OS.

william-silversmith commented 4 years ago

Maybe I can try generating random arrays and compressing them and see how quickly this generates a segfault.

william-silversmith commented 4 years ago

I did find a problem where the wrapper was not handling arrays with dimensions of size zero, I don't think it's causing a segfault though.

william-silversmith commented 4 years ago

Ok, so this is really dumb, but I identified a potential memory corruption issue back when I wrote the Python wrapper and then totally forgot about it.

https://github.com/seung-lab/compressedseg/blob/master/src/compressed_segmentation.pyx#L179-L182

Let me see if I can find a more elegant solution here.

StackOverflow shows someone else with a similar issue: https://stackoverflow.com/questions/59666307/convert-c-vector-to-numpy-array-in-cython-without-copying

william-silversmith commented 3 years ago

I did some more investigation and there is some really weird behavior with the compressed segmentation algorithm when I use larger cubes. The issue I identified above turned out to not be a big deal. I'll need to look more carefully into the algorithm but I'm going to work on this as a high priority as now I can investigate the issue.

schlegelp commented 3 years ago

Thanks! I didn't explicitly mention this (probably should have) but if you compare my above examples you will notice that in the one that reliably causes segfaults I ended up requesting a much bigger volume.

william-silversmith commented 3 years ago

I haven't figured this decompression bug out quite yet, but I did find a bug in the compressor code. One of the headers in the compression specification only allows indices into the rest of the array 24-bits and under. If the size and variance of the data becomes large, this value becomes silently exceeded and blurs into the rest of the 32-bit integer. I'm adding a safeguard into my version of that code, but previously compressed data could be vulnerable.

https://github.com/seung-lab/compressedseg/commit/b87b03bf75c9ab00953868871cdfa3baba184c39#diff-69fee3b1b9fb7be9e862236657a707dcR48-R50

Of course, if Neuroglancer can read the image data, it's unrelated to this limitation.

william-silversmith commented 3 years ago

I just released CloudVolume 3.0.0 which includes the safeguards update and fixes some oddities with the way we were handling C vs Fortran order. I don't know if it will help this bug, but give it a shot and see.

schlegelp commented 3 years ago

Hi Will. Sorry for not responding earlier. Works like a charm again! IDs are back and no more segfaults. Thanks a lot!

william-silversmith commented 3 years ago

Yay! Glad that worked.