search

seung-lab / cloud-volume

Read and write Neuroglancer datasets programmatically.
https://twitter.com/thundercloudvol
BSD 3-Clause "New" or "Revised" License
134 stars 47 forks source link

ValueError: cannot reshape array of size 4028826 into shape (512,512,16,1) #632

Open bang2003er opened 2 months ago

bang2003er commented 2 months ago

I am trying to download EM data from this website.

import numpy as np
import tifffile
from cloudvolume import CloudVolume
vol = CloudVolume("s3://open-neurodata/bumbarger/bumbarger13/image", mip=0, use_https=True)
print(vol.shape)
for i in tqdm(range(vol.shape[2])):
    img = vol[:,:,i]
    print(img.shape)

But I encountered the following error

(24576, 20480, 2762, 1)
  0%|                                                                                              | 0/2762 [00:00<?, ?it/s]File Read Error: 4028826 bytes, Bbox([11264, 11264, 0],[11776, 11776, 16], dtype=np.int64, unit='vx'), 3.9_3.9_50/11264-11776_11264-11776_0-16, errors: cannot reshape array of size 4028826 into shape (512,512,16,1)
  0%|                                                                                              | 0/2762 [03:51<?, ?it/s]
Traceback (most recent call last):
  File "test.py", line 15, in <module>
    img = vol[:,:,i]
  File "/home/zhangzb/anaconda3/envs/torch/lib/python3.8/site-packages/cloudvolume/frontends/precomputed.py", line 578, in __getitem__
    img = self.download(requested_bbox, self.mip)
  File "/home/zhangzb/anaconda3/envs/torch/lib/python3.8/site-packages/cloudvolume/frontends/precomputed.py", line 777, in download
    tup = self.image.download(
  File "/home/zhangzb/anaconda3/envs/torch/lib/python3.8/site-packages/cloudvolume/datasource/precomputed/image/__init__.py", line 211, in download
    return rx.download(
  File "/home/zhangzb/anaconda3/envs/torch/lib/python3.8/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 310, in download
    download_chunks_threaded(
  File "/home/zhangzb/anaconda3/envs/torch/lib/python3.8/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 618, in download_chunks_threaded
    schedule_jobs(
  File "/home/zhangzb/anaconda3/envs/torch/lib/python3.8/site-packages/cloudvolume/scheduler.py", line 150, in schedule_jobs
    return schedule_threaded_jobs(fns, concurrency, progress, total)
  File "/home/zhangzb/anaconda3/envs/torch/lib/python3.8/site-packages/cloudvolume/scheduler.py", line 39, in schedule_threaded_jobs
    tq.put(updatefn(fn))
  File "/home/zhangzb/anaconda3/envs/torch/lib/python3.8/site-packages/cloudvolume/threaded_queue.py", line 257, in __exit__
    self.wait(progress=self.with_progress)
  File "/home/zhangzb/anaconda3/envs/torch/lib/python3.8/site-packages/cloudvolume/threaded_queue.py", line 234, in wait
    self._check_errors()
  File "/home/zhangzb/anaconda3/envs/torch/lib/python3.8/site-packages/cloudvolume/threaded_queue.py", line 191, in _check_errors
    raise err
  File "/home/zhangzb/anaconda3/envs/torch/lib/python3.8/site-packages/cloudvolume/threaded_queue.py", line 153, in _consume_queue
    self._consume_queue_execution(fn)
  File "/home/zhangzb/anaconda3/envs/torch/lib/python3.8/site-packages/cloudvolume/threaded_queue.py", line 180, in _consume_queue_execution
    fn()
  File "/home/zhangzb/anaconda3/envs/torch/lib/python3.8/site-packages/cloudvolume/scheduler.py", line 32, in realupdatefn
    res = fn()
  File "/home/zhangzb/anaconda3/envs/torch/lib/python3.8/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 573, in process
    labels, bbox = download_chunk(
  File "/home/zhangzb/anaconda3/envs/torch/lib/python3.8/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 558, in download_chunk
    img3d = decode_fn(meta, filename, content, fill_missing, mip, 
  File "/home/zhangzb/anaconda3/envs/torch/lib/python3.8/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 642, in decode
    return _decode_helper(  
  File "/home/zhangzb/anaconda3/envs/torch/lib/python3.8/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 758, in _decode_helper
    return fn(
  File "/home/zhangzb/anaconda3/envs/torch/lib/python3.8/site-packages/cloudvolume/chunks.py", line 147, in decode
    return decode_raw(filedata, shape=shape, dtype=dtype)
  File "/home/zhangzb/anaconda3/envs/torch/lib/python3.8/site-packages/cloudvolume/chunks.py", line 328, in decode_raw
    return np.frombuffer(bytestring, dtype=dtype).reshape(shape, order='F')
ValueError: cannot reshape array of size 4028826 into shape (512,512,16,1)
william-silversmith commented 2 months ago

Hi, I ran this code for a little bit and was not able to reproduce the error. Do you have the latest version of CloudVolume? What value of i do you encounter the error at?