TypeError

[liuao520]

tcmalloc: large alloc 1769676800 bytes == 0x6a98000 @ 0x7f0c85b611e7 0x7f0c7f686d8a 0x7f0c7f6872bd 0x7f0c7f7c1e4e 0x7f0c7f56edb1 0x7f0c7f56f4be 0x7f0c7f556bb3 0x7f0c7f302dc5 0x50a12f 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x507be4 0x588d41 0x59fd0e 0x50d256 0x507be4 0x509900 0x50a2fd 0x50beb4 0x507be4 0x509900 0x50a2fd 0x50beb4 tcmalloc: large alloc 1769480192 bytes == 0x70956000 @ 0x7f0c85b611e7 0x7f0c7f686d8a 0x7f0c7f6872bd 0x7f0c7f7c203a 0x7f0c7f56edb1 0x7f0c7f56f4be 0x7f0c7f556bb3 0x7f0c7f302dc5 0x50a12f 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x507be4 0x588d41 0x59fd0e 0x50d256 0x507be4 0x509900 0x50a2fd 0x50beb4 0x507be4 0x509900 0x50a2fd 0x50beb4 Traceback (most recent call last): File "/root/.local/lib/python3.6/site-packages/oneflow/python/framework/job_instance.py", line 119, in PushBlob self.pushcb(ofblob.OfBlob(of_blob_ptr)) File "/root/.local/lib/python3.6/site-packages/oneflow/python/framework/input_blob_def.py", line 333, in Copy ofblob.CopyFromNdarray(copied) File "/root/.local/lib/python3.6/site-packages/oneflow/python/framework/ofblob.py", line 109, in CopyFromNdarray return self._CopyBodyFromNdarray(src_ndarray) File "/root/.local/lib/python3.6/site-packages/oneflow/python/framework/ofblob.py", line 117, in _CopyBodyFromNdarray copy_method(self.of_blobptr, src_ndarray) TypeError: Array of type 'float' required. Array of type 'unsigned byte' given

F1130 14:12:40.312253 3870 oneflow_internalPYTHON_wrap.cpp:4718] Swig::DirectorMethodException Check failure stack trace: @ 0x7f0c7fd813ad google::LogMessage::Fail() @ 0x7f0c7fd854ec google::LogMessage::SendToLog() @ 0x7f0c7fd80ed3 google::LogMessage::Flush() @ 0x7f0c7fd85f09 google::LogMessageFatal::~LogMessageFatal() @ 0x7f0c7f2b6890 (unknown) @ 0x7f0c7f60e902 oneflow::ForeignInputKernel::ForwardDataContent() @ 0x7f0c7f60ee4e oneflow::ForeignInputKernel::Forward() @ 0x7f0c7f620fa7 oneflow::Kernel::Launch() @ 0x7f0c7f326368 oneflow::Actor::AsyncLaunchKernel() @ 0x7f0c7f33976c oneflow::NormalForwardCompActor::Act() @ 0x7f0c7f3277ae oneflow::Actor::TryLogActEvent() @ 0x7f0c7f32a286 oneflow::Actor::ActUntilFail() @ 0x7f0c7f32a375 oneflow::Actor::HandlerNormal() @ 0x7f0c7f7c9301 oneflow::Thread::PollMsgChannel() @ 0x7f0c7f7c7f92 (unknown) @ 0x7f0c84443a50 (unknown) @ 0x7f0c855256db start_thread @ 0x7f0c8585e71f clone

[doombeaker]

TypeError: Array of type 'float' required. Array of type 'unsigned byte' given

请检查下看是不是数据类型给错了，也可以使用 flow.cast 算子转换 blob 的数据类型

[liuao520]

我尝试用cv2获取图像,再转换成numpy,然后用具体的函数decode转换成blob 但出现了以上的错误 代码如下 def read_image(image_dir, cfg):

img=Image.open(image_dir)
image=np.array(img)

@flow.global_function(function_config=flow.FunctionConfig())
def image_decode_job(
    images_def: tp.Numpy.Placeholder(shape=image.shape, dtype=flow.float)
)->tp.Numpy:

# convert to tensor buffer
  images_buffer = flow.tensor_to_tensor_buffer(images_def,2)
  decoded_images_buffer=flow.image_decode(images_buffer)
  # Remember to set a shape
  # convert back to tensor list
  return flow.tensor_buffer_to_tensor(
      decoded_images_buffer, dtype=flow.float, instance_shape=image.shape,
  )
# func_config = flow.FunctionConfig()
# func_config.default_data_type(flow.float)
# func_config.default_logical_view(flow.scope.mirrored_view())

decoded_images = image_decode_job(image)
# print(decoded_images.shape)
# print("000000000000000000000000000000")
return decoded_images

[liuao520]

PIL和CV2的获取图像都试过 ,都是相同的错误

[Ldpe2G]

试试将image输入image_decode_job之前转换成 float 类型

image = image.astype(np.float)

[liuao520]

麻烦 我按照您说的 整个过程都是float def read_image(image_dir, cfg):

image= Image.open(image_dir)
# image_files = [open(im, "rb") for im in images]
# image = image.read()#[imf.read() for imf in image_files]\
image=np.array(image).astype(np.float)
print(image.shape)
# img= cv2.imread(image_dir)
# image=np.array(img)#.astype(np.float32)

@flow.global_function(function_config=flow.FunctionConfig())
def image_decode_job(
    images_def: tp.Numpy.Placeholder(shape=image.shape, dtype=flow.float)
)->tp.Numpy:

# convert to tensor buffer
  images_buffer = flow.tensor_to_tensor_buffer(images_def,2)
  decoded_images_buffer=flow.image_decode(images_buffer)
  # Remember to set a shape
  # convert back to tensor list
  return flow.tensor_buffer_to_tensor(
      decoded_images_buffer, dtype=flow.float, instance_shape=image.shape,
  )
# func_config = flow.FunctionConfig()
# func_config.default_data_type(flow.float)
# func_config.default_logical_view(flow.scope.mirrored_view())
# image = image.astype(np.float)
decoded_images = image_decode_job(image)

return decoded_images

但还是报了错 区别于没加image.astype(np.float) 这次的报错是 2020-12-01 09:18:54.322733: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 (480, 640, 3) tcmalloc: large alloc 1769676800 bytes == 0x71ea000 @ 0x7f3b211db1e7 0x7f3b1ad00d8a 0x7f3b1ad012bd 0x7f3b1ae3be4e 0x7f3b1abe8db1 0x7f3b1abe94be 0x7f3b1abd0bb3 0x7f3b1a97cdc5 0x50a12f 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x507be4 0x588d41 0x59fd0e 0x50d256 0x507be4 0x509900 0x50a2fd 0x50beb4 0x507be4 0x509900 0x50a2fd 0x50beb4 tcmalloc: large alloc 1769480192 bytes == 0x710a8000 @ 0x7f3b211db1e7 0x7f3b1ad00d8a 0x7f3b1ad012bd 0x7f3b1ae3c03a 0x7f3b1abe8db1 0x7f3b1abe94be 0x7f3b1abd0bb3 0x7f3b1a97cdc5 0x50a12f 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x507be4 0x588d41 0x59fd0e 0x50d256 0x507be4 0x509900 0x50a2fd 0x50beb4 0x507be4 0x509900 0x50a2fd 0x50beb4 Traceback (most recent call last): File "/root/.local/lib/python3.6/site-packages/oneflow/python/framework/job_instance.py", line 119, in PushBlob self.pushcb(ofblob.OfBlob(of_blob_ptr)) File "/root/.local/lib/python3.6/site-packages/oneflow/python/framework/input_blob_def.py", line 333, in Copy ofblob.CopyFromNdarray(copied) File "/root/.local/lib/python3.6/site-packages/oneflow/python/framework/ofblob.py", line 109, in CopyFromNdarray return self._CopyBodyFromNdarray(src_ndarray) File "/root/.local/lib/python3.6/site-packages/oneflow/python/framework/ofblob.py", line 117, in _CopyBodyFromNdarray copy_method(self.of_blobptr, src_ndarray) TypeError: Array of type 'float' required. Array of type 'double' given

Array of type 'float' required. Array of type 'double' given

但我具体不知道是那个地方出了问题

[leaves-zwx]

试试将image输入image_decode_job之前转换成 float 类型

image = image.astype(np.float)

image = image.astype(np.float32)

np.float 实际上指的是 np.float64

[leaves-zwx]

我没太看明白这段示例代码为什么这样写。代码开始已经通过 pillow.Image.open() 函数将图片解码过了，得到 image 的 pixel tensor 后, 为什么要把它传入 global_function 里面再调用 image_decode 呢？

image_decode 里面的实现是利用 opencv 将 raw bytes 的图片数据转换成 pixel tensor，本质上跟 pillow.Image.open() 做的是一样的事情。

现在在 python 环境里面一般使用 cv2 和 pillow.Image 来进行图片解码，相关使用代码可以见示例

• https://github.com/Oneflow-Inc/oneflow/blob/c025b3b138a55a39de8e00ac5afb57dc37df73d6/oneflow/python/test/ops/image_test_util.py#L68
• https://github.com/Oneflow-Inc/oneflow/blob/c025b3b138a55a39de8e00ac5afb57dc37df73d6/oneflow/python/test/ops/image_test_util.py#L77

[MARD1NO]

你也可以在外部构造一个reader，设定一个batchsize，等读到指定batchsize，yield返回

可参考我的代码 https://github.com/MARD1NO/OneFlow_Transformer/blob/master/utils/reader.py#L87

[liuao520]

w我参考的是https://oneflow.readthedocs.io/en/master/image.html?highlight=decode#oneflow.image.decode

[MARD1NO]

我知道，API文档是我负责的，我看你代码就像是参考这个image_decode。

我想表达的是读取图片有很多种方法，OFRecord也行，numpy读进去也行（会损失一点点性能），这个算子只是解码用的，示例只是为了展示其解码的作用。