Getting error when serialize a tfds object

riibeirogabriel commented 3 years ago

What I need help with / What I was wondering I have tried serialize the response object from this code:

import tensorflow_datasets as tfds
(ds_train, ds_test), ds_info = tfds.load(
    'mnist',
    split=['train', 'test'],
    shuffle_files=True,
    as_supervised=True,
    with_info=True,
)
response = {
  "ds_train": ds_train,
  "ds_test": ds_test,
  "ds_info": ds_info,
}

But when i use the pickle.dump method to convert this dict to a binary file this exception is throwed: tensorflow.python.framework.errors_impl.InternalError: Tensorflow type 21 not convertible to numpy dtype.

Can someone help me?

Environment information (if applicable)

Operating System: Debian
Python version: 3.7
tensorflow-datasets/tfds-nightly version: stable

Conchylicultor commented 3 years ago

This is an issue with tensorflow, not TFDS. tf.Tensor and tf.data.Dataset object may not be pickable. I don't think this could be solved easily.

riibeirogabriel commented 3 years ago

@Conchylicultor You know if i can serialize using a tensorflow method? i read about the save() method, but i need a method that converts this object to a binary without write on disk.

tensorflow / datasets

Getting error when serialize a tfds object #3085