Feature for storing not only 'numpy arrays' but also file format required for training

[graykode]

Many deep learning tasks require file as well as numpy arrays to evaluate the trained model. These formats are applicable :

• .txt
• .jpg
• .png ....

Therefore, it is implemented so that it can be used as above through data saver :

data_config = DataConfig(
        endpoint="127.0.0.1:9000",
        access_key="minio",
        secret_key="miniosecretkey",
        dataset_name="mnist",
        attributes=[
            ("input_ids", "int64", (384)),
       ],
    )
data_saver = DataSaver(config=data_config, refresh=True)

# general data saver for saving numpy array
data_saver({
      "input_ids": batch[0],
}, filetype=False)
data_saver.disconnect()

# file data saver
data_saver({
      "key": `file path`,
}, filetype=True)
data_saver.disconnect()

ToDo

• Add local cache mapper for filetype dataset.