graykode / matorage

Matorage is tensor(multidimensional matrix) object storage manager for deep learning framework(Pytorch, Tensorflow V2, Keras)
https://matorage.readthedocs.io
Other
73 stars 8 forks source link

(feature) AWS S3 module #3

Closed graykode closed 4 years ago

graykode commented 4 years ago

Finish workflow for AWS S3 module

For an example:

import numpy as np
from matorage import *

if __name__ == '__main__':
    data_config = DataConfig(
        endpoint='s3.us-east-1.amazonaws.com',
        access_key='access_key',
        secret_key='secret_key',
        region='us-east-1',
        dataset_name="test_datasaver_s3",
        attributes=[DataAttribute("x", "float64", (2), itemsize=32)],
    )
    data_saver = DataSaver(config=data_config)
    x = np.asarray([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])
    data_saver({"x": x})
    data_saver.disconnect()