search

graykode / matorage

Matorage is tensor(multidimensional matrix) object storage manager for deep learning framework(Pytorch, Tensorflow V2, Keras)
https://matorage.readthedocs.io
Other
73 stars 8 forks source link

show bucket list each type(dataset, model and optimizer) #13

Closed graykode closed 4 years ago

graykode commented 4 years ago

Users who use matorage must remember the metadata when saving and use it when loading. Therefore, if only the storage endpoint is known, implement a function that can display a table with some dataset, model and optimizer currently.

from matorage import StorageConfig

if __name__ == '__main__':
    config = StorageConfig(
        endpoint='127.0.0.1:9000',
        access_key='minio',
        secret_key='miniosecretkey',
    )
    print(config.get_datasets())
    print(config.get_models())
    print(config.get_optimizers())
graykode commented 4 years ago 
/home/graykode/anaconda3/bin/python /home/graykode/matorage/test.py
08/23/2020 18:25:11 - INFO - matorage.utils - PID: 20853 -  PyTorch version 1.6.0+cu101 available.
08/23/2020 18:25:11 - INFO - matorage.utils - PID: 20853 -  PyTorch Vision version 0.7.0+cu101 available.
2020-08-23 18:25:12.547576: I tensorflow_io/core/kernels/cpu_check.cc:128] Your CPU supports instructions that this TensorFlow IO binary was not compiled to use: AVX2 FMA
08/23/2020 18:25:12 - INFO - matorage.utils - PID: 20853 -  TensorFlow version 2.2.0 available.
08/23/2020 18:25:12 - INFO - matorage.utils - PID: 20853 -  TensorFlow IO version 0.13.0 available.
[{'dataset_name': 'mnist', 'additional': {'framework': 'pytorch', 'mode': 'test'}, 'compressor': {'complevel': 0, 'complib': 'zlib'}, 'attributes': [{'name': 'image', 'shape': [1, 28, 28], 'type': 'float32'}, {'name': 'target', 'shape': [1], 'type': 'int64'}]}, {'dataset_name': 'mnist', 'additional': {'framework': 'pytorch', 'mode': 'train'}, 'compressor': {'complevel': 0, 'complib': 'zlib'}, 'attributes': [{'name': 'image', 'shape': [1, 28, 28], 'type': 'float32'}, {'name': 'target', 'shape': [1], 'type': 'int64'}]}]
[{'model_name': 'mnist_example', 'additional': {'framework': 'pytorch'}, 'compressor': {'complevel': 0, 'complib': 'zlib'}, 'model': {'0ed908321466c719126a428defd10618': {'epoch': 1}, '3a721f7993454976bbf22ac057b25f91': {'epoch': 4}, 'a5761902795ce4da13384a310dc119e7': {'epoch': 5}, 'd5364bfa35962740aba054ff727c47ea': {'epoch': 3}, 'f2445042c17ad08aec665f6647c0f045': {'epoch': 2}}}]
[{'optimizer_name': 'mnist_example', 'additional': {'framework': 'pytorch'}, 'compressor': {'complevel': 0, 'complib': 'zlib'}, 'optimizer': {'938': {'framework': 'pytorch', 'param_groups': [{'lr': 0.01, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0, 'amsgrad': False, 'params': [0, 1, 2, 3, 4, 5]}]}, '1876': {'framework': 'pytorch', 'param_groups': [{'lr': 0.01, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0, 'amsgrad': False, 'params': [0, 1, 2, 3, 4, 5]}]}, '2814': {'framework': 'pytorch', 'param_groups': [{'lr': 0.01, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0, 'amsgrad': False, 'params': [0, 1, 2, 3, 4, 5]}]}, '3752': {'framework': 'pytorch', 'param_groups': [{'lr': 0.01, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0, 'amsgrad': False, 'params': [0, 1, 2, 3, 4, 5]}]}, '4690': {'framework': 'pytorch', 'param_groups': [{'lr': 0.01, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0, 'amsgrad': False, 'params': [0, 1, 2, 3, 4, 5]}]}}}]

Process finished with exit code 0