graykode
commented
4 years ago @jinserk Thank you for your interest in the project!
- can I save any objects other than tensors, i.e. a tuple of tensors, a dict or a sparse tensors? In this case how to give the attributes?: As can be seen in the existing example, tensors in the form of tuple or dict can also be stored. We do not define a new operation for sparse sensors, but we can save it. As for the tensor for sparse, we will add it to the long-term plan. Below is an example of storage in the form of a tuple :
attributes=[ ('image', 'float32', (1, 28, 28)), ('target', 'int64', (1)) ]
traindata_saver({ 'image': image, 'target': target })
- If I add some more data samples to an existing dataset (in the case that periodically data samples could be added so if I have to refresh all the datasets with the added samples), will it be okay to add to the dataset and save it? : If you simply add more data (append mode) does not matter if you save using the existing config. However, if you want to refresh your data, it is not currently implemented in code. If you want to refresh dataset (this means the same as you want to remove buckets), you should use minio web site or [mc command](https://github.com/minio/mc) from minio. (`mc rb --force --dangerous local/<bucket_name>`). I will further implement the refresh method by adding a new option on datasaver like this:
```python
traindata_saver({
'image': image,
'target': target
}, refresh=True) # I will add this argument- If I use this in a distributed training, will the each dataloader occupy the amount of dataset? For example, if I'd like to use this with PyTorch DDP, then each process will have its own DataLoader and will load the data samples during the training -- so I wonder the loaded dataset size will be multiple of the original dataset size per process or not. :I understood this question as follows: With distributed data parallel (DDP), N data is replicated as many nodes as M, and the total number of N*M data is not generated? Yes, that's right. When dataset of
torch.matorageis initialized, it goes through a logic to pre-download data corresponding to config (https://github.com/graykode/matorage/blob/master/matorage/data/data.py#L80). However, this is very inefficient, So we recommend using the network access storage (NAS) option when using DDP. No new downloads of data are available when using NAS. Therefore, N datasets can be kept intact and DDP train can be performed. - It looks like supporting only PyTorch and TensorFlow now, but how about the numpy array or matrix for scikit-learn or XGBoost? Can I store some numpy objects as well? : When the tensor in Pytorch(torch.tensor) and Tensorflow(tf.Tensor) are saved, they are converted to numpy and saved. (As with tfrecord, the tensor can be encoded in the proto-buf format, but because of its lack of universality, the most popular numpy format was used.) Therefore, if the storage format is numpy array, you can save both scikit-learn or XGBOost. But of course, when it comes to data loaders, new additional implementations are needed. See how numpy array to be save! : (https://github.com/graykode/matorage/blob/master/tests/test_datasaver.py#L247)
Best Regard Tae Hwan
jinserk
It's really fantastic! Thank you so much for sharing this project. I had a quick test with
miniodocker process and confirmed it works really well as expected. I'd like to ask a few questions about the usage:DataLoaderand will load the data samples during the training -- so I wonder the loaded dataset size will be multiple of the original dataset size per process or not.