search

deepsphere / deepsphere-cosmo-tf1

A spherical convolutional neural network for cosmology (TFv1).
https://arxiv.org/abs/1810.12186
MIT License
131 stars 30 forks source link

Is there a way to use queue to read data? #35

Closed q1anzhen closed 4 years ago

q1anzhen commented 4 years ago

Hi, thanks for developing this great tool. It seems to be a good way to deal with spherical data. Now, i am trying to implement this package in my work and i basically realized the training a model using the demo method. I noticed that in the deme method, all the data will be read into the memory before training. But when I use a larger data set, it will inevitably have some insufficient memory problems. I am wondering if you have designed an interface that uses queue data, like reading data from some TFRecord files and using a iterator something. That seems to be used in many deep learning project with TensorFlow. Or do you have other ways to read larger datasets? Could you give me some hints?

Thanks a lot. :)

mdeff commented 4 years ago

Thanks for your kind words. We indeed always read from memory (because our machine had enough for our data) in our experiments. The easiest way if you already have a codebase is maybe to only reuse the NN layers and graph creation. Otherwise, if you can use PyTorch instead, our implementation has a dataloader that will lazily load batches when needed.

q1anzhen commented 4 years ago

Thanks for your kind words. We indeed always read from memory (because our machine had enough for our data) in our experiments. The easiest way if you already have a codebase is maybe to only reuse the NN layers and graph creation. Otherwise, if you can use PyTorch instead, our implementation has a dataloader that will lazily load batches when needed.

thx, i will look into it! :)