search

reiinakano / xcessiv

A web-based application for quick, scalable, and automated hyperparameter tuning and stacked ensembling in Python.
http://xcessiv.readthedocs.io
Apache License 2.0
1.27k stars 105 forks source link

How to deal with large amounts of data? #61

Open fabioperez opened 6 years ago

fabioperez commented 6 years ago

I am trying to use Xcessiv to an image classification project (together with Keras or PyTorch). By reading the walkthrough, I found that I have to pass the entire dataset as a (X, y) tuple. This is unfeasible for a large image dataset. How can I outcome this problem?

One solution that I thought about was to pass image paths as the X, and let fit load the data lazily. Is this the best approach?

PS: Thanks for creating and maintaining Xcessiv!

reiinakano commented 6 years ago

Hi @fabioperez, unfortunately I built Xcessiv without deep learning/image processing in mind. It is certainly not feasible to use Xcessiv with the large amounts of data necessary for deep learning (which are usually passed in to a neural network as batches, anyway).

Also, you give me too much credit. While I built Xcessiv, I certainly haven't had the time to maintain it in the last year, and I don't see that changing soon. :(