Is it possible to pass data set to parameter-less data() function (not load from the file)

[rayguang]

Hello all,

The data function is parameter-less and all the examples are loading data from disk or existing dataset. Is there anyway to pass the data set through memory variable? Is that something can be achieved via the data_args and function parameters?

For example, my data set is the output from an existing scikit learn pipeline and I would not know the details until runtime. Of course, I can dump the output to the disk and load it within the data(), but that will add a lot of I/O time considering a moderate data set size (1G+). OR equivalently, we can save the output to DB (e.g., Cassandra) and load it from DB.

Is there any better approach to pass the data set to hyperas directly in memory variable without save/load to a file/DB?

Thanks.

best_run, best_model = optim.minimize(model=create_model,
                                      data=data,
                                      algo=tpe.suggest,
                                      max_evals=5,
                                      trials=Trials()) 

[maxpumperla]

right now there isn't a good way of doing this afaict (but the community has found better hacks in some situations and might chime in with a surprise for you).

I didn't design hyperas with that use case in mind, so would have to go back and change a few things internally.

[rayguang]

thanks max! where can I find the hack from the community (searched through the internet but did not find any clue )

[maxpumperla]

I don't know if one exists in this case, just wanted to say that it always surprises me that a lot of people know my software better than me

[rayguang]

Thanks max.