search

smilesun / rlR

Deep Reinforcement Learning in R (Deep Q Learning, Policy Gradient, Actor-Critic Method, etc)
https://smilesun.github.io/rlR
Other
26 stars 4 forks source link

online learning with neural networks in R is slow #2

Open markusdumke opened 6 years ago

markusdumke commented 6 years ago

@smilesun Incremental learning with neural networks in R is very slow. I made some benchmarking experiments, mxnetis twice as fast as kerasbut still very slow. Not sure what we can do about this?

https://github.com/compstat-lmu/papers_2017_reinforcelearn_software/blob/da9659e583583d3b80d8da5d37f4bf3c876b6f8c/examples/online_learning_with_neural_networks.R#L133-L139

berndbischl commented 6 years ago

is there a possibility not to run the model update each step?

berndbischl commented 6 years ago

but to do it in "bulk" like very few iterations? together with replay?

smilesun commented 6 years ago

Yes, with a smart replay we could use mlr Bernd Bischl notifications@github.com schrieb am Fr. 1. Dez. 2017 um 12:28:

but to do it in "buld" like very few iterations? together with replay?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/compstat-lmu/papers_2017_reinforcelearn_software/issues/2#issuecomment-348471989, or mute the thread https://github.com/notifications/unsubscribe-auth/AAi1Rjvm6U-NroBhFfyg0JlviLUYpk9Jks5s7-LsgaJpZM4QyNw8 .

markusdumke commented 6 years ago

@smilesun Do you actually know any paper, where they do this? I think in all the Sutton book examples updates are made in each step. But it is certainly possible. Though I think it would still be good to have the usual incremental learning with one or more training examples at each step, because that's what people are used to

berndbischl commented 6 years ago

Though I think it would still be good to have the usual incremental learning with one or more training examples at each step, because that's what people are used to

i am not suggesting to throw this out. but if an efficient alternative is provided by us, then it doesnt hurt that much anymore if the 1-step update is so slow (for now).

markusdumke commented 6 years ago

yes of course, maybe we can have a parameter, after how many steps/episodes to update, which the user can set

smilesun commented 6 years ago

Maybe we could wrap around the c api for tensor flow

smilesun commented 6 years ago

currently I think the speed is ok, I only have the error below. 

/home/sunxd/anaconda2/lib/python2.7/site-packages/keras/callbacks.py:93: UserWarning: Method on_batch_begin() is slow compared to the batch update (0.127413). Check your callbacks.
  % delta_t_median)