search

ml5js / ml5-library

Friendly machine learning for the web! 🤖
https://ml5js.org
Other
6.41k stars 911 forks source link

Reinforcement Learning and/or NeuroEvolution #113

Open shiffman opened 6 years ago

shiffman commented 6 years ago

At some point we thinking about having reinforcement learning and/or neuro-evolution-style reinforcement learning examples in ml5. Some references:

With Genetic Algorithms and vanilla JS neural network

Deep Q Learning

MIT Self Driving Car course

shiffman commented 6 years ago

Another good reference thanks to @hardmaru!

http://paulsoulos.com/blog/2016/08/10/agent-perception-and-learning.html

hardmaru commented 6 years ago

Last year I wrote 2 articles about Evolution Strategies that might be a useful reference for beginners to intermediate users of evolutionary methods:

A Visual Guide to Evolution Strategies: http://blog.otoro.net/2017/10/29/visual-evolution-strategies/

Evolving Stable Strategies: http://blog.otoro.net/2017/11/12/evolving-stable-strategies/

Versatilus commented 6 years ago

https://github.com/openai/gym-http-api

I haven't yet taken the time to play around with it, but this looks promising! They even have bindings for JavaScript. The project is a little stale, but it might be workable.

On Thu, Apr 19, 2018 at 3:51 PM, hardmaru notifications@github.com wrote:

Last year I wrote 2 articles about Evolution Strategies that might be a useful reference for beginners to intermediate users of evolutionary methods:

A Visual Guide to Evolution Strategies: http://blog.otoro.net/2017/10/29/visual-evolution-strategies/

Evolving Stable Strategies: http://blog.otoro.net/2017/11/12/evolving-stable-strategies/

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ml5js/ml5-library/issues/113#issuecomment-382904384, or mute the thread https://github.com/notifications/unsubscribe-auth/ALpedb6vFZELtAf_lSBJQ3_pxT0kkGFPks5tqRTugaJpZM4TcAXZ .

kwichmann commented 6 years ago

I am currently fiddling with a small library for reinforcement learning, specifically for learning action value functions - Q-learning is one such algorithm. I'm hoping to support arbitrary approximation functions (including neural networks) as well as action replay and fixed q-targets.

I'll post more once I get a workable prototype up and running.

shiffman commented 6 years ago

Another interesting reference!

https://worldmodels.github.io/

hardmaru commented 6 years ago

Thanks, world models was built using p5.js, TensorFlow.js and evolution strategies.

A while back I made a p5.js demo that used neuroevolution to solve a swing-up inverted double pendulum problem:

http://otoro.net/ml/pendulum-esp-mobile/

This type of problem is very challenging for Q-learning to solve (I tried really hard when DeepRL was hyped up back then..)

shiffman commented 6 years ago

Another nice one!

http://web.sfc.keio.ac.jp/~t15704yn/falling/index.html

shiffman commented 6 years ago

Another reference for ideas and inspiration (so hard to keep up)!

https://github.com/thibo73800/metacar

dan-ryan commented 5 years ago

Neatapatic was the best JS library for Neuro-evolution but now is unmaintained so it would be nice if there was a replacement :)

dan-ryan commented 5 years ago

Here some more things that will help.

This is the Neat paper: http://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf

This explains Neat: http://blog.otoro.net/2016/05/07/backprop-neat/

The author of Neataptic explains the improvements he did: https://towardsdatascience.com/neuro-evolution-on-steroids-82bd14ddc2f6

Here are some newer papers on neuroevolution: https://medium.com/@moocaholic/2017-the-year-of-neuroevolution-30e59ae8fe18

I see both ml5js and Neataptic have the same MIT licence, so that will help.

DianaCarrilho commented 4 years ago

I'm doing a project at the moment which would require ML5.js to create a neural network that learns from reinforcement learning algorithms. Is this possible to do? :)

NullVoxPopuli commented 3 years ago

I'm doing the same. I've yet to find anyone do such a thing. Even for tensor flow - nothing. There is reimprove.js, but it seems overly complicated (implementation codewise), and no longer works with tensor flow 2. A good reinforcement learning library or set of tools should compose with tensor flow (or ml5), not wrap and replace. Wrapping and replacing prevents upgrades of the wrapped library. Generally, that's no issue with maintained libs as ml5, but every single reinforcement algorithm npm package out there seems unmaintained and broken :cry:

AidanNelson commented 3 years ago

I'm looking into options to use Unity's ML-Agents platform to train reinforcement learning agents, then run them on the web with Tensorflow.JS. I'll update this thread when I get further, but in the meantime, you can see an early prototype here.

This workflow does not allow training of reinforcement learning agents on the web (unfortunately!) but makes it easier to host and run inference on these agents using Tensorflow.JS.

Aidan