search

tensorflow / agents

TF-Agents: A reliable, scalable and easy to use TensorFlow library for Contextual Bandits and Reinforcement Learning.
Apache License 2.0
2.79k stars 722 forks source link

Is there a javascript version of this? #420

Closed NullVoxPopuli closed 3 years ago

NullVoxPopuli commented 4 years ago

I'm making an in-browser demonstration of reinforcement learning, and have been looking for something to use. There are a lot of standalone learning libs out there, but no examples of integrating with tensorflow neural networks

jasonmayes commented 4 years ago

Hi there, I am not on the agents team (I am with TensorFlow.js), and have not used this before so this is by no means the official answer but wanted to provide some other thought starters too in the mean time in case they are of use.

If you can export this resulting model to saved model format in Python you can load SavedModel format in Node.js without any conversion which you could then talk to very easily from some web front end via web sockets or such to pipe data around as needed between client/sever. However you may need to replicate any pre/post processing layers in Node.js for anything outside of the model itself - but if you do that could actually lead to faster end to end performance too due to the JIT compiler in JS. Some reference points: https://blog.tensorflow.org/2020/05/how-hugging-face-achieved-2x-performance-boost-question-answering.html https://blog.tensorflow.org/2020/01/run-tensorflow-savedmodel-in-nodejs-directly-without-conversion.html

However if you want the model to run entirely in the browser (no server side calls other than downloading the model files) then we have a command line converter (https://github.com/tensorflow/tfjs/tree/master/tfjs-converter) that can take Python saved model format and convert it to the JSON format we need for browser based models, however this 2nd method is subject to us supporting the ops used by the Python model.

Currently the ops TFJS supports is a subset of the TF Core ops as we a younger team and there a lot of ops to attempt to replicate. If the agents model uses something we do not currently support then that would need to be modified so that it only used our supported ops, or the missing ops would need to be contributed to the TF.js project which is also open sourced on GitHub. A list of TFJS supported ops in the browser can be found here: https://js.tensorflow.org/api/latest/#Operations

Good luck though, it would be great to see more RL examples in TFJS. Do let me know how you get on, curious to know if the library itself does ever get ported to JS :-)

NullVoxPopuli commented 4 years ago

Thanks for the information! I'll take a look through the resources, but the goal of my side project is to be entirely contained within the browser -- I don't want to set up a python development environment :) I'm using the WebGL backend with tensorflow right now, and performance is pretty good:tm:

I think I'll start with looking through tf_agent -- and maybe also this q-learning blog post, and maybe try to port something for my game's ai. If I see anything that I can reasonable contribute back to TF.js, I most certainly will!

Re: porting, I'm curious if more game-oriented abstractions are desired, as agents seem to be very "move" oriented -- and in games, there are an unknown number of moves per game, and you may want the entirety of a game to be a move-set/lesson, ya know? (I'm also still learning all the nomenclature, so idk if I'm using the right words).

jasonmayes commented 4 years ago

Fair enough! In that case I shall leave this for the agents team to answer as I have not yet touched RL myself although some folk in the community are experimenting with it in JS for browser based self driving cars etc. If you do get it working though do let me know - would love to see any new creations.

tfboyd commented 3 years ago

Closing due to age and lack of recent activity. If the issue still exists, please open a new issue and reference this one.