Send POST requests to automatically move your mouse with a neural network!
(Note: The api now only generates coordinates and no longer moves the mouse for you.)
This is the Python library containing the code for creating neural networks.
The training is done in the Colaboratory notebook. pymousegan
contains the models and training pipeline for the GAN.
Example notebooks are located at python/notebooks
git clone https://github.com/jchen42703/ai_mouse_movements.git
cd python
pip install .
numpy
tensorflow
pandas
matplotlib
(0, 0)
.(1, 1)
.The model used in the current version is a BidirectionalLSTMDecoderGenerator
from an AdditiveBasicGAN
with a BidirectionalLSTMDiscriminator
(with minibatch discrimination) and BidirectionalLSTMDecoderGenerator
. The full example is located at https://github.com/jchen42703/ai_mouse_movements/python/README.md.
Here are the model summaries:
cd js
npm install .
nodemon index.js
npm install
nodemon index.js
or node index.js
to run the server on PORT=3000
.POST
request (json
) to http://localhost:3000/
, such as:{
"start": [1, 1],
"destination": [82 ,55]
}
tf.keras
to .json
pip install tensorflowjs
tensorflowjs_converter --input_format=keras model/weights.h5 model/tfjs_model
@tensorflow/tfjs
@tensorflow/tfjs-node
express
nodemon
for conveniencePOST
request to https://localhost:3000/
express
handles the POST
request and calls the prediction function loadAndPredict
.[x, y, lag]
lag
is the time in ms
that the mouse stays at that coordinate{
"coords": [
[
1,
1,
24.451885223388672
],
[
1.789207100868225,
1.6034066677093506,
23.39274024963379
],
[
2.462282180786133,
2.276571035385132,
24.84036636352539
],
[
2.7074904441833496,
2.716768264770508,
26.283510208129883
],
[
2.862687110900879,
3.18359637260437,
27.842201232910156
],
...
]
}
On average, it runs from 390ms to 430ms