FGLab is a machine learning dashboard, designed to make prototyping experiments easier. Experiment details and results are sent to a database, which allows analytics to be performed after their completion. The server is FGLab, and the clients are FGMachines.
FGLab tries to follow the SemVer standard whenever possible. Releases can be found here. There are 3 ways to run FGLab: Installing locally, via Docker, or hosted on Heroku.
mkdir -p <working directory>/db
.mongod --dbpath <working directory>/db
.npm install
. npm install
also runs bower install
to install additional required packages..env
file in this directory. For most installations, it should be possible to copy example.env to .env
, but it may require customisation for non-standard MongoDB ports, or setting a different port for FGLab. An alternative is to set the following environment variables:
Run node lab
(or npm start
) to start FGLab. You can now access the user interface from a browser on the current machine at http://localhost:<FGLAB_PORT>
, where <FGLAB_PORT>
is 5080 by default. For remote access, you need to be able to access the machine FGLab is running on from your remote machine via a local network or the internet. Given the default port, you would replace http://localhost:5080
with http://lan-hostname:5080
or http://public-address.com:5080
, respectively.
Please read the overview to understand how FGLab and FGMachine cooperate - both are needed in order to run experiments. Afterwards, you should set up instances of FGMachine.
To update, run npm run update
.
Start a MongoDB container and link it to the FGLab container:
sudo docker run -d --name mongodb mongo
sudo docker run -d --name fglab --link mongodb:mongo -p 5080:5080 kaixhin/fglab
Although not recommended, it is possible to adjust project schema and other parts of the database. This can be accomplished either by connecting directly to MongoDB or via a GUI such as mongo-express.
sudo docker run -d --name mongo-express --link mongodb:mongo -p 8081:8081 mongo-express
The deploy button provisions a free dyno running FGLab on Heroku, with a free 500MB MongoDB database from MongoLab.
FGLab is based on several classes of object. One begins with a project, which involves adjusting variables to achieve the desired results. In machine learning, these variables are hyperparameters, which are set for the project. In a more general setting, the variables are simply options, which may therefore include implementation-dependent details. A project will then comprise of a set of experiments derived from adjusting options.
A project is created by uploading a JSON schema. JSON is a human-readable data-interchange format that is widely used and has mature libraries available for most programming languages.
The JSON schema represents a map/associative array (without nesting), where the values are an object comprising of several fields:
int
float
bool
string
enum
enum
See mnist.json as an example schema for a project. Each schema should be uploaded with the filename corresponding to the desired name for the project e.g. mnist.json
.
Often it is hard to specify some options in advance e.g. the type or structure of the machine learning model. Sometimes code may change, which would influence the results. The string
type can be used to address changing options and versioning manually e.g. cnn.v2
.
This is stored by FGLab, and is used to construct a form which lets one choose options and submit an experiment to an available machine. The options are sent to your machine learning program via the FGMachine client. Your machine learning program then accepts the different fields via command-line options, the details of which are in the FGMachine documentation. Note that the _id
field is reserved, as this will store the experiment ID as a string
.
FGMachine will spawn your machine learning program, which should produce output files to be sent from FGMachine to FGLab. The details of this is available in the FGMachine documentation.
Grid and random search optimisers have also been implemented in FGLab, to allow searching over a range of hyperparameter space. Multiple string values are delimited by commas (,
).
An experiment is one complete training and testing run with a specific set of options. Depending on the experiment it may be impossible to control for every source of randomness, so experiments with the same set of options will still be assigned unique IDs. Experiments have a unique ID, in addition to a project ID, a machine ID, the chosen options, the current status (running/success/fail), timestamps, results, and custom data; this provides a comprehensive record of the experiment as a whole.
The experiment page contains a "Logs" window, which uses WebSockets to display the experiment's stdout
and stderr
live. There is also an editable "Notes" text box that is automatically saved (at an interval of 0.5s), displaying on both the experiment page itself and the table of experiment results.
The current format for results is documented with FGMachine.
A FGMachine client registers itself with FGLab, providing hardware details as well as an address for interaction between FGLab and the machine. A machine (FGMachine) stores its own details, as well as a list of supported projects. Before a new experiment is chosen to be run, FGLab queries all machines in order to determine a machine with the capacity to run the experiment.
Note that machines are implementation-independent, and may well store their own (large) data on experiments, for example learnt parameters and logs. As mentioned before, these can be uploaded to FGLab's database.
Examples utilising the range of abilities of FGLab/FGMachine can be found in the examples folder.
Just set up PASSWORD variable without and quotes in .env file to protect your FGLab with a password. Note: you should type in that password into password field, when prompted to. Example:
PASSWORD=friend
The API is largely undocumented due to ongoing development introducing breaking changes. Ongoing documentation is available in RAML: api.raml. The following are noted for convenience:
Submit a new experiment with a set of options
POST /api/v1/projects/{projectId}/experiment
e.g. curl -X POST -H "Content-Type: application/json" -d '{projectOptions}' http://{FGLab address}/api/v1/projects/{projectId}/experiment
If the project does not exist, returns 400 {"error": "Project ID <projectId> does not exist"}
. If the projectOptions
are invalid, returns 400 {"error": "<validation message>"}
. If no machines are available to run the job, returns 501 {"error": "No machine capacity available"}
. If the machine fails to run the experiment for some reason, returns 500 {"error": "Experiment failed to run"}
. If successful, returns 201 {"_id": "<experimentId>"}
.
Start a batch job with a list of option sets
POST /api/v1/projects/{projectId}/batch?retry={retryTimeout (optional)}
e.g. curl -X POST -H "Content-Type: application/json" -d '[{projectOptions}]' http://{FGLab address}/api/v1/projects/{projectId}/batch?retry={retryTimeout (optional)}
The optional retry
parameter specifies the maximum time in seconds to wait before trying to run a queued job again after capacity has been reached (the interval is randomly picked from a uniform distribution). If the project does not exist, returns 400 {"error": "Project ID <projectId> does not exist"}
. If any of the projectOptions
are invalid, returns 400 {"error": "<validation message>"}
for the first set of options that are wrong. If successful, returns 201 {"status": "Started"}
. Future work aims to create a proper "optimiser" object that can be queried and have its work queue adjusted appropriately (hence differentiating it from a simple batch job queue).
Register a webhook for an event
POST /api/v1/webhooks
e.g. curl -X POST -H "Content-Type: application/json" -d '{webhookOptions}' http://{FGLab address}/api/v1/webhooks
webhookOptions
expects the following options
{
"url": "<URL to POST to>",
"objects": "<object collection to listen to (currently only 'experiments')>",
"object_id": "<object ID>",
"event": "<event to listen to (currently only 'started' or 'finished')>"
}
If a valid URL is not provided, returns 400 {"error": "Invalid or empty URL"}
. If a valid object collection is not provided, returns 400 {"error": "Object is not 'experiments'"}
. If a valid event is not provided, returns 400 {"error": "Event is not 'started' or 'finished'"}
. If an object ID is not provided, returns 400 {"error": "No object ID provided"}
. If successful, returns 201 {"status": "Registered", "options": <webhookOptions>"}
. When the event occurs, the JSON data used to register the webhook is returned.