search

keras-team / keras

Deep Learning for humans
http://keras.io/
Apache License 2.0
61.71k stars 19.43k forks source link

Visualization tools #254

Closed fchollet closed 7 years ago

fchollet commented 9 years ago

I've started experimenting a little bit with experiment visualization.

Here's a demo capable of monitoring a Keras experiment on a live-updating webpage: https://github.com/fchollet/hualos

What are your thoughts on what exactly we should be building? What do we need?

Here are some initial ideas:

fchollet commented 9 years ago

Being able to interact with the experiment would also be a must. We should be building the Bloomberg terminal of Deep Learning.

tleeuwenburg commented 9 years ago

Sounds amazing, Francois. It's 8pm Sunday here. I'll be taking this for a spin tomorrow during the day! Thanks so much for all your amazing work.

On 21 June 2015 at 12:48, François Chollet notifications@github.com wrote:

I've started experimenting a little bit with experiment visualization.

Here's a demo capable of monitoring a Keras experiment on a live-updating webpage: https://github.com/fchollet/hualos

What are your thoughts on what exactly we should be building? What do we need?

Here are some initial ideas:

-

typical workflow would require:

  • configure what you want to monitor and how via a callback constructor
    • be able to assign IDs to experiments and to monitor multiple experiments at the same time

  • monitoring capabilities would include:

  • loss and validation loss monitoring
    • accuracy and validation accuracy monitoring on a different scale
    • confusion matrix monitoring for classification tasks
    • layers weights monitoring
    • activations monitoring

— Reply to this email directly or view it on GitHub https://github.com/fchollet/keras/issues/254.

Tennessee Leeuwenburg http://myownhat.blogspot.com/ "Don't believe everything you think"

loyeamen commented 9 years ago

The ability to show a graph of the train and validation loss / accuracy during the training phase would be quite helpful.

aleju commented 9 years ago

I have some code here that generates a plot for the loss and accuracy values (live updated during training). I could try to put that into a simple callback that can be added to fit() as a parameter, i.e. fit(X, y, PlotterCallback()). Example plot: plot

loyeamen commented 9 years ago

If you could post the code that would be great Thanks

tleeuwenburg commented 9 years ago

Hi,

So the first thing I found is that this doesn't work on Python 3.4 due to gevent not being supported. Is Python 3.4 seen as a requirement? Apparently there is a gevent fork which is compatible. I'll try to get a working environment in a little while and look at the actual functionality.

Cheers, -Tennessee

On 21 June 2015 at 12:48, François Chollet notifications@github.com wrote:

I've started experimenting a little bit with experiment visualization.

Here's a demo capable of monitoring a Keras experiment on a live-updating webpage: https://github.com/fchollet/hualos

What are your thoughts on what exactly we should be building? What do we need?

Here are some initial ideas:

-

typical workflow would require:

  • configure what you want to monitor and how via a callback constructor
    • be able to assign IDs to experiments and to monitor multiple experiments at the same time

  • monitoring capabilities would include:

  • loss and validation loss monitoring
    • accuracy and validation accuracy monitoring on a different scale
    • confusion matrix monitoring for classification tasks
    • layers weights monitoring
    • activations monitoring

— Reply to this email directly or view it on GitHub https://github.com/fchollet/keras/issues/254.

Tennessee Leeuwenburg http://myownhat.blogspot.com/ "Don't believe everything you think"

ganarajpr commented 9 years ago

:+1: This is quite important. Could this perhaps be extended to be a GUI builder of NN's ?

fchollet commented 9 years ago

I do believe these tools should be framework-agnostic, ie. usable with different frameworks than Keras. Keras will interact with the visualization tools through a well-defined API, any framework that wants to plug into this API should be able to do so.

GUI NN building is not something I'm particularly interested in, since I believe text is more powerful than GUIs for manipulating complex systems. The main goal of these tools for the time being will be to expose everything that is going on inside an experiment, in a simple and visual way.

fchollet commented 9 years ago

So the first thing I found is that this doesn't work on Python 3.4 due to gevent not being supported. Is Python 3.4 seen as a requirement?

For now this is just a demo of how to get Keras and a webpage to communicate in real time, compatibility is not a concern. I'm sure we can replace gevent with something that will be Python 3 compatible when we move past the demo/exploration stage into actual implementation.

donglixp commented 9 years ago

I tried the demo. It is quite cool! It would be better if we plot the loss and accuracy in different figures. Maybe users can choose what they want to monitor (accuracy, loss, F1) in the dashboard. Another way is to enable users to define their own evaluation metric for the specific tasks, such as BLUE score for machine translation.

tristandeleu commented 9 years ago

I also just gave it a try and I really like it! I'm looking forward to move from console monitoring toward this centralized solution! In particular it would be very convenient to have the current batch (or training examples seen) per epoch and a progress bar like in verbose=1.

I also :+1: @donglixp regarding the metrics, even though I don't to what extent it would be possible.

AntreasAntoniou commented 9 years ago

Hey Francois,

Always a pleasure to use your libraries. Actually I would be very interested in a convolutional visualisation tool , so I can see what my network is "imagining" of you will. It's basically taking your load_weights method and mapping the tree of weights backwards towards the original image based on which pixel fires which neuron. Anyway awesome work man, my favourite deep learning lib right now next to Torch7 for lua

bottydim commented 8 years ago

Hey @fchollet ,

Before anything you are awesome! Thank you for enabling me to graduate!

As part of my research, I have just set out to develop an activation visualization tool for keras RNN models. It would be great to contribute to the hualos project, but I believe a paramount necessity is a Remote Monitor type of a callback within the predict or even evaluate function. It would publish the current:

What do you think?

hadim commented 8 years ago

To build a visualization tool, two things are needed :

My point of view is that the POST method is great because it allows remote monitoring (as aetros does) but how do you handle the case when the fitting process has already started for a while and you decide to monitor it ? The information is lost.

The log folder strategy allows you to access and explore 'Model' state when we want but the downside is that the folder need to be accessible for both the Keras instance and the visualization tool.

My opinion would go for the log folder strategy but I'd like to hear what do you think about that guys ?

The visualization tool should have the features highlighted by @fchollet in his first post of this issue.

bottydim commented 8 years ago

@hadim You are completely right!

Regarding the strategy, why don't we get the both of the two worlds i.e., publish the current information with a POST request and also keep a persistent object/file. We already have something similar in the form of the history object, which currently just stores the loss and accuracy. This way we could have both a live-monitoring functionality and information availability one. Additionally, the information could be explored after the experiment has concluded e.g., check out LSTMVis .

The idea is that the POST request will listen for the server response. If there have been any connection issues, the Callback would send the entire or only the missing parts from the log the moment the server becomes available again.

On the subject of the visualization tool, my thoughts were to build on top of Hualos and LSTMViz, but I agree with you that it should be a web-based application. Unfortunately, I have not worked with Tornado, but to my understanding, it is developed with '' long-lived connection to each user'' in mind, which is precisely what is going to happen with slowly trainable networks.

MartinThoma commented 7 years ago

It looks a bit as if fchollet/hualos is dead. @ollieglass did some improvements in https://github.com/ollieglass/hualos, as well as @Sebubu in https://github.com/Sebubu/hualos

@fchollet Do you want to keep developing Hualos or would you prefer if others continued the visualization project?

bottydim commented 7 years ago

@MartinThoma I have a strong interest in visualizing and understanding neural networks - this is the topic of my PhD :D If you are keen I am happy to collaborate!

fchollet commented 7 years ago

I will no longer be developing Hualos, feel free to take over.

On 4 April 2017 at 13:39, Botty Dimanov notifications@github.com wrote:

@MartinThoma https://github.com/MartinThoma I have a strong interest in visualizing and understanding neural networks - this is the topic of my PhD :D If you are keen I am happy to collaborate!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fchollet/keras/issues/254#issuecomment-291625007, or mute the thread https://github.com/notifications/unsubscribe-auth/AArWb9FxLBikL1T3P3WqBfaERC_rm6b0ks5rsqqagaJpZM4FIHf3 .

shacharm2 commented 7 years ago

Does it still work with Keras ?

stale[bot] commented 7 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.