TODO

[tbreloff]

I'll try to maintain a wishlist of plot recipes here.

• [x] Correlation Scatter Matrix (corrplot)
• [ ] Error/loss tracking
• [ ] Goodness of fit (with optional contours?)
• [x] ROC
• [x] ANN activations
• [ ] Classification histograms
• [x] Spike trains
• [ ] SVM plots

[Evizero]

You could add SVM related plots to the list since I will also put them into this package

[Evizero]

I recently see @trthatcher write on visualisation code for MLKernels.jl. Maybe these would also be a good fit for this package to keep dependencies for MLKernels to a minimum?

[trthatcher]

I didn't know about Plots.jl... this is pretty cool. Visualization of some of the feature extraction/dimensionality reduction techniques could be interesting.

@Evizero Don't worry... I'm planning on keeping MLKernel's dependencies non-existent.

[tbreloff]

Here's a new recipe I just added for OnlineAI:

using MLPlots, OnlineAI
net = buildClassificationNet(3,1,[15,10,5])
for i in 1:1000
    x = rand(3)
    update!(net, x, Float64[sum(x)])
end
plot(net)

[Evizero]

There seems to be a bug with gadfly when the position should be at y = 0. Also the cricles seem to be of fixed size, so if the plot size is reduced the circles stay the same absolute size

[Evizero]

What would be really cool is if the weights were drawn too

[Evizero]

by the way to code to produce the gif (I use a subplot so I don't have to find out how I would update the plot, but can instead just overwrite it):

using MLPlots, OnlineAI
gadfly()
net = buildClassificationNet(3,1,[15,10,5])
myplot = plot(net, size = (300, 300))
combo = subplot(myplot, n=1)
anim = Animation(combo)
for i in 1:1000
    x = rand(3)
    update!(net, x, Float64[sum(x)])
    if (i-1) % 50 == 0
        combo.plts[1] = plot(net, leg = false, size = (600, 500))
        frame(anim)
    end
end
gif(anim, "figures/nnet.gif", fps = 5)

[tbreloff]

There seems to be a bug with gadfly when the position should be at y = 0

That's weird

Also the cricles seem to be of fixed size, so if the plot size is reduced the circles stay the same absolute size

You should be able to override the markersize if you want.

What would be really cool is if the weights were drawn too

I know. My viz through Qwt has colored weights and biases, with varying marker sizes depending on layer size. In order to make all this work properly, I think I'll need to solve https://github.com/tbreloff/Plots.jl/issues/19 first.

[tbreloff]

Spike trains:

using OnlineAI, MLPlots
spikes = SpikeTrains(5)
for i in 1:1000
    push!(spikes, rand(1:5), rand())
end
spikes.plt

and the same with 50 neurons:

[Evizero]

:+1:

[diegozea]

I'm having problems with your example for OnlineAI:

WARNING: Error requiring OnlineAI from MLPlots:
LoadError: UndefVarError: _apply_recipe not defined
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:304
 in eval at /home/dzea/.julia/v0.4/MLPlots/src/MLPlots.jl:1
 in anonymous at /home/dzea/.julia/v0.4/Requires/src/require.jl:60
 in err at /home/dzea/.julia/v0.4/Requires/src/require.jl:47
 in anonymous at /home/dzea/.julia/v0.4/Requires/src/require.jl:59
 in withpath at /home/dzea/.julia/v0.4/Requires/src/require.jl:37
 in anonymous at /home/dzea/.julia/v0.4/Requires/src/require.jl:58
 in anonymous at /home/dzea/.julia/v0.4/Requires/src/require.jl:26
 in map at abstractarray.jl:1305
 in loadmod at /home/dzea/.julia/v0.4/Requires/src/require.jl:25
 [inlined code] from /home/dzea/.julia/v0.4/Requires/src/require.jl:12
 in require at /home/dzea/.julia/v0.4/MacroTools/src/utils.jl:108
while loading /home/dzea/.julia/v0.4/MLPlots/src/OnlineAI/onlineai.jl, in expression starting on line 7

julia> plot(net)
[Plots.jl] Initializing backend: pyplot
INFO: Precompiling module PyPlot...
ERROR: MethodError: `convertToAnyVector` has no method matching convertToAnyVector(::OnlineAI.NeuralNet)
 in createKWargsList at /home/dzea/.julia/v0.4/Plots/src/plot.jl:263
 in createKWargsList at /home/dzea/.julia/v0.4/Plots/src/plot.jl:330
 in plot! at /home/dzea/.julia/v0.4/Plots/src/plot.jl:93
 in plot at /home/dzea/.julia/v0.4/Plots/src/plot.jl:56

[tbreloff]

Possible you need the dev branch of Plots and/or OnlineAI? What does Pkg.status() say?

On Dec 16, 2015, at 11:31 AM, Diego Javier Zea notifications@github.com wrote:

I'm having problems with your example for OnlineAI:

WARNING: Error requiring OnlineAI from MLPlots: LoadError: UndefVarError: _apply_recipe not defined in include at ./boot.jl:261 in include_from_node1 at ./loading.jl:304 in eval at /home/dzea/.julia/v0.4/MLPlots/src/MLPlots.jl:1 in anonymous at /home/dzea/.julia/v0.4/Requires/src/require.jl:60 in err at /home/dzea/.julia/v0.4/Requires/src/require.jl:47 in anonymous at /home/dzea/.julia/v0.4/Requires/src/require.jl:59 in withpath at /home/dzea/.julia/v0.4/Requires/src/require.jl:37 in anonymous at /home/dzea/.julia/v0.4/Requires/src/require.jl:58 in anonymous at /home/dzea/.julia/v0.4/Requires/src/require.jl:26 in map at abstractarray.jl:1305 in loadmod at /home/dzea/.julia/v0.4/Requires/src/require.jl:25 [inlined code] from /home/dzea/.julia/v0.4/Requires/src/require.jl:12 in require at /home/dzea/.julia/v0.4/MacroTools/src/utils.jl:108 while loading /home/dzea/.julia/v0.4/MLPlots/src/OnlineAI/onlineai.jl, in expression starting on line 7 julia> plot(net) [Plots.jl] Initializing backend: pyplot INFO: Precompiling module PyPlot... ERROR: MethodError: convertToAnyVector has no method matching convertToAnyVector(::OnlineAI.NeuralNet) in createKWargsList at /home/dzea/.julia/v0.4/Plots/src/plot.jl:263 in createKWargsList at /home/dzea/.julia/v0.4/Plots/src/plot.jl:330 in plot! at /home/dzea/.julia/v0.4/Plots/src/plot.jl:93 in plot at /home/dzea/.julia/v0.4/Plots/src/plot.jl:56 — Reply to this email directly or view it on GitHub.

[diegozea]

Thanks! It is working after a Pkg.checkout("Plots")

I want to do the ROC recipe using ROCAnalysis. I only need to define Plots._apply_recipe(d::Dict, roc::ROCAnalysis.Roc{T}, don't I?

[Evizero]

I looked into ROCAnalysis. I think we are better off with the MLBase version when it comes to machine learning. The design decisions in ROCAnalysis are very unusual. The expected parameters for roc are tedious to compute

[Evizero]

That being said, if it fits your use-case then I don't want to discourage you. Since we are using require I don't see any reason not to merge it. But if you are undecided then I'd suggest looking into MLBase and see if it fits your interests better

[tbreloff]

Cool! If your Roc object holds everything you need to do the plot, then yes you should just need to implement the _apply_recipe method. Use NeuralNet or the loss functions as guides, and don't hesitate to ask questions (can consider opening a new issue for it)

On Dec 16, 2015, at 11:48 AM, Diego Javier Zea notifications@github.com wrote:

Thanks! It is working after a Pkg.checkout("Plots")

I want to do the ROC recipe using ROCAnalysis. I only need to define Plots._apply_recipe(d::Dict, roc::ROCAnalysis.Roc{T}, don't I?

— Reply to this email directly or view it on GitHub.

[diegozea]

I want to annotate the AUC number close to the point (0.9, 0.1)... How can I do that?

[tbreloff]

Lots of ways, here's one:

using Plots
pyplot()
x = rand(5)
scatter(x, ann=[(2,x[2]-0.05,text("this is #2")), (3,x[3],text(" this is #3", :left, 20, :red, :monospace))])

The tuples are: (x, y, text). ann is an alias for annotation

[diegozea]

Thanks! Now I have:

function Plots._apply_recipe(d::Dict, roc::ROCAnalysis.Roc, auc::Bool=false; kw...)
.
.
.
    if auc
        AUC = round(1 - ROCAnalysis.auc(roc), 4)
        get!(d, :annotation, (0.9, 0.05, text("AUC $(AUC)", :center, 10, :monospace)))
    end

I'm using the auc parameter set to false as default, because plot! superposes the annotations. You need to call it as plot(roc, true) to see the AUC value. Is it fine? Is there a way to know if the call to _apply_recipe was did for plot or plot! inside the function?

[tbreloff]

Is there a way to know if the call to _apply_recipe was did for plot or plot! inside the function?

Actually, _apply_recipe is called within plot! (which is also called by plot). A hack which could probably work... you could test for plt.n == 0, as this variable gets incremented after the call to _apply_recipe, and the second time you call it, plt.n should be positive.

[diegozea]

I can't see plt inside _apply_recipe... I'm getting plt not defined.

[tbreloff]

Oh sorry. You're right, no plt available. If you give me a compelling use-case, I could add an additional arg to the method... Maybe a keyword arg? Let me know what you wanted to do... There might be a better way.

On Dec 16, 2015, at 2:38 PM, Diego Javier Zea notifications@github.com wrote:

I can't see plt inside _apply_recipe... I'm getting plt not defined.

— Reply to this email directly or view it on GitHub.

[Evizero]

I think a clean way to see if its a plot or plot! would be a great addition.

[tbreloff]

Ok I'll consider replacing the "issubplot" keyword with a "plot_object" keyword, passing in either the Plot or Subplot objects. Then, in addition to checking the type, you could check the "plot_object.n" field.

I'll also check out how tough it would be to rename those "n" and "p" fields.

On Dec 16, 2015, at 5:25 PM, Christof Stocker notifications@github.com wrote:

I think a clean way to see if its a plot or plot! would be a great addition.

— Reply to this email directly or view it on GitHub.

[davidavdav]

Great that you have an interface for ROCAnalysis.

I would want to make a case, though, that the default plot area for a ROC curve IMHO should be square, as errors if one type cannot be assumed to more/less important than errors of the other type, without having done a cost analysis. Non-square plots would visually make one less important than the other.

(I also think that false positive vs false negative is better than false positive vs true positives, and that AUCs should be expressed as an error rate and not the complement of an error rate, but the rest of the world doesn't).