avik-pal / Wandb.jl

Unofficial Julia bindings for logging experiments to wandb.ai
https://avik-pal.github.io/Wandb.jl/stable/
MIT License
80 stars 10 forks source link

Not all histograms appear correctly in wandb #15

Closed rkube closed 2 years ago

rkube commented 2 years ago

Hi, logging histograms with Wandb works, but not consistently. This pseudo-code:


ps = Flux.params(model)
for epoch in 1:num_epochs

    loss, back = Zygote.pullback(ps) do
        loss = ...
    end
    grads = back(one(loss))

    if (output_time)
        (x_test, y_test) = first(loader_test)
        y_real = model(x_test) |> cpu;

         grads_out = grads[ps[1]][:] |> cpu;

        log(wandb_logger, Dict("hist y_real" => Wandb.Histogram(y_real),
                                                  "grads" => Wandb.Histogram(grads))
    end
end

works for y_real but gives an error message for grads in the wandb dashboard:

Sometimes it works

Sometimes it doesn't

Both grads_out and y_real are CPU vectors when they are passed to the logger:

julia> typeof(grads_D1)
Vector{Float32} (alias for Array{Float32, 1})

julia> typeof(y_real)
Matrix{Float32} (alias for Array{Float32, 2})

julia> size(grads_D1)
(216,)

julia> size(y_real)
(2, 128)
avik-pal commented 2 years ago

"grads" => Wandb.Histogram(grads)

Seems to be a typo. You perhaps meant to pass grads_out?

rkube commented 2 years ago

Yeah, that's a typo. The actual training script is more complicated. The summary tab on the web interface actually shows the histogram data correctly, so it's probably an issue with the web interface and not this package.

rkube commented 2 years ago

I had to clear my workspace, using the menu in the lower left of the web interface

Clear workspace