train_loss is not found

[ilkarman]

I run the plot_surface code like so:

    /usr/bin/python -u /local/mnt/workspace/ikarmano/Gitlab/sagd/loss-landscape/plot_surface.py --cuda \
    --x=-1:1:51 --y=-1:1:51 --model_file models/32_32_32_32_32_32_32_32_32_32_32_32_32_32_32cnn.t \
    --dir_type weights --xnorm filter --xignore biasbn --ynorm filter --yignore biasbn --plot

And it seem to calculate the loss fine:

Evaluating rank 2  90/2601  (3.5%)  coord=[ 0.56 -0.96]     train_loss= 21.470  train_acc=14.54     time=5.28   sync=0.00
Evaluating rank 2  91/2601  (3.5%)  coord=[ 0.6  -0.96]     train_loss= 22.225  train_acc=14.10     time=5.65   sync=0.00
Evaluating rank 2  92/2601  (3.5%)  coord=[ 0.64 -0.96]     train_loss= 23.044  train_acc=13.67     time=5.92   sync=0.00
Evaluating rank 2  93/2601  (3.6%)  coord=[ 0.68 -0.96]     train_loss= 23.935  train_acc=13.33     time=5.71   sync=0.00
Evaluating rank 2  94/2601  (3.6%)  coord=[ 0.72 -0.96]     train_loss= 24.905  train_acc=13.02     time=5.65   sync=0.00
Evaluating rank 2  95/2601  (3.7%)  coord=[ 0.76 -0.96]     train_loss= 25.958  train_acc=12.66     time=5.50   sync=0.00
Evaluating rank 2  96/2601  (3.7%)  coord=[ 0.8  -0.96]     train_loss= 27.100  train_acc=12.37     time=5.99   sync=0.00
Evaluating rank 2  97/2601  (3.7%)  coord=[ 0.84 -0.96]     train_loss= 28.334  train_acc=12.13     time=5.85   sync=0.00
Evaluating rank 2  98/2601  (3.8%)  coord=[ 0.88 -0.96]     train_loss= 29.666  train_acc=11.91     time=5.71   sync=0.00
Evaluating rank 2  99/2601  (3.8%)  coord=[ 0.92 -0.96]     train_loss= 31.101  train_acc=11.69     time=5.58   sync=0.00

However, the plot functions do not work because 'train_loss' is not found:

train_loss is not found in ../models/32_32_32_32_32_32_32_32_32_32_32_32_32_32_32cnn.t_weights_xignore=biasbn_xnorm=filter_yignore=biasbn_ynorm=filter.h5_[-1.0,1.0,51]x[-1.0,1.0,51].h5

And if I print the keys(), it's just:

<KeysViewHDF5 ['dir_file', 'xcoordinates', 'ycoordinates']>

Not sure what I'm doing wrong?

[ljk628]

Hi @ilkarman, our code saves the surface values by the rank 0 process in default after collecting values calculated by multiple processes, as you can see in https://github.com/tomgoldstein/loss-landscape/blob/master/plot_surface.py#L88 and https://github.com/tomgoldstein/loss-landscape/blob/master/plot_surface.py#L136.

It seems that you are not using mpi and your process rank value is 2, so it might be the reason why the surface values are not saved into the h5 file. It could be an easy fix if you change the default rank values to 2 or figure out why it is not zero.

[ilkarman]

Thank you! One of the params for crunch() was being overwritten instead of rank.