plotting trayectory

[javismiles]

Good day, could you give me an example of the right command to plot the trajectory of one of the optimization processes using plot_trajectory.py?

specifically for example, the one that would match with the surface generated by: model_300.t7_weights_xignore=biasbn_xnorm=filter_yignore=biasbnynorm=filter.h5[-1.0,1.0,201]x[-1.0,1.0,201].h5

Also, I understand plot_trayectory uses PCA. These generated trajectories, with what level of precision do they match with their related point cloud generated by plot_surface.py?

thank you very much Jav

[kelly08385]

I have the same question, I had successfully run the plot example and get the contour output pdf. I want to plot the trajectory on the contour now. Do anyone have example?

What should I save while training the model?

[pranaymodukuru]

I even have the same question.

What to save during training and how should I process the saved values?

[ranery]

have you figured out how to achieve that?

[ljk628]

Sorry for the late reply! @javismiles

To plot the trajectory in the PCA directions, you need to prepare the checkpoint files during training, e.g., you have trained a ResNet56 model for 300 epochs on CIFAR10 and save the checkpoints in the format of model_{epoch}.pth every 10 epochs in path_to_model_folder. The command to plot the trajectory is

python plot_trajectory.py --dataset cifar10 --model resnet56 --model_folder path_to_model_folder \
--start_epoch 0 --max_epoch 300 --save_epoch 10 --prefix model_ --surffix .pth

If you want use specific direction file rather than the PCA direction, you can specify with the option --dir_file path_to_direction_file.

Note that the trajectory loss values do not exactly correspond to the projected loss contours for the following reasons:

1) The loss landscape dynamically changes during training and we only present the "local" landscape around the final solution. To plot the loss landscape around the solution, we have to fix the final BN statistics and parameters. While the real loss value per checkpoint should be calculated based on specific BN statistics and parameters during training. Therefore the checkpoints faraway from the final solution will have much larger discrepancies between real loss value and the projected one. However, the trajectory close to the final solution should be close to the true values when the BN statistics settles down and are close to the final ones.

2) the projection error. The projection of trajectory by the two PCA directions is an approximation and the recovered corresponding point on loss landscape may not have the exact model parameters as the checkpoints.

3) there are more iterates clustered near the end of the optimization trajectory, and fewer at the begging, the resulting PCA directions represent the end of optimization with more fidelity than the start of optimization.

For above reasons it's hard to accurately capture the trajectory of SGD. We believe superior way can be developed to plot the true SGD trajectories.

[alqurri77]

I got the below error when trying to plot the trajectory:

plot_trajectory.py --model resnet56 --model_folder /tmp/ahmed \ --dir_type weights --prefix landscape/loss-landscape-master/all_models/my_model --suffix .pth --max_epoch 5

/tmp/ahmed/landscape/loss-landscape-master/all_models/my_model5.pth  (0.0000, 0.0000)
Traceback (most recent call last):
  File "/tmp/ahmed/landscape/loss-landscape-master/plot_trajectory.py", line 66, in <module>
    plot_2D.plot_trajectory(proj_file, dir_file)
  File "/tmp/ahmed/landscape/loss-landscape-master/plot_2D.py", line 87, in plot_trajectory
    plt.plot(f['proj_xcoord'], f['proj_ycoord'], marker='.')
  File "/tmp/ahmed/ana/anaconda3/lib/python3.11/site-packages/matplotlib/pyplot.py", line 3578, in plot
    return gca().plot(
           ^^^^^^^^^^^
  File "/tmp/ahmed/ana/anaconda3/lib/python3.11/site-packages/matplotlib/axes/_axes.py", line 1721, in plot
    lines = [*self._get_lines(self, *args, data=data, **kwargs)]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ahmed/ana/anaconda3/lib/python3.11/site-packages/matplotlib/axes/_base.py", line 303, in __call__
    yield from self._plot_args(
               ^^^^^^^^^^^^^^^^
  File "/tmp/ahmed/ana/anaconda3/lib/python3.11/site-packages/matplotlib/axes/_base.py", line 505, in _plot_args
    x = x[:, np.newaxis]
        ~^^^^^^^^^^^^^^^
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/tmp/ahmed/ana/anaconda3/lib/python3.11/site-packages/h5py/_hl/dataset.py", line 758, in __getitem__
    return self._fast_reader.read(args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "h5py/_selector.pyx", line 361, in h5py._selector.Reader.read
  File "h5py/_selector.pyx", line 107, in h5py._selector.Selector.apply_args
ValueError: 2 indexing arguments for 1 dimensions