how to plot training loss

[turmeric-blend]

how to plot training loss as well, maybe in tensorboard or manually in jupyter. This is good to check if the model has overfit or underfit by comparing training loss and validation loss. As of my current understanding tensorboard callback only plots test (validation) loss.

[jankrepl]

The simplest solution is putting your training dataloader into the val_dataloaders dictionary (in the constructor of Run). You can actually put as many dataloaders in this dictionary as you want. At the end of each epoch the TensorBoardCallback is going to log into TensorBoard the desired metrics of all dataloaders.

To give an example, I am going to use the same variables as here https://deepdow.readthedocs.io/en/latest/auto_examples/end_to_end/getting_started.html#sphx-glr-auto-examples-end-to-end-getting-started-py

run = Run(network,
          loss,
          dataloader_train,
          val_dataloaders={
                            'test': dataloader_test,
                            'train': dataloader_train  # <--- HERE
          },
          optimizer=torch.optim.Adam(network.parameters(), amsgrad=True),
          callbacks=[EarlyStoppingCallback(metric_name='loss',
                                           dataloader_name='test',
                                           patience=15)])

Note that once the training is done you can also get all relevant metrics as pd.DataFrame via history.metrics.

[turmeric-blend]

Ok that solves plotting the training loss.

I have 2 further questions regarding plotting metrics.

1. what if I want to plot other scalar metrics other than loss, for example if I want to plot the Mean Sharpe Ratio per test dataset (a scalar value like loss) in tensorboard itself. This allows for easier comparison between changes in model performance to hyperparameter tuning. Note benchmarks could be plotted in tensorboard as well although this would be a one time thing.

2. from your getting started tutorial, you plotted the network performance vs benchmarks in terms of Sharpe Ratio, MaxDD, how are these point-by-point metrics calculated exactly? As these metrics require a certain time length, [x1, x2, ...xN] to obtain a single scalar of say, Sharpe Ratio. I have two guesses, either by a forward/backward looking fixed moving average method (eg if taking 3 days to calculate the Sharpe Ratio, and we have a total of 7 return days, then only 5 scalar Sharpe Ratio values would be returned) or an increasing moving average method (eg if taking 3 days to calculate the Sharpe Ratio, and we have a total of 7 return days, then 7 scalar Sharpe Ratio values would be returned where the first scalar is based on 1 day, second based on 2 days, and the rest on 3 days)

My questions may be a bit confusing so feel free to ask me for clarification. I really appreciate your help!

[jankrepl]

1. The constructor of Run accepts a keyword argument metrics. It serves exactly the same purpose as you described https://github.com/jankrepl/deepdow/blob/cab9cac9d9212dd839951f65a9c0b49ca961eec7/deepdow/experiments.py#L110
2. Almost all loses in deepdow are some summary statistics of the buy and hold strategy over horizon timesteps. So for example assume that
   • We fix a date - 10.10.2020 and we buy all the assets in the portfolio (based on w=network(X))
   • We hold this portfolio 20 days (horizon=20)

Each day we will have 1 new datapoint representing the portfolio return. Finally, on the 30th October 2020 (assuming no gaps) we will have 20 datapoints representing daily portfolio returns over the holding period. In general, deepdow losses simply take these 20 datapoints and compute summary statistics like mean, standard deviation, sharpe ratio, etc. By changing the initial dates (moving along the x-axis) you can generate the plots shown in the example (plot_metrics).

The mental image you can keep in your head is the sketch here https://deepdow.readthedocs.io/en/latest/source/basics.html#loss