Tests

[taqtiqa-mark]

Unit and integration tests are absent, as observed in #6. I am less sanguine that the examples suffice.

Perhaps a compromise is to replicate Schwartz &Smith, Figure 1:

and to then use the Filipovic & Larsen model to produce the same figure.

This is not strictly a test suite, but should give some assurance there are no egregious implementation errors (e.g. no negative prices, no arbitrageable prices, etc. I leave those test details to the authors.)

[peilun-he]

A new section about tests is added: https://github.com/peilun-he/PDSim/commit/13f64dc6acff907ac41eeeb308d94ecdcd34256d.

This section describes how users can test whether parameter values are suitable by viewing the estimated contract plot.

[taqtiqa-mark]

The addition of these figures and text is an improvement.

The term "test" could be considered ambiguous:

As in #6, here I refer to a suite of software implementation unit and integration tests. The proposed change, in https://github.com/peilun-he/PDSim/commit/13f64dc6acff907ac41eeeb308d94ecdcd34256d, deals with some thing that is closer to statistical tests in the context of estimation.

Nonetheless, the additions are not wasted and with some tweaking could prove more useful. Since it is unrelated to this issue, I'll address these points in another issue.

Given the subtleties that arise in these implementations, I believe a more rigorous implementation test suite is warranted - both unit and integration tests.

However, I indicated a compromise could be:

1. Replicating the Schwartz &Smith, Figure 1, using your implementation of Schwartz &Smith
2. Producing the same figure using your implementation of Filipovic, D., & Larsson, M. (2016)

[peilun-he]

More tests for Schwartz and Smith model and polynomial diffusion model using R scripts are added: https://github.com/peilun-he/PDSim/commit/091cbeb9c9f803c15cf613ee552c91b2d499741f.

For Figure 1 from Schwartz & Smith, our plots for futures prices offer greater detail, as we provide "unsmoothed" estimations for each trading day, along with the corresponding "unsmoothed" confidence intervals. Additionally, given the distinct spot price structure of the polynomial diffusion model, we do not anticipate the equilibrium price level to converge to the spot price over an extended period. So this plot is not applicable to our model.

[taqtiqa-mark]

I have added a related comment here:

https://github.com/peilun-he/PDSim/commit/091cbeb9c9f803c15cf613ee552c91b2d499741f#commitcomment-138492536

For Figure 1 from Schwartz & Smith, our plots for futures prices offer greater detail,

I believe it is important to show that you replicate the properties of their model shown in that figure. Please see my comment above.

Additionally, given the distinct spot price structure of the polynomial diffusion model, we do not anticipate the equilibrium price level to converge to the spot price over an extended period. So this plot is not applicable to our model.

You keep referring to "our model" - is this not just the FP model? If not please clarify how "your" model differs from FP. This plot is applicable to the FP model, if the FP model reduces to the SS model.

Does the FP model not reduce to the SS model?

[peilun-he]

For the original Schwartz Smith model, as the long-term factor $\xi_t$ follows a OU process with drift $\gamma = 0$, $\xi_t$ will go to infinity as time $t$ goes to infinity. Therefore, the spot price will converge to the long-term factor $\xi_t$. But for our model, $\gamma \ne 0$. There is no such convergence between the long-term factor and the spot price.

Instead, we added another plot to show the accuracy of our simulation, compared to the real data: https://github.com/peilun-he/PDSim/commit/8e58f47b4301c06200cf0135b4b65d2683b972e3

[taqtiqa-mark]

Lets pause and take a step back. There are several concepts of accuracy and testing that are muddying the waters.

Please reproduce Figure 1 from the SS paper using the your implementation of the SS paper.

[peilun-he]

Dear Prof. Van de Vyver,

This is a response from Prof. Peters in consultation with the corresponding author Mr. He, intended to clarify aspect of this response to your request for unit testing.

Firstly, we agree that there is not currently a comprehensive suite of unit tests to accompany this submitted package, and we acknowledge that an alternative set of demonstrations of the codes validity is therefore warranted.

What remains is to agree what is a sensible choice of tests that will achieve the objective of providing assurance that the code is accurately implemented in lieu of unit tests being present.

The request for replication of the SS plot is a reasonable suggestion. However, the implementation of this request in a prior response has been performed using different data and different model parameters. This does not invalidate the accuracy and suitability of this result in the context in which it is presented. The reason we argue this point is because in the illustration provided we simulated the synthetic data from the model with fixed known parameters. Then we used this data and fixed parameters to estimate the state sequence of SS model, and then construct the estimated futures price. The replication of estimated futures price using the estimated state sequence from the Kalman Filter replicates accurately the futures data trajectory. This is not be feasible with such accuracy if there was likely to be erroneous implementation of the code relating the estimation of the code for the SS model.

Furthermore, the same type of exercise was repeated for a synthetic case study in the polynomial diffusion models (Filipovic & Larsson). Again, it was demonstrated that the code used to estimate the polynomial diffusion states sequence, subsequently used to estimate the futures price dynamic, also very accurately replicates the simulated data. This leads us to conclude that again the polynomial diffusion model code is accurate.

The reason that we argue that the two synthetic experiments cannot be made identical to each other arises from the fact that these two model are not nested. What this means is that there are no parameter setting in the polynomial diffusion model that can be identified to exactly match the SS model.

However, there are examples of parameter restrictions which are unrealistic examples in practice which involve setting several model parameters to 0 in the polynomial diffusion model which will produce a constant relation between the futures price from the restricted polynomial diffusion model and log futures price from SS (this restriction is provided in mathematical detail in the README file in section "Difference between Futures Prices"). This restricted example illustrates the fact that in the case that the polynomial diffusion is reduced to a model with a non-stochastic state evolution, restriction on the state equations innovation error variances are require to be set to 0 for this relation to be achieved.

Hence, this is why we have performed both the synthetic studies seperately for the SS and polynomial diffusion model as outlined above.

We hope this clarifies both why we did the synthetic examples in the manner in which we did them, and also makes clear that the SS model is not a special case or a nested model of the polynomial diffusion model.

Both Mr. He and co-authors deeply appreciate your time and consideration on this review and we hope that we have clarified the justification for our response to this request. Of course, if there are any additional aspects you wish to see we are more than happy to work with you to finalise this review as needed.

[taqtiqa-mark]

The request for replication of the SS plot is a reasonable suggestion. However, the implementation of this request in a prior response has been performed ...

My apologies if that is the case. Please point me to where this plot is:

using different data and different model parameters.

Why would you consider that to be a replication?

[peilun-he]

Figure 1 has been produced in the "Replicating Schwartz and Smith's Results" section (1st plot). We created the plot of reference from Schwartz and Smith (2000), using the parameters from the paper. Additionally, we generated a similar plot for the polynomial diffusion model (2nd plot). Furthermore, a new navigation bar titled "Unit tests" is created in the application. This feature enables users to test any set of parameters by simulating multiple trajectories using these parameters and then the coverage rate by the 95% confidence interval is provided. The detailed description in the "Unit Test" section.