search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
725 stars 38 forks source link

[REVIEW]: PDSim: A Shiny App for Polynomial Diffusion Model Simulation and Estimation #5762

Closed editorialbot closed 4 months ago

editorialbot commented 1 year ago

Submitting author: !--author-handle-->@peilun-he<!--end-author-handle-- (Peilun He) Repository: https://github.com/peilun-he/polynomial-diffusion-model-simulation-and-estimation Branch with paper.md (empty if default branch): Version: v2.0 Editor: !--editor-->@samhforbes<!--end-editor-- Reviewers: @taqtiqa-mark, @bkrayfield Archive: Pending

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/01ca1963062f7aa2ac70867648ad9548"><img src="https://joss.theoj.org/papers/01ca1963062f7aa2ac70867648ad9548/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/01ca1963062f7aa2ac70867648ad9548/status.svg)](https://joss.theoj.org/papers/01ca1963062f7aa2ac70867648ad9548)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@taqtiqa-mark & @bkrayfield, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @samhforbes know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @bkrayfield

📝 Checklist for @taqtiqa-mark

editorialbot commented 1 year ago

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf
editorialbot commented 1 year ago 
Software report:

github.com/AlDanial/cloc v 1.88  T=0.01 s (2021.2 files/s, 163357.3 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
R                               22            148            218           1135
Markdown                         3            100              0            485
TeX                              1              7              0             65
YAML                             1              1              4             18
Rmd                              1             29             47              6
-------------------------------------------------------------------------------
SUM:                            28            285            269           1709
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository
editorialbot commented 1 year ago

Wordcount for paper.md is 1949

editorialbot commented 1 year ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- None

MISSING DOIs

- 10.1287/mnsc.46.7.893.12034 may be a valid DOI for title: Short-Term Variations and Long-Term Dynamics in Commodity Prices
- 10.2139/ssrn.2479826 may be a valid DOI for title: Polynomial Diffusions and Applications in Finance
- 10.1137/19m1283264 may be a valid DOI for title: A multifactor polynomial framework for long-term electricity forwards with delivery period
- 10.1109/jproc.2003.823141 may be a valid DOI for title: Unscented filtering and nonlinear estimation

INVALID DOIs

- None
editorialbot commented 1 year ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

samhforbes commented 1 year ago

Hi @taqtiqa-mark, @bkrayfield this is our review thread. Please check the instructions in the initial post about how to generate your checklist, and proceed with the review. Typically I would encourage you to raise any issues on the software repository directly, and simply link back to here, but if there are any wider points that need discussion you are welcome to do that here.

bkrayfield commented 1 year ago

Review checklist for @bkrayfield

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

bkrayfield commented 1 year ago

Hello @samhforbes, I completed at least one round of review on the repository. I opened any concerns I had with the software repository directly. It has only been a week since those issues were raised. How long should I wait to see if they are addressed before completing my review?

bkrayfield commented 1 year ago

I have four outstanding issues before I would feel comfortable checking all of the boxes.

peilun-he/PDSim/issues/4 peilun-he/PDSim/issues/5 peilun-he/PDSim/issues/6 peilun-he/PDSim/issues/7

peilun-he commented 1 year ago

Hi @bkrayfield, thank you very much for your comments. I will discuss all these issues with co-authors and get back to you as soon as possible.

samhforbes commented 1 year ago

Hi @peilun-he we look forward to your responses, and thanks for your work reviewing this @bkrayfield We also look forward to your review @taqtiqa-mark - please let me know if I can help with anything.

peilun-he commented 1 year ago

Hi @bkrayfield Thanks you very much for your feedback to our package. I have made some updates based on it and commented on each issue. Please let me know if you have further concerns.

samhforbes commented 1 year ago

Hi @bkrayfield how do you feel about the revisions made in answer to your comments?

bkrayfield commented 1 year ago

Hello @samhforbes, I have resolved most of the issues. I am still looking over the tests related to issue . I am also waiting on @taqtiqa-mark for issue #4 .

samhforbes commented 1 year ago

Great, thank you both. @taqtiqa-mark I can see you have been active in issues, but please also make sure to complete the checklist here so I can confirm you have signed off when you're done!

taqtiqa-mark commented 1 year ago

Apologies, RL and paid work have intervened. I hope to complete this over the weekend.

samhforbes commented 12 months ago

Just a reminder to see if we can help with anything @taqtiqa-mark @bkrayfield

bkrayfield commented 11 months ago

@samhforbes I am done, and good. I still have an issue open https://github.com/peilun-he/PDSim/issues/4, but I am ready to close if @taqtiqa-mark is good with it.

samhforbes commented 11 months ago

This is all looking really good. @taqtiqa-mark would you mind finishing off your checklist please if you are happy with everything?

taqtiqa-mark commented 11 months ago

Review checklist for @taqtiqa-mark

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

taqtiqa-mark commented 11 months ago

Currently blocking issues for me are:

Of these only issue 14 may require non-trivial effort to resolve. The rest I expect can be resolved without much effort.

peilun-he commented 11 months ago

@taqtiqa-mark I have solved some of these issues. I will discuss all others with co-authors and get back to you as soon as possible. Thank you for your comments and wish you have a happy new year!

samhforbes commented 9 months ago

Hi @peilun-he how is this progressing?

peilun-he commented 9 months ago

@samhforbes Only a few issues are left. I'm still working on them and I expected they can be solved by the end of this week.

samhforbes commented 8 months ago

Hi @peilun-he I hope you had a nice break. I'm just checking in to see how is this progressing?

peilun-he commented 8 months ago

Hi @samhforbes Hope you had a good holiday too. We have 5 issues left now. I'm working on a new version which should be able to solve all of them. This version will be done this week.

samhforbes commented 7 months ago

Hi @peilun-he I just want to check in here - am I right in thinking from the issues raised that we should expect an update about unit testing and docker containers or alternative around the end of April?

taqtiqa-mark commented 7 months ago

@samhforbes , from my PoV I'd expect all issues to be addressed. I'm particularly concerned about the inability or reluctance to reproduce some figures from one of the original papers - of course it is possible the original paper is incorrect. However, I have directed the authors to one existing replication of one of the figures at issue, so I am doubtful that will be the case. Hopefully we find out.

taqtiqa-mark commented 7 months ago

Blocking issues are:

peilun-he commented 7 months ago

Hi @samhforbes , yes we are working on a new version about unit testing, docker installation, and the two requried figures. Also we will be making some changes to the UI to incorporate the unit testing, which will take some time. All these should be done by the end of April and hopefully this version will solve all issues. Sorry for the delay.

samhforbes commented 7 months ago

Noted, thank you both!

peilun-he commented 7 months ago

Dear @taqtiqa-mark, we will have a completed version of all requestd items by May 10th - just to let you know it is progressing.

samhforbes commented 6 months ago

Hi @peilun-he how is this progressing?

peilun-he commented 6 months ago

Hi @samhforbes, we have updated the new version and are wating for the reviewer feedback.

taqtiqa-mark commented 6 months ago

@samhforbes, I have spent considerable time and effort to try and assist the authors revise their submission. Previously I asked:

Please only update these issues when you believe you have implemented what I have suggested is need. Or ask an editor to disregard the suggestions.

Consequently, I have taken the authors update as their final response.

Unfortunately, I must recommend their submission be rejected and they not be invited to resubmit it, nor any related work.

There are numerous reasons for this. However, I will address one that, to my mind, is determinate. Namely, their persistent refusal to properly cite the source of their ideas, and properly attribute the work of others. I have raised this issue to little effect:

Yet the authors persist in making the claim they do more than (badly) implement the model of Filipovic & Larsen, they continue to claim some degree of authorship:

Related, as I noted previously, the authors steadfastly refused to replicate some existing results, see here, here, here, here and 3-months later here, . Only doing so when it was brought to your attention as an issue. As I noted at the time I had to find an existing replication, despite earlier asking them this question directly, and ask they at least mimic that authors calculations. They were incapable of even that level of effort.

To make matters worse, they have (again) seen fit to refuse to acknowledge that prior art (replication) they are mimicing. While there is a reference in the readme file, nowhere in the body of their work do they cite and acknowledge the work of Moeti Ncube, and the role it played in helping them resolve their profound confusion about the objects they have been dealing with.

As I indicated above there remain substantial defects in the current submission. Defects I am confident they cannot remedy without substantial assistance from myself or one similarly well versed in the art. However, it is the fact the authors continue to hold out the work of Filipovic & Larsen as "our model", and their choice not to acknowledge the prior art of M. Ncube, which lead me to recommend the submission be rejected and the authors not be invited to resubmit it, nor any related work.

I understand the editors may review the facts and arrive at a different conclusion.
I acknowledge that reasonable minds can disagree on these points, and respect their right to do so. I that case, I simply ask the editors remove my name as a reviewer available to the journal.

Sincerely Mark Van de Vyver

peilun-he commented 6 months ago

Dear Editor,

First of all, thank you for keeping access to the repository open and enabling us to respond to Dr Van de Vyver's comments. For us, the paper review process through open dialogue has been a highly stimulating and positive experience until this very moment, regardless of the outcome. In appreciation of the reviewers' time and efforts dedicated to improving our work, we have added the Acknowledgments section to the paper. Furthermore, we will address the comments made by the Reviewer:

1.

Namely, their persistent refusal to properly cite the source of their ideas, and properly attribute the work of others.

We have updated the references according to the Reviewer's suggestion about the previous work done by Ncube (2010) on the Schwartz-Smith (2000) model. Now, we have also cited the work by Goodwin (2015) and He (2020). Therefore, thanks to the Reviewer's comments, we have included the above three references in the README.md file. We have also included references to the hyperlinks for Ncube, Goodwin and He's code in the paper.

You keep referring to "our model" - is this not just the FP(sic) model? If not please clarify how "your" model differs from FP(sic). Yet the authors persist in making the claim they do more than (badly) implement the model of Filipovic & Larsen, they continue to claim some degree of authorship: https://github.com/peilun-he/PDSim/blob/66939da6a1bf4de325486a2bc8adea4665a33289/README.md?plain=1#L336-L338

To clarify immediately, we have never claimed ownership of Filipovic and Larsson's model. Indeed, quoting from the above link:

"... the parameter $\gamma$ is set to zero. However, in our extended model, we introduce the flexibility for this mean-reversion parameter associated with the long-term factor to take on non-zero values. Arbitrage-free pricing..."

"our" only appeared once, where it was used in reference to the extended Schwartz-Smith model with $\gamma \ge 0$ as opposed to the original Schwartz and Smith (2000) model with $\gamma = 0$.

We have rectified this error by completely rephrasing the relevant paragraph. The paragraph (containing the quote with "our extended model" above) now reads as this.

2.

Related, as I noted previously, the authors steadfastly refused to replicate some existing results, see here, here, here, here and 3-months later here.

These two figures depict the paths of the futures' price processes simulated using Filipovic and Larsson (Polynomial Diffusion), and Schwartz and Smith's models. After this, in approximately 3 months, we reproduced Figure 1 from Schwartz and Smith's (2000) paper, using the estimates from this paper. The graph is titled 'Simulated Spot Price and long-term component (SS model)' and can be found under "Replicating Schwartz and Smith's Results" section. We have also produced the graph for 'Simulated Spot Price and long-term component (PD model)' based on Filipovic and Larsson's model. For comparison, this graph can also be found under "Replicating Schwartz and Smith's Results" section. Further, the joint plot of the Simulated Futures Prices obtained from the SS and PD models can be found in the section "Simulation Accuracy". We remain very thankful to the Reviewer for pointing out that the "PDSim" had a fault, it was unable to simulate the paths of spot prices from the original Schwartz-Smith's model with $\gamma=0$ due to indefiniteness caused by division on zero in the code; this problem has been rectified since then by adding a new function, specifically designed to handle this scenario. Further, thanks to the reference to Ncube (2010) shared by the Reviewer, we were able to download the data provided by the author on Matlab File Exchange. We sincerely apologise for not noticing the folder with the data earlier; we have now been able to replicate the following graph from Ncube (2010). The required estimates are obtained using the original SS(2000) model with $\gamma=0$ and the graph "Estimated spot price and equilibrium price" that can be viewed under "Replicating Schwartz and Smith's Results" section. The R code for producing this graph is given here.

3.

As I noted at the time I had to find an existing replication, despite earlier asking them this question directly, and ask they at least mimic that authors calculations. They were incapable of even that level of effort.

As we addressed, in the reply to 2, using the data from Ncube (2010) and the parameters estimates from Schwartz and Smith (2000), we replicated the figure from Schwartz and Smith (2000) with ($\gamma=0$) for the Estimated Spot Price and Equilibrium Spot Price. This graph is included in the section "Replicating Schwartz and Smith's Results" section.

4.

To make matters worse, they have (again) seen fit to refuse to acknowledge that prior art (replication) they are mimicing. While there is a reference in the readme file, nowhere in the body of their work do they cite and acknowledge the work of Moeti Ncube, and the role it played in helping them resolve their profound confusion about the objects they have been dealing with.

As we indicated above, we have now cited Ncube (2010)'s work in reference to Schwartz-Smith's original model, where the long-term equilibrium process represents a geometric Brownian motion (gBm). Additionally, we have included other references related to the implementation of algorithms for parameter estimation of Schwartz-Smith's extended model with $\gamma \ge 0$, where the long-term equilibrium process is an Ornstein-Uhlenbeck process. The relevant paragraph is here.

Kind regards, Peilun He, Nino Kordzakhia, Gareth Peters and Pavel Schevchenko

samhforbes commented 5 months ago

Hi all, I just wanted to provide something by way of update. We are looking at this internally, as I think we have reached the limits of what we can achieve without considerable further back and forth, which is not fair on the current reviewers. @taqtiqa-mark thanks for your review and comments - and these are all certainly noted, and I can see where the issues exist here. I'm really appreciative of the time and comments you've put in, and feel free to unsubscribe from theGitHub thread to avoid further notifications if you wish. @peilun-he thanks for the additional clarification. Sam

peilun-he commented 4 months ago

Dear @samhforbes , It has been around a month since we provided a detailed response to reviewers, providing all the requested items and changes as well as a detailed response letter explaining the changes and adjustments made as per the reviewers requests. If there is anything else we need to address? We are currently now waiting for your response to this revision. Thank you for your time and consideration. Kind regards, Peilun He

samhforbes commented 4 months ago

Hi @peilun-he I appreciate your patience. This submission has been discussed with various editors and EiCs at JOSS. In this case I’m afraid we are going to have to reject this manuscript at JOSS on the back of this review procedure. There’s clearly potential in this software, and given this has been a very long review I hope the feedback has been useful in improving the work. I would recommend submitting the work to another software journal in this case, and sincerely wish you the best of luck with it. Thank you to both reviewers for a thorough job.

samhforbes commented 4 months ago

@editorialbot reject

editorialbot commented 4 months ago

Paper rejected.