search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
712 stars 38 forks source link

[REVIEW]: ParaMonte: A high-performance serial/parallel Monte Carlo simulation library for C, C++, Fortran #2741

Closed whedon closed 3 years ago

whedon commented 3 years ago

Submitting author: @shahmoradi (Amir Shahmoradi) Repository: https://github.com/cdslaborg/paramonte Version: v1.5.1 Editor: @VivianePons Reviewer: @milancurcic, @williamfgc Archive: 10.5281/zenodo.4749957

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/f964b6e22c71515c310fbe3843ad4513"><img src="https://joss.theoj.org/papers/f964b6e22c71515c310fbe3843ad4513/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/f964b6e22c71515c310fbe3843ad4513/status.svg)](https://joss.theoj.org/papers/f964b6e22c71515c310fbe3843ad4513)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@milancurcic & @williamfgc, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @VivianePons know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @milancurcic

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @williamfgc

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 3 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @milancurcic, @williamfgc it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- None

MISSING DOIs

- 10.1088/0004-637x/766/2/111 may be a valid DOI for title: A Multivariate Fit Luminosity Function and World Model for Long Gamma-Ray Bursts
- 10.1093/mnras/stv714 may be a valid DOI for title: Short versus long gamma-ray bursts: a comprehensive study of energetics and prompt gamma-ray correlations
- 10.1007/s11222-006-9438-0 may be a valid DOI for title: DRAM: efficient adaptive MCMC
- 10.1093/bioinformatics/btp162 may be a valid DOI for title: GNU MCSim: Bayesian statistical inference for SBML-coded systems biology models
- 10.18637/jss.v035.i04 may be a valid DOI for title: PyMC: Bayesian stochastic modelling in Python

INVALID DOIs

- None
williamfgc commented 3 years ago

@whedon @VivianePons I have trouble when trying to accept the invitation above. Also, I can't edit my checklist. I'd appreciate help with it as I'm currently reviewing the software.

whedon commented 3 years ago

I'm sorry human, I don't understand that. You can see what commands I support by typing:

@whedon commands
williamfgc commented 3 years ago

@whedon commands

whedon commented 3 years ago

Here are some things you can ask me to do:

# List Whedon's capabilities
@whedon commands

# List of editor GitHub usernames
@whedon list editors

# List of reviewers together with programming language preferences and domain expertise
@whedon list reviewers

EDITORIAL TASKS

# Compile the paper
@whedon generate pdf

# Compile the paper from alternative branch
@whedon generate pdf from branch custom-branch-name

# Ask Whedon to check the references for missing DOIs
@whedon check references

# Ask Whedon to check repository statistics for the submitted software
@whedon check repository
williamfgc commented 3 years ago

@whedon list reviewers

whedon commented 3 years ago

Here's the current list of reviewers: https://bit.ly/joss-reviewers

VivianePons commented 3 years ago

@whedon re-invite @williamfgc as reviewer

VivianePons commented 3 years ago

That should fix the issue

whedon commented 3 years ago

OK, the reviewer has been re-invited.

@williamfgc please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

williamfgc commented 3 years ago

I have a few initial recommendations upon my review. I hope the authors find this helpful to transition ParaMonte from academic code to production quality software:

  1. I couldn't find the use of automated test frameworks for unit and functional tests. I'd recommend using ctest as the project has CMake support and create automated tests. It seems there are a set of tests under src/test, but they are not executed automatically. Also a Fortran unit testing framework could be a good alternative.
  2. CI: this project is missing continuous integration. I find it difficult to accept it without comprehensive testing when changes are made to the repo. CI builds trusts among users and developers.
  3. Contributing: it's not clear the contribution policy. This is tied to 1. and 2. and the lack of CI infrastructure (either GitHub Actions, CircleCI, Travis, on a single platform would be helpful, etc.).
  4. Installation: add support for CMake installation. While there is a large effort to support different platform via scripts, it would be beneficial to reuse CMake portable capabilities to build and install the library portably. This could be added to their CI.
  5. Code coverage: once CI is installed please add code coverage (e.g. Coverity, Codecov)
  6. Code documentation: it's hard to find API level documentation (e.g. doxygen) for internal functions so any new developer to the project can pick up the original author's intention. I see some files have it while others (SpecBase_Description_mod.f90)[https://github.com/cdslaborg/paramonte/blob/master/src/ParaMonte/SpecBase_Description_mod.f90] don't.
  7. The paper focuses too much on previous work publications, rather than exposing the merits of the software infrastructure and (most importantly) impact. I'd focus it on the community impact: use cases, important integration with other projects, etc.
  8. Distribution and packaging (optional): it would be good to have the option for standard distribution packages (conda, spack). Install scripts have to be constantly updated and maintained. Without proper CI, bugs could pile up pretty quickly.

I'd be happy to expand on the above if required, but I believe they are necessary to meet minimal software quality standards.

VivianePons commented 3 years ago

Hi @shahmoradi have you had any time to look at the comments made by @williamfgc ?

shahmoradi commented 3 years ago

@VivianePonsThank you for your reminder. I am working on it right now as we speak.

milancurcic commented 3 years ago

The items I checked off are okay. Few remain unchecked because of issues with installation.

I tested serial binary releases for Fortran with both GNU and Intel. Both worked as expected and I was able to build and run the example program.

I wasn't able to use the parallel releases because they require MPICH. I tend to work with OpenMPI so it'd be nice if binary releases with OpenMPI were available. I opened an issue for this.

I tried building from source which failed. I opened an issue for this.

There was also a slight issue with wording in ACKNOWLEDGMENT.md which make it sound like the license requires citing the software. I opened an issue for this.

I confirm and re-iterate points 1 and 3 by @williamfgc as important. Points 2, 5, and 8 I consider nice to have but not essential. Point 6 I couldn't confirm--I had no problem finding the API docs and found them thorough.

I suggest tackling these before proceeding with the review.

williamfgc commented 3 years ago

@milancurcic thanks for confirming. I modified some points to make them clear based on your comments.

VivianePons commented 3 years ago

Hi all, @shahmoradi can I ask where we are on this? (No rush, I just want to make sure we are still on track!)

shahmoradi commented 3 years ago

I have addressed @williamfgc comments and I am working on @milancurcic 's comments. I should be able to back with a full revision by tomorrow, hopefully.

shahmoradi commented 3 years ago

@williamfgc

Thank you for your extensive detailed feedback, which significantly improved the quality of this codebase. We will address each of your comments below.

  1. The library tests.
    The library originally had a significant number of tests that used to be automatically executed. But at some point, someone (perhaps myself) decided to temporarily disable the tests, which then became permanent since nobody reactivated the tests.

    We have now added a comprehensive set of 866 new unique tests that cover nearly 100% of the entire library as well as the various functionalities of the samplers in the library. Given the three possible major parallelism configurations of the library, there are overall 3 * 866 = 2598 tests for all serial as well as the MPI and Coarray parallel builds. The code coverage report for all three builds for the most recent version of the library is available here:

    Since the library relies on preprocessing directives to implement different parallelism paradigms (serial, MPI, and Coarray Fortran), the tests had to be also adapted and preprocessed according to the characteristics of the library for different parallelism paradigms. No testing framework that we have inspected so far, met the minimum requirements for testing this package under the different build configurations. We have therefore developed a unit testing framework that is specifically tailored for the needs of this library. The tests in the library are separated into two different categories:

    1. basic: Tests that verify the functionality and accuracy of the fundamental backbone modules and procedures of the library.
    2. sampler: Tests that verify the functionality, semantics, and accuracy of the library's samplers.

    By default both categories of tests are deactivated for any particular build for the following reasons:

    1. The tests can take significant time and memory, in particular, when building the library for multiple different configurations. This can also easily break Continuous Integration workflows like Travis CI that limit the maximum length of the stdout log file.
    2. Some of the tests such as those that verify the restart functionality of the library's sampler in parallel require external intervention in the regular working of the samplers. This in turn requires the MPI/Coarray processes to be able to communicate the error messages with each other to avoid deadlocks. These error message communications between different processes are not normally needed as the failing process automatically calls mpi_abort() in MPI or error stop in Coarray to halt the entire program.
      However, such exceptions and errors should be gracefully handled during testing, which requires the addition of extra error handling code snippets to the sampler routines that are otherwise deactivated in normal production runs.
      These extra exception and error handling communications in parallel modes can be quite expensive if they appear in the parallel production code. As such, the library's tests are by default switched off, unless the developer (or the user) readily but explicitly enables the tests via the -t all or --test all or --codecov build flag. For example, 
      ./install.sh --lang fortran --build testing --lib dynamic --par none --test all

      will build the library for usage from the Fortran language with all tests activated. The tests can be also limited to one category, if desired, to reduce the testing and build time. This is particularly needed during the development of the library when the focus is on one particular functionality in the library. For example, 

      ./install.sh --lang c --build testing --lib dynamic --par mpi -t basic

      will test the library build for parallel usage from the C language and will only activate the basic tests of the library. The default mode, as mentioned above, is, 

      ./install.sh -t none

      However, when generating code coverage, all tests are activated by default. For example, 

      ./install.sh --codecov

      and, 

      ./install.sh --codecov -t all

      are equivalent and, 

      ./install.sh --codecov -t none

      leads to an error message, since a minimal set of tests are required to generate the code coverage report.

  2. Continuous Integration.
    We have now enabled Travis CI for this library. All builds are successful. Due to technical difficulties with Coarray Fortran and Windows, CAF parallel builds and builds for Windows are currently excluded from Travis CI. Intel has recently released its new version of Fortran compilers free of charge for all platforms including Windows. This will hopefully enable the testing and CI of this library also on Windows in the near future.

  3. Contribution policy.
    We have now laid out a set of guidelines for contributing to the core of library or other contributions:

  4. CMake support.
    CMake is indeed already supported on Linux and macOS. However, direct use of CMake to build the library is suboptimal, in particular, for the end-users as CMake cannot install missing libraries on the user's system if needed. One of our primary goals in the development of this package was to make a compiled-language library as accessible as possible to anyone, even those who do not have experience with compiled languages or CMake or different library dependencies. To achieve this goal, we had to develop multiple layers of build scripts that lay the ground for CMake to successfully build the library. The current build scripts are capable of automatically installing all missing components including the CMake application (if the minimum required version is not detected on the system), GNU compilers, as well as OpenMPI, MPICH MPI, and OpenCoarrays libraries.

    The situation on Windows is slightly more complex. While library build with CMake was originally supported on Windows, we eventually dropped it in favor of developing a home-grown separate Windows Batch build system for the library exclusively on Windows. The reason for this decision was CMake's poor support of the Fortran programming language at the time, especially on Windows. The CMake has dramatically improved over the past three years and given the recent availability of Intel compilers on Windows free of charge, CMake support for Windows is now among top priorities to add to the library.

  5. Code coverage.
    Code coverage reports are now added for every new release of the library and are permanently available on GitHub in a separate repository for the three most different builds of the library (the three parallelism paradigms):

    We have strived for adding an automated code coverage report via codecov but so far in vain. We will continue our efforts to generate automated code coverage reports and analyses via codecov/TravisCI or similar platforms.

  6. Documentation.

  7. Manuscript's contents and focus.
    We have now significantly reduced the amount of technical contents of the paper.

  8. Package redistribution and CI.
    We absolutely agree that CI is necessary for the success and longevity of this repository and we have now set up an automated build and testing workflow with Travis-CI. We also agree that having a standard distribution method is essential in the long term. There have been, however, a few challenges along our way on identifying and setting up the best method of distribution, most importantly, the many different configurations with which the library can be built, such as the programming languages, the MPI library dependencies, etc. Furthermore, there is the question of which packaging and redistribution method would be the best. On Windows and macOS, the task might be rather straightforward as only a few popular methods exist. The situation, however, is completely different on Linux OS.

    Currently, the users can access the prebuilt versions of the library for almost any production configuration they might need. The prebuilt libraries are always available on the GitHub release page of the project. Downloading and using these prebuilt libraries takes only three steps from the bash terminal. For example, the prebuilt library for the C programming language can be downloaded and run via, 

    libname=libparamonte_c_linux_x64_intel_release_dynamic_heap
wget https://github.com/cdslaborg/paramonte/releases/latest/download/$libname.tar.gz
tar xvzf $libname.tar.gz && cd $libname && ./build.sh && ./run.sh && cd ..

@milancurcic

Thank you for your valuable feedback, which significantly improved the quality of this work. Below, we will address your comments and questions,

  1. OpenMPI releases.
    Thank you for pointing this out. As you mentioned, OpenMPI is a popular implementation. We have now added the prebuilt versions of the library with OpenMPI in addition to MPICH. This required a complete revision of the naming convention used in the library across all languages and documentations. The new naming convention automatically suffixes,

    1. any MPICH build of the library with _mpich,
    2. any OpenMPI build of the library with _openmpi,
    3. any Intel MPI build of the library with _impi,
      A complete description of the naming convention used in the library is provided in the library's documentation.

  2. Build failure.
    Thank you for pointing this out. This apparently happens on systems where either the MPI library or the GNU compilers are already installed and meet the minimum version requirements of ParaMonte. As such, the build scripts would not feel the need to create a custom setup.sh to set up the environmental variables. However, it would require the setup file's existence, which would subsequently interrupt the build by throwing an exception.

    We believe this issue is now fixed in the latest release of the library.

  3. license and ACKNOWLEDGMENT.md.
    Thank you again for pointing this out. The existing ACKNOWLEDGMENT.md was a relic of the original license of the library. This is now fixed.

  4. points 1 and 3 by @williamfgc.
    Please see our responses to comments by @williamfgc.

VivianePons commented 3 years ago

Thank you very much, I am glad to see that the review process has been so successful and has improved the quality of the software :) I leave it to @williamfgc and @milancurcic to review your comments and the improvements that have been made

Happy new year to all of you and thank your for the provided work

shahmoradi commented 3 years ago

@milancurcic @williamfgc Thank you again for your constructive feedback so far. Just to make sure we have not missed any of your comments, are there any other outstanding issues needed to address here?

williamfgc commented 3 years ago

Dear @shahmoradi , thanks for the hard work on getting these points addressed. Would you kindly send an updated version of your manuscript? I'd like to wrap up the review as well.

shahmoradi commented 3 years ago

@williamfgc Thank you. My impression was that @whedon would automatically build the PDF from the Markdown file. Until then, I attach a PDF of the revised manuscript to this comment. The Markdown file on GitHub is also readable, although not as comfortable as the PDF.

VivianePons commented 3 years ago

@whedon generate pdf

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

williamfgc commented 3 years ago

@shahmoradi I just reviewed the revised manuscript and looks appropriate for JOSS. Thanks for addressing the changes and fixing the defect I filed above. Minor thing: Funding and Acknowledgment can be one single section on the paper. Looks good to me, the only one thing I still think could be added in the future is CI, but I'll leave it to @VivianePons as the editor and @milancurcic to decide on final publication. Thanks!

milancurcic commented 3 years ago

Good to go from me, thank you all. :+1:

A note regarding "Functionality" and why I left it unchecked: I was reading through the docs and played for a while with the example program and configurations, however I wasn't able to get far, even with the docs. I believe this was due to the domain-specific knowledge that I don't have. But I assume that for a domain expert, these parameters are meaningful and they wouldn't have trouble generalizing the example application while following the docs.

VivianePons commented 3 years ago

Hi thank you both @williamfgc and @milancurcic for the thorough reviews and @shahmoradi for all the improvements.

@williamfgc you left many boxes unchecked even though I understand that you are satisfied with the software. Maybe that's just an oversight? Could you go through the list and see if everything checks (and indeed CI is not required even though it's always good to have)

@milancurcic is there any functionality that you were not able to reproduce? Or anything you wanted to try and were not able to do? Even if you feel you're not a domain expert, your input might be useful for proving the documentation for example (and as an editor, I would really much like seeing this "functionality" box checked to get the paper published!) Also, you left unchecked the "Performance" box (even there is no performance claims in the paper, it should be checked)

Again thank you for your time! I believe we are getting close to final publication!

williamfgc commented 3 years ago

@VivianePons yes, it's just an oversight from my side. I just went through the list and marked what is completed. Thanks for double checking and leading the review process.

milancurcic commented 3 years ago

@VivianePons

Also, you left unchecked the "Performance" box (even there is no performance claims in the paper, it should be checked)

Sorry, I misunderstood the prompt. Checked!

@milancurcic is there any functionality that you were not able to reproduce? Or anything you wanted to try and were not able to do?

No, sorry, I wasn't clear. Everything I tried seems to work and do things. What I struggled with was fully understanding what it all means, which is the domain-specific part. I will check the "Functionality" box to the best of my knowledge, however, I believe it will take a domain-expert to truly confirm that the software does what it says. I know and trust the Authors so I feel comfortable checking this box.

VivianePons commented 3 years ago

Thanks @milancurcic for the clarification, I will keep it in mind.

@williamfgc please let us know if you see extra changes to be made to the software / paper or if you have any suggestions while finishing the review

williamfgc commented 3 years ago

@VivianePons it's all good from my side, the points in my original review were addressed. Thanks!

VivianePons commented 3 years ago

Thank you very much @williamfgc ! I see that you have left unchecked the Performance box (it need to be checked if there are no performance claims in the paper) and also "automated tests" and "Community guidelines", is there anything to be done still?

Also, @milancurcic was not completely confident checking the Functionality box mostly because he was not a domain expert and did not fully understand the functionality of the software. Can you tell me to what extent you were able to follow on your end?

williamfgc commented 3 years ago

@VivianePons I checked the boxes and put some follow up comments in bold. I don't want to be the bottleneck for final publication :) . Thanks for guiding the JOSS review process, I enjoyed it.

VivianePons commented 3 years ago

Thank you very much! And thanks again @milancurcic These have been very thorough review and I believe the quality of the software has greatly increased

I will now start the final steps towards publication

VivianePons commented 3 years ago

@whedon generate pdf

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

VivianePons commented 3 years ago

@whedon check references

whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.3847/1538-4357/abb9b7 is OK

MISSING DOIs

- 10.1088/0004-637x/766/2/111 may be a valid DOI for title: A Multivariate Fit Luminosity Function and World Model for Long Gamma-Ray Bursts
- 10.1093/mnras/stv714 may be a valid DOI for title: Short versus long gamma-ray bursts: a comprehensive study of energetics and prompt gamma-ray correlations
- 10.3847/1538-4357/abb9b7 may be a valid DOI for title: A Multilevel Empirical Bayesian Approach to Estimating the Unknown Redshifts of 1366 BATSE Catalog Long-Duration Gamma-Ray Bursts
- 10.3847/1538-4357/abb9b7 may be a valid DOI for title: A Multilevel Empirical Bayesian Approach to Estimating the Unknown Redshifts of 1366 BATSE Catalog Long-Duration Gamma-Ray Bursts
- 10.1007/s11222-006-9438-0 may be a valid DOI for title: DRAM: efficient adaptive MCMC
- 10.1093/bioinformatics/btp162 may be a valid DOI for title: GNU MCSim: Bayesian statistical inference for SBML-coded systems biology models
- 10.18637/jss.v035.i04 may be a valid DOI for title: PyMC: Bayesian stochastic modelling in Python

INVALID DOIs

- None
VivianePons commented 3 years ago

Hi @shahmoradi

I believe you're missing a word in this sentence "the library was as an open-source project" line 72 of the PDF.

Also, could you check the missing DOI that were flagged by whedon?

Thank you

shahmoradi commented 3 years ago

Good to go from me, thank you all. 👍

A note regarding "Functionality" and why I left it unchecked: I was reading through the docs and played for a while with the example program and configurations, however I wasn't able to get far, even with the docs. I believe this was due to the domain-specific knowledge that I don't have. But I assume that for a domain expert, these parameters are meaningful and they wouldn't have trouble generalizing the example application while following the docs.

@milancurcic Thanks for pointing this out. Such feedbacks are highly valuable, Could you tell me which examples you tried, so that we review them again for accuracy and comprehensibility? Thanks again.

shahmoradi commented 3 years ago

@shahmoradi I just reviewed the revised manuscript and looks appropriate for JOSS. Thanks for addressing the changes and fixing the defect I filed above. Minor thing: Funding and Acknowledgment can be one single section on the paper. Looks good to me, the only one thing I still think could be added in the future is CI, but I'll leave it to @VivianePons as the editor and @milancurcic to decide on final publication. Thanks!

Thank you @williamfgc . Your comments along with @milancurcic 's were highly valuable and crucial to making this a quality software, although there is still ample room for perfection. I will include your minor revision requests in the final draft.

shahmoradi commented 3 years ago

@VivianePons Thank you for your corrections. I confirm that the DOIs are correct to the best of my knowledge. I have also revised the manuscript based on the latest corrections requested by @VivianePons and @williamfgc. If there is anything else to implement, please let me know. Thank you again for your excellent edit and reviews.

VivianePons commented 3 years ago

@whedon check references

whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.3847/1538-4357/abb9b7 is OK

MISSING DOIs

- 10.1088/0004-637x/766/2/111 may be a valid DOI for title: A Multivariate Fit Luminosity Function and World Model for Long Gamma-Ray Bursts
- 10.1093/mnras/stv714 may be a valid DOI for title: Short versus long gamma-ray bursts: a comprehensive study of energetics and prompt gamma-ray correlations
- 10.3847/1538-4357/abb9b7 may be a valid DOI for title: A Multilevel Empirical Bayesian Approach to Estimating the Unknown Redshifts of 1366 BATSE Catalog Long-Duration Gamma-Ray Bursts
- 10.3847/1538-4357/abb9b7 may be a valid DOI for title: A Multilevel Empirical Bayesian Approach to Estimating the Unknown Redshifts of 1366 BATSE Catalog Long-Duration Gamma-Ray Bursts
- 10.1007/s11222-006-9438-0 may be a valid DOI for title: DRAM: efficient adaptive MCMC
- 10.1093/bioinformatics/btp162 may be a valid DOI for title: GNU MCSim: Bayesian statistical inference for SBML-coded systems biology models
- 10.18637/jss.v035.i04 may be a valid DOI for title: PyMC: Bayesian stochastic modelling in Python

INVALID DOIs

- None
VivianePons commented 3 years ago

@whedon generate pdf

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

VivianePons commented 3 years ago

Hi @shahmoradi what I am asking is to add those DOI to your paper in your bibliography so they are not flagged at "Missing DOI" by whedon. I checked only the first one and it's indeed the DOI for this cited article in your paper

danielskatz commented 3 years ago

👋 @shahmoradi - this is either waiting for your action, or for you to tell the editor that you have completed it.