Submitting author: @ilyamandel (Ilya Mandel) Repository: https://github.com/TeamCOMPAS/COMPAS Version: v02.27.00 Editor: @christinahedges Reviewer: @katiebreivik, @HeloiseS Archive: 10.5281/zenodo.5847518

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/ef98e783b7fa1c269179100e48f270f4"><img src="https://joss.theoj.org/papers/ef98e783b7fa1c269179100e48f270f4/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/ef98e783b7fa1c269179100e48f270f4/status.svg)](https://joss.theoj.org/papers/ef98e783b7fa1c269179100e48f270f4)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@katiebreivik & @HeloiseS, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

Make sure you're logged in to your GitHub account
Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @christinahedges know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @katiebreivik

✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@ilyamandel) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] #3952
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[ ] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @HeloiseS

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@ilyamandel) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @katiebreivik, @HeloiseS it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

PDF failed to compile for issue #3838 with the following error:

 Can't find any papers to compile :-(

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.87 s (211.7 files/s, 299613.3 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
HTML                            16           1118             64         191788
C++                             34           4236           8418          12755
C/C++ Header                    38           2516           2225           9642
Python                          27           2251           2489           7881
TeX                             29           1536            464           5056
Jupyter Notebook                22              0           6075           1429
Markdown                        13            549              0           1006
make                             1             24             12             96
YAML                             3             22              4             87
Bourne Shell                     1              4              6             16
Dockerfile                       1              8              5             15
-------------------------------------------------------------------------------
SUM:                           185          12264          19762         229771
-------------------------------------------------------------------------------

Statistical information for the repository '44bbb039a5d860687e27dc42' was
gathered on 2021/10/20.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Alejandro Vigna-Gome            13           117            135            0.15
COMPAS                           2             4              3            0.00
Cjid                             1           181             88            0.16
Debatri                          5           111            110            0.14
FloorBroekgaarden                1             2              0            0.00
Ilya Mandel                     11           102             38            0.09
Lieke van Son                   37          1847            318            1.33
Lokesh Khandelwal                2            24              5            0.02
Mike Lau                        20          1761           1337            1.90
Ray Seikel                       1           413            287            0.43
Reinhold Willcox               200         22075          20193           25.91
Simon Stevenson                 18           138             86            0.14
Tim                              3            16              3            0.01
Tom Wagg                        65          5576            968            4.01
cneijssel                        1          2004            219            1.36
floor broekgaarden              11          2641           2274            3.01
ilyamandel                      95          2042           3281            3.26
imandel                         23           338            131            0.29
jeffriley                      148         32292          24576           34.86
jriley                          20         35762           1616           22.92

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Alejandro Vigna-Gome         48           41.0         14.3               18.75
Debatri                      15           13.5         18.6               53.33
Floor Broekgaarden            2          100.0          0.9               50.00
Ilya Mandel                  90           88.2          0.5               11.11
Lieke van Son              1597           86.5          2.3               12.96
Lokesh Khandelwal             4           16.7         22.7                0.00
Mike Lau                    249           14.1         12.3               15.26
Ray Seikel                    2            0.5         13.2                0.00
Reinhold Willcox           3975           18.0         13.3               22.57
Simon Stevenson              62           44.9         13.9               16.13
Tim                          12           75.0         15.5                0.00
Tom Wagg                   4929           88.4          4.5                8.99
cneijssel                  1514           75.5         19.0               19.02
floor broekgaarden           83            3.1         21.0                8.43
ilyamandel                 1201           58.8         13.8               21.23
jeffriley                 15768           48.8         11.0               24.32
jriley                    22862           63.9          1.4               26.62

@whedon generate pdf from branch JOSS

Attempting PDF compilation from custom branch JOSS. Reticulating splines etc...

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

@whedon check references from branch JOSS

Attempting to check references... from custom branch JOSS

@ilyamandel, @katiebreivik , @HelioseS – This is the review thread for the paper. Please don't hesitate to message me here if you have questions.

Please read the "Reviewer instructions & questions" in the first comment above to get started. If you get lost, you can also see the reviewer guidelines.

Both reviewers have checklists at the top of this thread (in that first comment) with the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. If you are concerned about a requirement, please discuss it here on this thread 🧵 . Feel free to post about questions/concerns as they come up as you go through your review.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention this issue (#3838) so that a link is created to this thread (and I can keep an eye on what is happening).

We aim for the review process to be completed within about 4-6 weeks but please make a start well ahead of this as JOSS reviews are by their nature iterative and any early feedback you may be able to provide to the author will be very helpful in meeting this schedule. When you're finished with your checklist, leave a comment and @ me to let everyone know your review is complete.

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1017/S0305004100021150 is OK
- 10.1086/171341 is OK
- 10.1086/176813 is OK
- 10.3847/1538-4357/aab34c is OK
- 10.1088/0264-9381/32/7/074001 is OK
- 10.1088/0264-9381/32/2/024001 is OK
- 10.1103/PhysRevLett.18.379 is OK
- 10.1007/BF00651498 is OK
- 10.1093/mnras/stv1161 is OK
- 10.1093/mnras/stx1576 is OK
- 10.1093/mnras/stx2933 is OK
- 10.1051/0004-6361/201629420 is OK
- 10.1093/mnras/sty2190 is OK
- 10.1103/RevModPhys.74.1015 is OK
- 10.1038/nature06333 is OK
- 10.3847/1538-4357/836/2/244 is OK
- 10.3847/1538-4357/836/2/244 is OK
- 10.3847/1538-4357/aad044 is OK
- 10.1038/nature24030 is OK
- 10.1038/nature10095 is OK
- 10.1038/nature11521 is OK
- 10.1103/PhysRevLett.116.061102 is OK
- 10.1103/PhysRevLett.119.161101 is OK
- 10.1088/0264-9381/33/13/134001 is OK
- 10.1086/513562 is OK
- 10.1051/0004-6361/201628980 is OK
- 10.1038/nature18322 is OK
- 10.3847/2041-8205/824/1/L8 is OK
- 10.1093/mnras/stw1772 is OK
- 10.1093/mnras/sty2714 is OK
- 10.3847/2041-8213/aa9bf6 is OK
- 10.1038/nature23453 is OK
- 10.3847/2041-8213/aad800 is OK
- 10.1093/mnras/stw379 is OK
- 10.1051/0004-6361/201628133 is OK
- 10.1126/science.1223344 is OK
- 10.1088/0067-0049/215/1/15 is OK
- 10.1146/annurev-astro-081811-125615 is OK
- 10.3847/1538-4357/aa6af9 is OK
- 10.1086/500363 is OK
- 10.1088/0067-0049/208/2/19 is OK
- 10.1051/0004-6361/201525830 is OK
- 10.1086/319719 is OK
- 10.1088/0004-637X/749/1/91 is OK
- 10.1088/0067-0049/192/1/3 is OK
- 10.1088/0067-0049/220/1/15 is OK
- 10.3847/1538-4365/aaa5a8 is OK
- 10.3847/1538-4357/835/2/165 is OK
- 10.1093/mnras/stw2260 is OK
- 10.3847/1538-4357/aa6f5e is OK
- 10.3847/1538-4357/aacea4 is OK
- 10.3847/2041-8213/aa5caa is OK
- 10.1093/mnras/sty3193 is OK
- 10.1103/PhysRevLett.121.161103 is OK
- 10.1103/PhysRevLett.116.201301 is OK
- 10.1093/mnras/104.5.273 is OK
- 10.1086/190103 is OK
- 10.1038/364421a0 is OK
- 10.1038/364423a0 is OK
- 10.1086/165412 is OK
- 10.1086/160960 is OK
- 10.1086/513736 is OK
- 10.1093/mnras/stw2786 is OK
- 10.1093/mnras/stx1430 is OK
- 10.1086/168974 is OK
- 10.1086/161701 is OK
- 10.1088/0004-637X/716/1/114 is OK
- 10.1088/0004-637X/743/1/49 is OK
- 10.1103/PhysRevD.95.103010 is OK
- 10.3847/2041-8205/824/1/L12 is OK
- 10.3847/2041-8213/aa7045 is OK
- 10.1103/PhysRevD.93.084029 is OK
- 10.1103/PhysRevD.95.124046 is OK
- 10.1103/PhysRevD.96.024058 is OK
- 10.1103/PhysRevLett.122.011101 is OK
- 10.3847/2041-8205/832/1/L2 is OK
- 10.1103/PhysRevLett.120.151101 is OK
- 10.1093/mnras/stw1942 is OK
- 10.1088/0004-637X/810/1/58 is OK
- 10.1038/ncomms14906 is OK
- 10.1093/mnras/stx1764 is OK
- 10.1093/mnras/sty2463 is OK
- 10.3847/2041-8213/ab1bdf is OK
- 10.1086/147659 is OK
- 10.1086/107290 is OK
- 10.1051/0004-6361/201936528 is OK
- 10.1093/mnras/sty908 is OK
- 10.3847/2041-8213/aa9f0c is OK
- 10.1038/nature12569 is OK
- 10.3847/0004-637X/831/2/144 is OK
- 10.1038/nature08579 is OK
- 10.1038/s41550-018-0568-z is OK
- 10.1126/science.1203601 is OK
- 10.1093/mnras/stw273 is OK
- 10.1051/0004-6361/201629685 is OK
- 10.3847/1538-4357/aa8408 is OK
- 10.1103/PhysRevD.97.043014 is OK
- 10.1088/0067-0049/190/1/1 is OK
- 10.3847/1538-4365/aa6fb6 is OK
- 10.1111/j.1365-2966.2004.07446.x is OK
- 10.1051/0004-6361/201220007 is OK
- 10.1051/0004-6361/201322714 is OK
- 10.1088/0004-637X/782/1/7 is OK
- 10.1088/0004-637X/814/1/58 is OK
- 10.1088/0004-637X/805/1/20 is OK
- 10.1051/0004-6361/201833025 is OK
- 10.1051/0004-6361/201016113 is OK
- 10.1051/0004-6361/201526617 is OK
- 10.1093/mnras/stz2558 is OK
- 10.1086/168190 is OK
- 10.1093/mnras/281.1.257 is OK
- 10.1093/mnras/291.4.732 is OK
- 10.1046/j.1365-8711.1998.01658.x is OK
- 10.1046/j.1365-8711.2002.05038.x is OK
- 10.1046/j.1365-8711.2001.04220.x is OK
- 10.1088/0004-637X/714/2/1217 is OK
- 10.1051/0004-6361/201935332 is OK
- 10.1088/978-0-7503-1278-3 is OK
- 10.1051/0004-6361:20010127 is OK
- 10.1086/133478 is OK
- 10.1086/153315 is OK
- 10.1051/0004-6361/201730698 is OK
- 10.1093/mnras/staa549 is OK
- 10.1093/mnras/stx3174 is OK
- 10.1086/150487 is OK
- 10.1038/369127a0 is OK
- 10.1093/mnras/291.3.569 is OK
- 10.1086/338805 is OK
- 10.1007/s12036-017-9474-5 is OK
- 10.1093/mnras/stw1275 is OK
- 10.1093/mnras/sty2230 is OK
- 10.1093/mnras/sty3087 is OK
- 10.1103/PhysRev.136.B1224 is OK
- 10.1051/0004-6361/201322068 is OK
- 10.3847/1538-3881/aabc4f is OK
- 10.1051/0004-6361/201525945 is OK
- 10.1051/0004-6361/201629612 is OK
- 10.1111/j.1365-2966.2012.21549.x is OK
- 10.1093/mnras/stt1106 is OK
- 10.1093/mnras/stv2733 is OK
- 10.1051/0004-6361:20000147 is OK
- 10.1093/mnras/stx027 is OK
- 10.3847/2041-8213/aaa28c is OK
- 10.1086/147589 is OK
- 10.1143/PTP.30.460 is OK
- 10.1093/mnras/stv3002 is OK
- 10.1051/0004-6361/201117769 is OK
- 10.1088/0004-637X/748/1/42 is OK
- 10.1393/ncr/i2016-10120-8 is OK
- 10.1093/mnras/stt213 is OK
- 10.1126/science.aat6506 is OK
- 10.1093/mnras/stv776 is OK
- 10.1093/mnras/stx816 is OK
- 10.1088/0264-9381/31/15/155005 is OK
- 10.1093/mnras/sty983 is OK
- 10.1093/mnras/sty2249 is OK
- 10.1093/mnras/stv3002 is OK
- 10.3847/1538-4357/ab3981 is OK
- 10.1086/421551 is OK
- 10.1093/mnras/203.4.1049 is OK
- 10.1103/PhysRevD.52.821 is OK
- 10.1086/513603 is OK
- 10.1086/516712 is OK
- 10.1103/PhysRevLett.98.231101 is OK
- 10.1103/PhysRevD.83.024037 is OK
- 10.1086/591218 is OK
- 10.1093/mnras/staa1417 is OK
- 10.1093/mnras/stt1268 is OK
- 10.1146/annurev-astro-081309-130834 is OK
- 10.1086/118116 is OK
- 10.1007/BF00638971 is OK
- 10.1088/0004-637X/769/2/109 is OK
- 10.1093/mnras/sty306 is OK
- 10.1111/j.1365-2966.2010.17167.x is OK
- 10.1111/j.1365-2966.2012.21672.x is OK
- 10.3847/1538-4357/ab7014 is OK
- 10.1051/0004-6361/201936204 is OK
- 10.1103/PhysRevLett.32.324 is OK
- 10.1086/310296 is OK
- 10.1038/nature09466 is OK
- 10.1126/science.1233232 is OK
- 10.1088/0004-637X/741/2/103 is OK
- 10.1088/0004-637X/783/1/10 is OK
- 10.1088/0004-637X/757/1/69 is OK
- 10.3847/0004-637X/818/2/124 is OK
- 10.1093/mnras/staa059 is OK
- 10.1146/annurev-astro-081915-023322 is OK
- 10.1086/431543 is OK
- 10.1093/mnras/stw1684 is OK
- 10.1093/mnras/stx2577 is OK
- 10.1051/0004-6361/201832839 is OK
- 10.1146/annurev.astro.46.060407.145222 is OK
- 10.1086/308158 is OK
- 10.1086/375341 is OK
- 10.1146/annurev-astro-082708-101737 is OK
- 10.1017/pasa.2017.52 is OK
- 10.1086/164809 is OK
- 10.1086/160871 is OK
- 10.1093/mnras/274.2.461 is OK
- 10.1086/177974 is OK
- 10.1146/annurev-astro-081913-040025 is OK
- 10.1111/j.1365-2966.2009.14506.x is OK
- 10.1086/173033 is OK
- 10.3847/1538-4365/aab5b0 is OK
- 10.1103/PhysRevD.101.103021 is OK
- 10.1016/j.ppnp.2004.07.001 is OK
- 10.3847/1538-4365/aaca30 is OK
- 10.3847/1538-4357/ab6d77 is OK
- 10.1086/161749 is OK
- 10.3847/1538-4357/aadbae is OK
- 10.3847/1538-4357/aad2d2 is OK
- 10.3847/1538-4357/aab95f is OK
- 10.1051/0004-6361/201220636 is OK
- 10.1051/0004-6361/201220273 is OK
- 10.1051/0004-6361/201321986 is OK
- 10.1086/340494 is OK
- 10.1080/10556799308230566 is OK
- 10.1086/431786 is OK
- 10.1146/annurev-astro-081710-102602 is OK
- 10.1088/0067-0049/213/2/34 is OK
- 10.1086/145971 is OK
- 10.1093/mnrasl/slt164 is OK
- 10.1093/mnrasl/slz183 is OK
- 10.1017/pasa.2020.31 is OK
- 10.1093/mnras/stz3542 is OK
- 10.1088/0004-637X/744/1/12 is OK
- 10.3847/1538-4357/aa5729 is OK
- 10.1017/S1743921311011124 is OK
- 10.3847/2041-8213/aa91c9 is OK
- 10.1103/PhysRevD.93.112004 is OK
- 10.3847/1538-4357/ab1e4d is OK
- 10.3847/1538-4357/ab3b06 is OK
- 10.1086/312422 is OK
- 10.1093/mnras/stv139 is OK
- 10.1086/508914 is OK
- 10.1088/0004-637X/808/2/132 is OK
- 10.1088/0004-637X/755/2/89 is OK
- 10.1086/423264 is OK
- 10.1126/science.aan0106 is OK
- 10.1111/j.1365-2966.2004.07340.x is OK
- 10.1038/nature03293 is OK
- 10.1086/505169 is OK
- 10.1093/mnras/sty3087 is OK
- 10.1093/mnras/stw1772 is OK
- 10.1103/PhysRevD.100.064060 is OK
- 10.1093/mnras/stw379 is OK
- 10.1088/0004-637X/779/1/72 is OK
- 10.3847/0004-637X/819/2/108 is OK
- 10.1088/0004-637X/806/2/263 is OK
- 10.7935/GT1W-FZ16 is OK
- 10.3847/1538-4357/ab4c31 is OK
- 10.1088/0004-637X/722/2/1985 is OK
- 10.1093/mnras/stv852 is OK
- 10.1111/j.1365-2966.2004.08355.x is OK
- 10.1093/mnras/stv2659 is OK
- 10.1086/422901 is OK
- 10.1086/500363 is OK
- 10.3847/1538-4357/aa6af9 is OK
- 10.1146/annurev-astro-081811-125615 is OK
- 10.1038/nature10159 is OK
- 10.1086/166683 is OK
- 10.3847/1538-4357/aaae6b is OK
- 10.1038/nature11697 is OK
- 10.1093/mnrasl/slu022 is OK
- 10.1093/mnras/stw1652 is OK
- 10.3847/1538-4357/ab4fe5 is OK
- 10.3847/1538-4357/ab1b41 is OK
- 10.1093/mnras/stx1576 is OK
- 10.1051/0004-6361/201629420 is OK
- 10.1051/0004-6361:20010099 is OK
- 10.1111/j.1365-2966.2009.15123.x is OK
- 10.1086/466521 is OK
- 10.1086/312496 is OK
- 10.1086/172089 is OK
- 10.1051/0004-6361/202037710 is OK
- 10.1088/0004-637X/727/2/95 is OK
- 10.1103/PhysRevD.86.124007 is OK
- 10.1103/PhysRevD.98.081501 is OK
- 10.1088/0264-9381/30/12/123001 is OK
- 10.3847/2041-8213/aa920c is OK
- 10.1086/311680 is OK
- 10.1142/S0218271815300128 is OK
- 10.1103/PhysRevLett.119.161101 is OK
- 10.1088/0004-637X/716/1/615 is OK
- 10.1093/mnras/stz1150 is OK
- 10.1093/mnras/sty1999 is OK
- 10.1093/mnras/stz3190 is OK
- 10.1051/0004-6361/201526192 is OK
- 10.1051/0004-6361/201629844 is OK
- 10.1051/0004-6361:20021184 is OK
- 10.1088/0004-637X/796/2/121 is OK
- 10.1086/133478 is OK
- 10.1051/0004-6361:20010127 is OK
- 10.1146/annurev.astro.38.1.113 is OK
- 10.3847/1538-4357/ab45e6 is OK
- 10.1103/PhysRevD.100.043012 is OK
- 10.3847/1538-4357/ab3426 is OK
- 10.1093/mnras/stv3002 is OK
- 10.3847/1538-4357/836/2/244 is OK
- 10.1093/mnras/stw2562 is OK
- 10.1051/0004-6361/201423641 is OK
- 10.1051/0004-6361/201423447 is OK
- 10.3847/2041-8213/ab960f is OK
- 10.1038/s41550-020-1014-6 is OK
- 10.1103/PhysRevD.100.023007 is OK
- 10.1086/190103 is OK
- 10.1103/PhysRevX.9.031040 is OK
- 10.1093/mnras/stz226 is OK
- 10.3847/1538-4357/aab34c is OK
- 10.3847/2041-8213/aa9bf6 is OK
- 10.1051/0004-6361:20052862 is OK
- 10.1088/2041-8205/715/2/L138 is OK
- 10.1086/177505 is OK
- 10.1093/mnras/280.4.1035 is OK
- 10.1134/S1063772909100047 is OK
- 10.1088/0004-637X/759/1/52 is OK
- 10.1016/0370-1573(91)90064-S is OK
- 10.3847/1538-4357/aa634a is OK
- 10.1038/nature17425 is OK
- 10.1093/mnras/stz2840 is OK
- 10.1093/mnras/staa2177 is OK
- 10.1093/mnras/stz2057 is OK
- 10.1146/annurev-astro-081913-040025 is OK
- 10.1051/0004-6361:20077545 is OK
- 10.1080/01621459.2000.10473909 is OK
- 10.1086/153853 is OK
- 10.1103/PhysRevLett.116.201301 is OK
- 10.1086/375341 is OK
- 10.1086/154524 is OK
- 10.1146/annurev.aa.24.090186.001553 is OK
- 10.1093/mnras/sty1485 is OK
- 10.1038/nature18322 is OK
- 10.1111/j.1365-2966.2011.19019.x is OK
- 10.1093/mnrasl/sly063 is OK
- 10.1086/160871 is OK
- 10.1093/mnras/stw379 is OK
- 10.1051/0004-6361/201628133 is OK
- 10.1093/mnras/stw1219 is OK
- 10.1093/mnras/stv1161 is OK
- 10.1080/00401706.1995.10484303 is OK
- 10.1145/218380.218498 is OK
- 10.1038/nphoton.2013.177 is OK
- 10.1038/nature24471 is OK
- 10.1103/PhysRevD.97.041301 is OK
- 10.1088/0004-637X/812/1/40 is OK
- 10.1086/165412 is OK
- 10.1086/160960 is OK
- 10.3847/1538-3881/aabc4f is OK
- 10.1086/191508 is OK
- 10.1146/annurev.aa.21.090183.002015 is OK
- 10.1046/j.1365-8711.2000.03426.x is OK
- 10.1088/0004-637X/757/1/36 is OK
- 10.1051/0004-6361/201935842 is OK
- 10.1088/0004-637X/801/1/32 is OK
- 10.1051/0004-6361/201218966 is OK
- 10.1093/mnras/268.4.871 is OK
- 10.1051/0004-6361/201628193 is OK
- 10.1103/PhysRevD.96.023012 is OK
- 10.1086/307992 is OK
- 10.1126/science.aai8635 is OK
- 10.1046/j.1365-8711.2002.05745.x is OK
- 10.1088/0004-637X/804/2/148 is OK
- 10.1111/j.1365-2966.2012.21127.x is OK
- 10.1093/mnras/stx1634 is OK
- 10.1093/mnras/stw645 is OK
- 10.1093/mnrasl/slu183 is OK
- 10.1111/j.1365-2966.2007.11668.x is OK
- 10.1086/159942 is OK
- 10.1126/science.aai8635 is OK
- 10.1093/pasj/psv073 is OK
- 10.1086/430728 is OK
- 10.1093/pasj/52.3.499 is OK
- 10.1086/319125 is OK
- 10.1086/320343 is OK
- 10.1051/0004-6361:20020550 is OK
- 10.1111/j.1365-2966.2012.21881.x is OK
- 10.1093/mnrasl/slw218 is OK
- 10.1086/501516 is OK
- 10.1038/nature13791 is OK
- 10.1046/j.1365-8711.1999.02626.x is OK
- 10.1051/0004-6361/201730472 is OK
- 10.1088/0004-637X/725/1/496 is OK
- 10.1086/165716 is OK
- 10.1086/522073 is OK
- 10.1111/j.1365-2966.2005.09087.x is OK
- 10.1086/176778 is OK
- 10.1086/497331 is OK
- 10.1111/j.1365-2966.2008.13064.x is OK
- 10.1086/161749 is OK
- 10.1093/mnras/sty2190 is OK
- 10.1103/PhysRevD.89.084006 is OK
- 10.1103/PhysRevD.95.024010 is OK
- 10.1103/PhysRevD.93.044007 is OK
- 10.1103/PhysRevD.93.044006 is OK
- 10.1103/PhysRevLett.113.151101 is OK
- 10.1103/PhysRevD.47.2198 is OK
- 10.1017/S1743921317000059 is OK
- 10.1103/PhysRevC.58.1804 is OK
- 10.1086/374637 is OK
- 10.1086/307476 is OK
- 10.1038/nature24453 is OK
- 10.3847/1538-4357/aafe0e is OK
- 10.1046/j.1365-8711.2001.04022.x is OK
- 10.1088/0004-6256/142/6/197 is OK
- 10.3847/1538-4357/aa7e89 is OK
- 10.1093/mnras/stv1161 is OK
- 10.1093/mnras/stx2933 is OK
- 10.1093/mnras/stx2123 is OK
- 10.1126/science.aau4005 is OK
- 10.3847/1538-4357/ab498b is OK
- 10.1093/mnras/stv2462 is OK
- 10.1088/2041-8205/732/1/L17 is OK
- 10.1086/312343 is OK
- 10.1086/174525 is OK
- 10.1086/167702 is OK
- 10.1051/0004-6361/201833297 is OK
- 10.1088/0004-637X/708/1/9 is OK
- 10.3847/1538-4365/aaca30 is OK
- 10.1093/mnras/sty2035 is OK
- 10.1146/annurev-astro-081913-035926 is OK
- 10.1088/0004-637X/722/2/1946 is OK
- 10.1103/PhysRevLett.119.061101 is OK
- 10.1103/RevModPhys.29.547 is OK
- 10.1086/340304 is OK
- 10.1093/mnras/staa756 is OK
- 10.1093/mnras/stab973 is OK
- 10.1093/mnras/staa756 is OK
- 10.1086/518997 is OK
- 10.3847/1538-4357/ab518b is OK
- 10.3847/2041-8213/abbadd is OK
- 10.3847/1538-4357/ab9809 is OK
- 10.1088/0264-9381/32/10/105009 is OK
- 10.1103/PhysRevD.74.041501 is OK
- 10.1103/PhysRevD.98.084036 is OK
- 10.1088/0004-6256/145/1/8 is OK
- 10.1007/lrr-2016-1 is OK
- 10.1086/184741 is OK
- 10.1007/s41114-018-0012-9 is OK
- 10.1093/mnras/173.3.729 is OK
- 10.1088/2041-8205/807/2/L24 is OK
- 10.1017/S1743921317003209 is OK
- 10.1086/172058 is OK
- 10.1093/mnras/250.4.701 is OK
- 10.1093/mnras/stt1106 is OK
- 10.1086/340098 is OK
- 10.1086/338805 is OK
- 10.1038/253698a0 is OK
- 10.1007/s12036-017-9474-5 is OK
- 10.1051/0004-6361/201731518 is OK
- 10.1088/0004-637X/719/1/722 is OK
- 10.1111/j.1365-2966.2012.21549.x is OK
- 10.1093/mnras/stv2733 is OK
- 10.1086/341197 is OK
- 10.1086/421713 is OK
- 10.1093/mnras/stv2903 is OK
- 10.1088/2041-8205/798/1/L19 is OK
- 10.1111/j.1365-2966.2010.18147.x is OK
- 10.1086/305614 is OK
- 10.3847/1538-4357/ab9d85 is OK
- 10.1093/mnras/stw426 is OK
- 10.1093/mnras/sty2190 is OK
- 10.1086/521026 is OK
- 10.1088/0004-637X/814/1/58 is OK
- 10.1051/0004-6361:20066129 is OK
- 10.1051/0004-6361/200912827 is OK
- 10.1086/177423 is OK
- 10.1086/177974 is OK
- 10.1086/305085 is OK
- 10.1086/309400 is OK
- 10.1111/j.1467-9469.2011.00756.x is OK
- 10.1198/106186004X12803 is OK
- 10.3847/2041-8205/830/1/L18 is OK
- 10.1088/0004-637X/810/1/58 is OK
- 10.3847/2041-8205/832/1/L2 is OK
- 10.1093/mnras/stx1764 is OK
- 10.1088/1361-6382/aa552e is OK
- 10.3847/1538-4357/aa8408 is OK
- 10.3847/2041-8213/aad800 is OK
- 10.3847/1538-4357/ab1b21 is OK
- 10.1093/mnras/sty2234 is OK
- 10.1086/154524 is OK
- 10.1109/MCSE.2007.53 is OK
- 10.1109/MCSE.2011.37 is OK
- 10.1109/MCSE.2007.55 is OK
- 10.1103/PhysRevD.98.081501 is OK
- 10.1146/annurev.astro.45.051806.110615 is OK
- 10.1088/0264-9381/32/7/074001 is OK
- 10.1088/0264-9381/32/2/024001 is OK
- 10.1103/PhysRevLett.118.221101 is OK
- 10.1111/j.1365-2966.2006.10233.x is OK
- 10.1086/175268 is OK
- 10.1086/319641 is OK
- 10.1086/186143 is OK
- 10.1086/306265 is OK
- 10.1086/312475 is OK
- 10.1093/mnras/stx1430 is OK
- 10.1046/j.1365-8711.1999.02862.x is OK
- 10.1038/nature18322 is OK
- 10.1093/mnrasl/slw152 is OK
- 10.1093/mnras/291.3.569 is OK
- 10.1046/j.1365-8711.1998.01923.x is OK
- 10.1093/mnras/stx2123 is OK
- 10.3847/1538-4357/ab7335 is OK
- 10.1017/pasa.2017.51 is OK
- 10.1093/mnras/stu381 is OK
- 10.1038/nphys214 is OK
- 10.1093/mnrasl/slz183 is OK
- 10.1093/mnras/sty2714 is OK
- 10.1038/nature23453 is OK
- 10.1093/mnrasl/slv054 is OK
- 10.1088/0264-9381/27/11/114007 is OK
- 10.1086/375713 is OK
- 10.1086/307174 is OK
- 10.1086/430515 is OK
- 10.1086/187251 is OK
- 10.1111/j.1365-2966.2005.08997.x is OK
- 10.1086/340370 is OK
- 10.1046/j.1365-8711.2003.06616.x is OK
- 10.1111/j.1365-2966.2004.08373.x is OK
- 10.1093/mnras/stu1913 is OK
- 10.1086/505169 is OK
- 10.1086/339860 is OK
- 10.1111/j.1365-2966.2011.19019.x is OK
- 10.1086/306933 is OK
- 10.1126/science.1096986 is OK
- 10.1086/181803 is OK
- 10.1093/mnras/stt2503 is OK
- 10.1051/0004-6361/201937293 is OK
- 10.3847/0004-637X/831/2/190 is OK
- 10.1093/mnras/stv620 is OK
- 10.1093/mnras/stt037 is OK
- 10.1038/nature10365 is OK
- 10.1007/s41114-017-0006-z is OK
- 10.1155/2016/6341974 is OK
- 10.1051/0004-6361:20078841 is OK
- 10.3847/0004-637X/829/2/110 is OK
- 10.1088/0004-637X/775/2/113 is OK
- 10.1088/0004-637X/774/1/25 is OK
- 10.1088/0004-637X/775/1/18 is OK
- 10.1088/2041-8205/736/1/L21 is OK
- 10.1086/311680 is OK
- 10.1086/154860 is OK
- 10.1111/j.1365-2966.2010.16864.x is OK
- 10.1086/181612 is OK
- 10.1086/184740 is OK
- 10.1088/2041-8205/776/2/L31 is OK
- 10.1111/j.1365-2966.2012.20985.x is OK
- 10.1093/mnras/169.2.229 is OK
- 10.1146/annurev.aa.17.090179.001241 is OK
- 10.1088/0004-637X/807/2/115 is OK
- 10.1093/mnras/stu2404 is OK
- 10.1088/0004-637X/725/1/940 is OK
- 10.1103/PhysRevD.87.123524 is OK
- 10.1093/mnras/stz1651 is OK
- 10.1103/PhysRevD.93.084029 is OK
- 10.1093/mnras/stu871 is OK
- 10.1088/0264-9381/27/11/114106 is OK
- 10.1038/340126a0 is OK
- 10.1093/mnras/sts295 is OK
- 10.1093/mnras/stv1419 is OK
- 10.1088/1475-7516/2014/06/026 is OK
- 10.1093/mnras/sts295 is OK
- 10.1086/380561 is OK
- 10.1111/j.1365-2966.2007.12738.x is OK
- 10.1086/305614 is OK
- 10.1088/0004-637X/725/2/1918 is OK
- 10.1088/0004-637X/757/1/91 is OK
- 10.1088/0004-637X/741/2/103 is OK
- 10.1086/523620 is OK
- 10.1093/mnras/stv009 is OK
- 10.1046/j.1365-8711.1999.02437.x is OK
- 10.1093/mnras/stx2923 is OK
- 10.1093/mnras/stw1772 is OK
- 10.1051/0004-6361/201322198 is OK
- 10.1051/0004-6361/201220636 is OK
- 10.1093/mnras/stv1753 is OK
- 10.1088/0004-637X/717/2/724 is OK
- 10.1080/01621459.1949.10483310 is OK
- 10.2172/4423221 is OK
- 10.3847/2041-8213/aa9367 is OK
- 10.1051/0004-6361/201525830 is OK
- 10.1088/0264-9381/27/17/173001 is OK
- 10.3847/2041-8213/aa905c is OK
- 10.1126/science.aap9811 is OK
- 10.3847/1538-4357/abd3a0 is OK
- 10.1038/s41550-017-0285-z is OK
- 10.3847/2041-8213/aa8fc7 is OK
- 10.1126/science.aaq0049 is OK
- 10.1126/science.aap9580 is OK
- 10.3847/2041-8213/aa9478 is OK
- 10.3847/2041-8213/aa91b3 is OK
- 10.3847/2041-8213/aa9029 is OK
- 10.1051/0004-6361/201833025 is OK
- 10.1103/PhysRevD.93.084029 is OK
- 10.1103/PhysRevD.76.061504 is OK
- 10.1038/nature24303 is OK
- 10.1038/nature24291 is OK
- 10.3847/2041-8213/aa8edf is OK
- 10.3847/2041-8213/aa9d82 is OK
- 10.1093/pasj/psx121 is OK
- 10.3847/2041-8213/aa90b6 is OK
- 10.3847/2041-8213/aaa07d is OK
- 10.1038/s41467-017-01152-9 is OK
- 10.1103/PhysRevD.100.063021 is OK
- 10.1103/PhysRevD.100.043011 is OK
- 10.3847/0004-637X/825/1/52 is OK
- 10.1086/587832 is OK
- 10.1038/s41550-019-0880-2 is OK
- 10.3847/2041-8213/aa991c is OK
- 10.1086/321359 is OK
- 10.1093/mnras/sty1683 is OK
- 10.1093/mnras/stw3225 is OK
- 10.1093/mnras/stv2195 is OK
- 10.1093/mnras/stv990 is OK
- 10.1088/2041-8205/778/2/L23 is OK
- 10.1093/mnras/stz3223 is OK
- 10.1086/192237 is OK
- 10.1088/0004-637X/801/2/90 is OK
- 10.3847/2041-8213/ab3800 is OK
- 10.1088/0004-637X/785/1/28 is OK
- 10.1146/annurev-astro-081915-023322 is OK
- 10.1093/mnras/stv721 is OK
- 10.1103/PhysRevLett.116.061102 is OK
- 10.1103/PhysRevD.88.043007 is OK
- 10.1088/0264-9381/33/7/075009 is OK
- 10.1088/1742-6596/228/1/012012 is OK
- 10.1093/mnras/260.3.675 is OK
- 10.1007/BF00873079 is OK
- 10.1088/0264-9381/29/12/124007 is OK
- 10.1103/PhysRevLett.119.161101 is OK
- 10.1088/0004-637X/746/1/48 is OK
- 10.1103/PhysRevD.85.044061 is OK
- 10.1038/323310a0 is OK
- 10.1093/mnras/stz1147 is OK
- 10.1103/PhysRevLett.121.021303 is OK
- 10.1088/0004-637X/776/1/47 is OK
- 10.1093/mnras/stz1131 is OK
- 10.1103/PhysRevD.97.023009 is OK
- 10.1051/0004-6361/201630188 is OK
- 10.1103/PhysRevD.98.123017 is OK
- 10.1103/PhysRevD.95.044024 is OK
- 10.1086/497062 is OK
- 10.1086/186493 is OK
- 10.1111/j.1365-2966.2008.13402.x is OK
- 10.5281/zenodo.592845 is OK
-  10.25080/Majora-92bf1922-00a  is OK
- 10.1038/s41586-020-2649-2 is OK
- 10.1038/s41592-019-0686-2 is OK
- 10.1109/MCSE.2007.53 is OK
- 10.1109/MCSE.2011.37 is OK
- 10.1051/0004-6361/201322068 is OK
- 10.3847/1538-3881/aabc4f is OK
- 10.1111/j.1365-2966.2009.14711.x is OK
- 10.1086/319455 is OK
- 10.1111/j.1365-2966.2005.09669.x is OK
- 10.1111/j.1365-2966.2012.21083.x is OK
- 10.3847/1538-4357/ab1b21 is OK
- 10.1093/mnras/stw1083 is OK
- 10.1093/mnras/stz216 is OK
- 10.1086/155109 is OK
- 10.1093/mnras/staa002 is OK
- 10.1093/mnras/staa3390 is OK
- 10.1093/mnras/staa3043 is OK
- 10.1093/mnras/stz2335 is OK
- 10.1093/mnras/staa2264 is OK
- 10.1093/mnras/staa1784 is OK
- 10.1093/mnras/stab850 is OK
- 10.1086/304238 is OK
- 10.1051/0004-6361/201730472 is OK
- 10.1016/j.physrep.2007.02.002 is OK
- 10.1146/annurev-astro-081811-125534 is OK
- 10.1146/annurev.astro.44.051905.092532 is OK
- 10.1146/annurev.astro.43.072103.150558 is OK
- 10.1146/annurev-astro-081913-035926 is OK
- 10.1093/mnras/stw1083 is OK
- 10.1093/mnras/stab1291 is OK
- 10.1016/j.newar.2010.09.023 is OK
- 10.1086/175268 is OK
- 10.1051/0004-6361/202038707 is OK
- 10.1017/pasa.2020.31 is OK
- 10.1103/PhysRevLett.121.161101 is OK
- 10.3847/2041-8213/ab50c5 is OK
- 10.3847/2041-8213/ab481c is OK
- 10.3847/2041-8213/ab822f is OK
- 10.1038/248030a0 is OK
- 10.1007/BF02345020 is OK
- 10.1093/mnras/112.6.583 is OK
- 10.1093/mnras/sty2186 is OK
- 10.3847/2041-8213/aa991c is OK
- 10.1103/PhysRevD.97.021501 is OK
- 10.1103/PhysRevD.98.123017 is OK
- 10.3847/2041-8213/aaa401 is OK
- 10.1093/mnras/sty3047 is OK
- 10.3847/1538-4357/aac6e5 is OK
- 10.1103/PhysRevD.100.023015 is OK
- 10.1088/1361-6382/ab5f7c is OK
- 10.1103/PhysRevLett.125.101102 is OK
- 10.3847/2041-8213/aba493 is OK
- 10.1007/s41115-020-0008-5 is OK
- 10.1038/s41586-020-03059-w is OK
- 10.1007/s00159-017-0106-5 is OK
- 10.1111/j.1365-2966.2012.20767.x is OK
- 10.1086/186905 is OK
- 10.3847/1538-4357/abde4a is OK
- 10.1146/annurev-astro-082708-101833 is OK
- 10.1051/0004-6361/202039219 is OK
- 10.3847/2041-8213/abd5b7 is OK
- 10.1103/PhysRevD.98.083017 is OK
- 10.1126/science.abb3363 is OK
- 10.3847/2041-8213/ab30ca is OK
- 10.1051/0004-6361/201936669 is OK
- 10.1051/0004-6361/201834525 is OK
- 10.1093/mnras/sty1655 is OK
- 10.1017/pasa.2017.51 is OK
- 10.1093/mnras/stw941 is OK
- 10.1088/0004-637X/755/2/123 is OK
- 10.1051/0004-6361/202037744 is OK
- 10.1051/0004-6361/201935854 is OK
- 10.1017/pasa.2019.31 is OK
- 10.1017/pasa.2018.47 is OK
- 10.1093/mnras/stt1612 is OK
- 10.3847/1538-4357/ab0020 is OK
- 10.3847/1538-4357/aa6afe is OK
- 10.1046/j.1365-8711.2003.06420.x is OK
- 10.1146/annurev.astro.38.1.143 is OK
- 10.1051/0004-6361/201424356 is OK
- 10.1051/0004-6361/201016113 is OK
- 10.1093/mnras/stz3064 is OK
- 10.1086/185922 is OK
- 10.1038/s41550-021-01360-w is OK
- 10.1086/172789 is OK
- 10.1051/0004-6361/202038707 is OK
- 10.3847/0004-637X/825/1/71 is OK
- 10.1088/0264-9381/28/9/094013 is OK
- 10.1103/PhysRevLett.116.241103 is OK
- 10.1051/0004-6361/201322068 is OK
- 10.1038/s41586-020-2649-2 is OK
- 10.1093/mnras/stz3542 is OK

MISSING DOIs

- 10.1093/mnras/stab2716 may be a valid DOI for title: Impact of Massive Binary Star and Cosmic Evolution on Gravitational Wave Observations I: Black Hole - Neutron Star Mergers
- 10.1103/physrevd.100.043012 may be a valid DOI for title: Reconstructing phenomenological distributions of compact binaries via gravitational wave observations
- 10.1093/mnras/stz226 may be a valid DOI for title: Constraints on Binary Black Hole Populations from LIGO-Virgo Detections
- 10.1093/mnras/stz1175 may be a valid DOI for title: Black Hole Mergers from Quadruples
- 10.3847/2515-5172/ab66be may be a valid DOI for title: What GW170729’s exceptional mass and spin tells us about its family tree
- 10.3847/1538-4357/ab2f92 may be a valid DOI for title: SN 2016iet: The Pulsational or Pair Instability Explosion of a Low Metallicity Massive CO Core Embedded in a Dense Hydrogen-Poor Circumstellar Medium
- 10.1093/pasj/psx108 may be a valid DOI for title: The Detection Rates of Merging Binary Black Holes Originating from Star Clusters and Their Mass Function
- 10.1093/mnras/staa720 may be a valid DOI for title: R-process enrichment in ultrafaint dwarf galaxies
- 10.1103/physrevd.101.063021 may be a valid DOI for title: Gravitational waves or deconfined quarks: what causes the premature collapse of neutron stars born in short gamma-ray bursts?
- 10.1093/mnras/staa157 may be a valid DOI for title: The high redshift SFR-M* relation is sensitive to the employed star formation rate and stellar mass indicators: Towards addressing the tension between observations and simulations
- 10.1017/cbo9780511536281.017 may be a valid DOI for title: Formation and evolution of compact stellar X-ray sources
- 10.1103/physrevd.104.063030 may be a valid DOI for title: Detecting Gravitational Waves With Disparate Detector Responses: Two New Binary Black Hole Mergers
- 10.3847/1538-4357/ab733f may be a valid DOI for title: 2-OGC: Open Gravitational-wave Catalog of binary mergers from analysis of public Advanced LIGO and Virgo data
- 10.1103/physrevd.102.083026 may be a valid DOI for title: Gravitational-wave inference in the catalog era: evolving priors and marginal events
- 10.1088/0004-637x/764/2/166 may be a valid DOI for title: The rotation rates of massive stars: the role of binary interaction through tides, mass transfer, and mergers
- 10.1007/s11222-008-9059-x may be a valid DOI for title: Adaptive Importance Sampling in General Mixture Classes
- 10.22323/1.306.0065 may be a valid DOI for title: Single and binary stellar progenitors of long-duration gamma-ray bursts
- 10.7146/dpb.v6i80.6496 may be a valid DOI for title: Blindfold games are harder than games with perfect information
- 10.1080/01621459.2000.10473909 may be a valid DOI for title: Safe and effective importance sampling
- 10.2307/1268522 may be a valid DOI for title: A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code
- 10.1080/00224065.1981.11978748 may be a valid DOI for title: An approach to sensitivity analysis of computer models: Part I—Introduction, input variable selection and preliminary variable assessment
- 10.1051/0004-6361/201424286 may be a valid DOI for title: Bonnsai: a Bayesian tool for comparing stars with stellar evolution models
- 10.1051/0004-6361/201629685 may be a valid DOI for title: Delay-time distribution of core-collapse supernovae with late events resulting from binary interaction
- 10.1051/0004-6361/201935842 may be a valid DOI for title: Constraining the masses of microlensing black holes and the mass gap with Gaia DR2
- 10.1051/0004-6361/200912827 may be a valid DOI for title: Population synthesis of binary carbon-enhanced metal-poor stars
- 10.1103/physrevd.100.023007 may be a valid DOI for title: A Highly Spinning and Aligned Binary Black Hole Merger in the Advanced LIGO First Observing Run
- 10.1038/nature18322 may be a valid DOI for title: The first gravitational-wave source from the isolated evolution of two stars in the 40–100 solar mass range
- 10.1088/0004-637x/810/1/61 may be a valid DOI for title: Early-type Eclipsing Binaries with Intermediate Orbital Periods
- 10.1051/0004-6361/201321576 may be a valid DOI for title: PopCORN: Hunting down the differences between binary population synthesis codes
- 10.1051/0004-6361/201321753 may be a valid DOI for title: The effect of common-envelope evolution on the visible population of post-common-envelope binaries
- 10.1103/physrevd.100.043012 may be a valid DOI for title: Reconstructing phenomenological distributions of compact binaries via gravitational wave observations
- 10.3847/2041-8213/ab5b9a may be a valid DOI for title: LISA and the Existence of a Fast-Merging Double Neutron Star Formation Channel
- 10.1086/127051 may be a valid DOI for title: Nuclear Reactions in Stars and Nucleogenesis
- 10.1086/344585 may be a valid DOI for title: Probing the Neutron-Capture Nucleosynthesis History of Galactic Matter
- 10.1111/j.1365-2966.2005.09087.x may be a valid DOI for title: A statistical study of 233 pulsar proper motions
- 10.1093/mnras/stu1738 may be a valid DOI for title: Galaxies on FIRE (Feedback In Realistic Environments): stellar feedback explains cosmologically inefficient star formation
- 10.1016/j.physrep.2007.02.009 may be a valid DOI for title: Nucleosynthesis and remnants in massive stars of solar metallicity
- 10.1086/522073 may be a valid DOI for title: A new look at the binary characteristics of massive stars
- 10.1017/pasa.2016.52 may be a valid DOI for title: Dawes review 6: The impact of companions on stellar evolution
- 10.1086/181708 may be a valid DOI for title: Discovery of a pulsar in a binary system
- 10.12942/lrr-2014-3 may be a valid DOI for title: The evolution of compact binary star systems
- 10.1103/physrevlett.76.352 may be a valid DOI for title: Pulsar recoil and gravitational radiation due to asymmetrical stellar collapse and explosion
- 10.1080/10556799308230566 may be a valid DOI for title: Asymmetric neutrino emission and formation of rapidly moving pulsars
- 10.1086/431786 may be a valid DOI for title: The neutrino bubble instability: A mechanism for generating pulsar kicks
- 10.1038/253698a0 may be a valid DOI for title: Two kinds of stellar collapse
- 10.1086/338805 may be a valid DOI for title: The velocity distribution of isolated radio pulsars
- 10.1007/978-94-010-1483-0_5 may be a valid DOI for title: Late Stages of Close Binary Systems
- 10.1038/nature10529 may be a valid DOI for title: Two populations of X-ray pulsars produced by two types of supernova
- 10.1051/0004-6361/201220636 may be a valid DOI for title: Three-dimensional neutrino-driven supernovae: Neutron star kicks, spins, and asymmetric ejection of nucleosynthesis products
- 10.1007/s12036-017-9474-5 may be a valid DOI for title: A new look at distances and velocities of neutron stars
- 10.1088/0004-6256/137/2/3358 may be a valid DOI for title: The high angular resolution multiplicity of massive stars
- 10.1088/0004-637x/751/1/4 may be a valid DOI for title: An updated look at binary characteristics of massive stars in the Cygnus OB2 association
- 10.1111/j.1365-2966.2012.21317.x may be a valid DOI for title: A spectroscopic survey on the multiplicity of high-mass stars
- 10.1088/0004-637x/719/1/722 may be a valid DOI for title: Further evidence for the bimodal distribution of neutron-star masses
- 10.1086/160960 may be a valid DOI for title: Approximations to the radii of Roche lobes
- 10.1088/2041-8205/715/2/l138 may be a valid DOI for title: The effect of metallicity on the detection prospects for gravitational waves
- 10.1088/0004-637x/717/2/724 may be a valid DOI for title: Adiabatic mass loss in binary stars. I. Computational method
- 10.1088/0004-637x/812/1/40 may be a valid DOI for title: Adiabatic mass loss in binary stars. II. From zero-age main sequence to the base of the giant branch
- 10.1086/165412 may be a valid DOI for title: Thresholds for rapid mass transfer in binary systems. I-Polytropic models
- 10.1051/0004-6361/201322714 may be a valid DOI for title: Theoretical uncertainties of the Type Ia supernova rate
- 10.3847/1538-4357/aa8140 may be a valid DOI for title: Accretion Disk Assembly During Common Envelope Evolution: Implications for Feedback and LIGO Binary Black Hole Formation
- 10.3847/1538-4357/aa6117 may be a valid DOI for title: Common Envelope Wind Tunnel: Coefficients of Drag and Accretion in a Simplified Context for Studying Flows around Objects Embedded within Stellar Envelopes
- 10.1016/j.physrep.2007.02.008 may be a valid DOI for title: Formation of double compact objects
- 10.1086/133321 may be a valid DOI for title: Common envelopes in binary star evolution
- 10.1086/161701 may be a valid DOI for title: Double white dwarfs as progenitors of R Coronae Borealis stars and Type I supernovae
- 10.1086/168974 may be a valid DOI for title: Common envelope evolution and double cores of planetary nebulae
- 10.1088/0004-637x/716/1/114 may be a valid DOI for title: On the binding energy parameter λof common envelope evolution
- 10.3847/1538-4357/836/2/244 may be a valid DOI for title: Pulsational pair-instability supernovae
- 10.1086/190103 may be a valid DOI for title: Neutrino Processes and Pair Formation in Massive Stars and Supernovae.
- 10.1103/physrevlett.18.379 may be a valid DOI for title: Dynamics of supernova explosion resulting from pair formation
- 10.1051/0004-6361/201628980 may be a valid DOI for title: The effect of pair-instability mass loss on black-hole mergers
- 10.1088/1361-6382/aa552e may be a valid DOI for title: Use of gravitational waves to probe the formation channels of compact binaries
- 10.3847/1538-4357/aab34c may be a valid DOI for title: Measuring the binary black hole mass spectrum with an astrophysically motivated parameterization
- 10.1086/154524 may be a valid DOI for title: The binary pulsar-Physical processes, possible companions, and evolutionary histories
- 10.1016/j.physletb.2020.135402 may be a valid DOI for title: Binary Neutron Star Mergers with Missing Electromagnetic Counterparts as Manifestations of Mirror World
- 10.1007/978-94-010-3087-8_42 may be a valid DOI for title: Pulsars and close binary systems
- 10.1086/306265 may be a valid DOI for title: Evolution of binary compact objects that merge
- 10.1103/physrevd.86.124007 may be a valid DOI for title: Black-hole–neutron-star mergers: Disk mass predictions
- 10.1103/physrevd.84.104017 may be a valid DOI for title: Will black hole-neutron star binary inspirals tell us about the neutron star equation of state?
- 10.1088/2041-8205/791/1/l7 may be a valid DOI for title: Prospects for Joint Gravitational-wave and Electromagnetic Observations of Neutron-star-Black-hole Coalescing Binaries
- 10.3847/0004-637x/820/1/28 may be a valid DOI for title: The dense matter equation of state from neutron star radius and mass measurements
- 10.1103/physrevd.85.104045 may be a valid DOI for title: Estimating parameters of coalescing compact binaries with proposed advanced detector networks
- 10.1103/physrevd.49.2658 may be a valid DOI for title: Gravitational waves from merging compact binaries: How accurately can one extract the binary’s parameters from the inspiral waveform?
- 10.22323/1.331.0013 may be a valid DOI for title: Multi-messenger observations of a binary neutron star merger
- 10.3847/1538-4357/ab8d24 may be a valid DOI for title: A Search for Neutron Star-Black Hole Binary Mergers in the Short Gamma-Ray Burst Population
- 10.1287/opre.1.5.263 may be a valid DOI for title: Methods of Reducing Sample Size in Monte Carlo Computations
- 10.1007/10720995_68 may be a valid DOI for title: Constraints on mass ejection in black hole formation derived from black hole X-ray binaries
- 10.1007/3-540-31186-6_27 may be a valid DOI for title: An adaptive importance sampling technique
- 10.1016/0021-9991(77)90121-8 may be a valid DOI for title: Nonphysical sampling distributions in Monte Carlo free-energy estimation: Umbrella sampling
- 10.1088/0004-6256/142/6/197 may be a valid DOI for title: Toward a Unification of Star Formation Rate Determinations in the Milky Way and Other Galaxies
- 10.1093/mnras/staa1324 may be a valid DOI for title: Weighing in on black hole binaries with BPASS: LB-1 does not contain a 70M_⊙ black hole
- 10.1017/s1743921318008037 may be a valid DOI for title: High mass X-ray binaries as progenitors of gravitational wave sources
- 10.3847/1538-4357/ab6458 may be a valid DOI for title: The Explosion of Helium Stars Evolved With Mass Loss
- 10.1093/mnras/staa2681 may be a valid DOI for title: Black hole, neutron star and white dwarf merger rates in AGN disks
- 10.1093/mnras/stab2716 may be a valid DOI for title: Impact of Massive Binary Star and Cosmic Evolution on Gravitational Wave Observations I: Black Hole - Neutron Star Mergers
- 10.1093/mnrasl/slaa196 may be a valid DOI for title: Is GW190521 the merger of black holes from the first stellar generations?

INVALID DOIs

- 10.1007/s41114-018-0012-9, 10.1007/lrr-2016-1 is INVALID
- 10.1007/s41114-018-0012-9, 10.1007/lrr-2016-1 is INVALID
- https://doi.org/10.3847/1538-4357/ab584d is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1016/j.physrep.2007.06.002 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1016/j.newar.2004.09.020 is INVALID because of 'https://doi.org/' prefix

Hello all!

I made a start on the review - plenty left to do code-side. For now here are a few comments on the paper:

State of the field: Do the authors describe how this software compares to other commonly-used packages?

Some text should be added to the paper to address this crateria - now since the word limit won't let you do such a discussion justice AND there is the apj paper which does have a rather extensive list in the intro, is it worth citing the sister paper? I don't know how the ApJ/JOSS link works and whether I should consider the two paper as one and tick that box or ask you to add a sentence saying you expand on that in the ApJ publication. @christinahedges can you advise?

Now for 2 very minor comments:

stating that massive stars are born with similarly massive companions based on Sana et al. is very 2012 of you ;) Moe and Di Stefano 2017 has updated observations and I'm pretty sure it doesn't show that massive stars have an excess of systems with mass ratios ~1 (they have nifty distributions and table - see figure 2 and table 13). It's saturday as i'm doing this so I'll check that and amend as necessary; also the updated reference would be good :)
"embarassingly" -> although i appreciate the emphasis the emotive language maybe doesn't belong here ;) pick another word or drop it?

Hi @HeloiseS ,

First of all, thank you very much for pointing out the documentation (and python submit script) improvements as issues on the COMPAS github page. Much appreciated -- we'll aim to address them ASAP.

On your two "minor points":

Happy to add a citation to Moe & DiStefano (2017), of course, who actually include a specific twin subpopulation; however, we just meant the more modest statement that massive stars are more likely to have massive companions than if the companions were independently drawn from the IMF, rather than focus on details of the mass distribution.
See https://en.wikipedia.org/wiki/Embarrassingly_parallel .

@ilyamandel

i see what you mean now - thanks for the clarification
well bless my heart - every day is a school day :rofl: thanks for the link, looking forward to reading about it!

Further discussion on the `pythonSubmit.py` files

Okay so I think I have put my finger on why these files feels so odd:

1 - first, as i mentioned before there are some weird constructions: for example methods are called to create list when (at the moment) it seems you could just make the lists attributes without the need to call a function and return something. Another thing that caught my eye is the function combineCommandLineOptionsDictIntoShellCommand which exists as a separate entity and not a method of the class, when it seems like its only job is to operate on the things within the class.

2- The definition of the class itself isn't very pythonic (to me at least). I would expect that class to have the architecture to contain the necessary data and the tools to put it together in a command compas can read, but not the specific numbers and options for a specific simulation. It seems (from skimming through the folders) that you are redefining the class over and over again in each pythonSubmit.py file. What I would expect instead is for these options to be parsed to each instances of the class, for example with a yaml file. So basically my thinking is that COMPAS should come with a unique pythonProgramOptions class that can be imported, and it's the config files that the user would tweak and then give to the pipeline to run compas. For example:

from compas import pythonProgramOptions

myoptions = pythonProgramOptions(myconfig.yaml)
myoptions.call_shell_command() # if you want to call directly
myoptions.shell_command # if you want to retrieve the string and do somthing with it

now the question is... how much of a pain would that be regarding backwards compatibility? If you're curious to see what i did with this class (i refactored it for my own use) i can fork a suggestion for the class and yaml file (unless Jeff prefers JSON :wink: ) for you to look at and see if it would work with the general workflow of compas.

EDIT: Also fyi the demo ran perfectly and the plots i get from it look exactly as the ones you include in th directory so the example works :+1:

Hi @HeloiseS , Thank you very much for all of your suggestions so far.

A quick update:

We've updated the documentation to clarify some of the workflow issues you raised.
We included assert statements in pythonSubmit.py to make sure that the environment variables are set and the executable file exists.
We updated the awkward sentence about companion masses to "Most massive stars---those with initial masses greater than 8 $M_\odot$---are born with another massive star as a companion [@Sana:2012Sci;@Moe:2017ApJS]. "

We are grateful for your suggestion regarding an overhaul of pythonSubmit.py, which we agree is perhaps not optimal. It is just meant as a simple and convenient way to update the COMPAS command line arguments -- so, indeed, the idea is that users would edit pythonSubmit itself, effectively treating it as an inlist text file. Note that it is not necessary at all to use pythonSubmit to run COMPAS -- one could, say, store the desired command line in a text file. So our feeling is that it's not the most urgent development priority at the moment.

Definitely not urgent - but I'd like to get the feeling of @katiebreivik and how she found handling the COMPAS calls with the python class. If you say it's intended for users to make their own python utility then it's worth noting that it was time consuming and fiddly to put the example into my own code since the config is embedded into a class and it took a while to get the reformatting to work properly without errors since there are so many parameters.

Let's put a pin in that for now, but i'd like to discuss that further later on :)

@HeloiseS and @ilyamandel, in response to your question, I believe that 1) the paper does need to discuss the state of the art (including other similar packages, and how this tool compares) 2) there should be enough room in the paper currently to add this discussion, it does not need to have extreme detail, but it should be present in the JOSS submission (not just the AAS submission).

I hope this helps clarify the question!

:wave: @katiebreivik, please update us on how your review is going (this is an automated reminder).

:wave: @HeloiseS, please update us on how your review is going (this is an automated reminder).

Update! I'm going through each page of the user manual and tutorial and it's going very smoothly for the most part. I'll make a few general recommendations about the user manual and tutorial at the end when I've digested the whole thing and I've successfully ran all the notebooks. That being said, I have a couple of requests regarding the command list descriptions that i know i'm going to make so I might as well tell you now :smile:

Program option list and default values

Topic-based command listing: The alphabetical list with all the details on the commands is really good and exhaustive. Problem is if you don't know what options do exist in compas it's a tough way to figure it out. Could we also have a summary page (or table or else) of just the command names grouped by topic? That can help people find the commands they might need and then they can look up the exact descriptions in the alphabetical list. To be fair, the effective naming conventions of the commands do a lot of that job (e.g. the kicks are all together), but it doesn't do everything, for example LBV options are at the end of a very long list of logs and might be overlooked, or --detailed-output is after a bunch of debug commands.

Some of the command options are basically references, but many people won't know what they actually mean. It would be good to at least provide a link to the references (and so commands do already have references), and if you think it's relevant an equation or section number. To make your life easier I have made a note as I read of all the commands that need a ref(s), they are listed below.

            --common-envelope-lambda-prescription 
            --common-envelope-mass-accretion-prescription 
            --common-envelope-slope-kruckow 
            --cool-wind-mass-loss-multiplier
            --eccentricity-distribution 
            --envelope-state-prescription
            --initial-mass-function
            --kick-magnitude-distribution
            --kick-phi-1 -> typo "for the" repeated
            --luminous-blue-variable-multiplier
            --luminous-blue-variable-prescription 
            --mass-ratio-distribution
            --mass-transfer-rejuvenation-prescription
            --metallicity-distribution
            --pulsational-pair-instability-prescription
            --remnant-mass-prescription
            --rotational-velocity-distribution
            --semi-major-axis-distribution
            --stellar-zeta-prescription

Other questions:
- --case-bb-stability-prescription has an option TREAT_AS_OTHER_MT , what does that mean?
- --common-envelope-allow-main-sequence-survive: does it mean they always do or they're just not constrained to merge?
- --common-envelope-lambda-multiplier What does that do? Why not just have a larger lambda param value?
- --debug-classes description has vactor instead of vector. Also what are the options?
- --envelope-state-prescription What is legacy? (if that's taken care of by adding references ignore me)
- --help state that -h and --help don't do the same thing on this page too
- --kick-direction what's the wedge? is poles just the poles or an opening angle around the pole (from the ApJ I'm guessing that's the cone you mention on p18)? perpendicular to the orbit I guess? what's in-plane?
- --kick-magnitude-random Is the random distribution taken from the --kick-magnitude-distribution speficied or defaulted? Manual says the float is used to draw the kick magnitude but not sure what that means in that context since distributions need more numbers than that right?
- --kick-magnitude-sigma-CCSN-BH is that for a maxwellian kick distribution (and so assuming that --kick-magnitude-distribution goes to default)? What if the kick distribution has been set to something that doesn't take a sigma value?
- --kick-mean-anomaly-1 What's this ?:D
- --mass-ratio-distribution No Moe and Di stefano :(?
- --neutrino-mass-loss-BH-formation-value "Amount of mass lost in neutrinos during BH formation (either as fraction or in solar masses, depending on the value of --neutrino-mass-loss-bh-formation)" Is the default in msol or as a fraction? does it turn into solar masses above 1 or when's that threshold?

--debug-classes description has vactor instead of vector. Also what are the options?

I'll fix the "vactor". These are user (actually, more correctly, developer) defined. When a developer wants to do some debugging, they can add debug statements that have an associated "class" string - so they can dot many debug statements around the code with different class strings, then when they run the code they can specify for which of those class strings they want the debug statements printed - they can print those that match just one class string, or an array of class strings. It's just a way of making life a bit easier for a developer looking for a problem - they can turn on/off different debug statements in the code without having to change the code.

Log classes are similar, but they aren't really used any more - though they could be. Similar concept to debug classes, but instead of debug statement with debug class strings, we have log statements with log class strings. At the moment, all log statements have as an associated class string the name of the file they are logging to - so users could turn logging to particular files on and off using the '--log-classes' option, but there are also other options to control which log files we produce, so that's a bit redundant. Log classes were implemented in the first iteration of the logging class, but over time the need for them lessened.

--help state that -h and --help don't do the same thing on this page too

Sure, good idea.

--mass-ratio-distribution No Moe and Di stefano :(?

I thought I saw or heard discussion about that being in the plans. Oh, actually - that requires a longer explanation about moving sampling outside of the COMPAS C++ code, so no, probably not in the C++ code.

I'll leave explanations of the other options to others who can probably provide more detail than me. Reinhold is a good person to ask about the kick-related options (but Ilya and others will be able to answer those questions too).

Thanks!!

I'll leave explanations of the other options to others who can probably provide more detail than me. Reinhold is a good person to ask about the kick-related options (but Ilya and others will be able to answer those questions too).

@ilyamandel do you want me to add this previous post about command clarification to the issue #686 so that Reinhold et al. can join in?

Sorry for the silence, @HeloiseS . I'll handle the options clarification over the next 24 hours -- it should be sufficiently straightforward.

@whedon generate pdf from branch JOSS

Attempting PDF compilation from custom branch JOSS. Reticulating splines etc...

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

@whedon generate pdf from branch JOSS

Attempting PDF compilation from custom branch JOSS. Reticulating splines etc...

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

@HeloiseS , @christinahedges , @katiebreivik -- I updated the text to include a brief context of other available rapid binary population synthesis tools, as suggested. [Please ignore the one broken reference in the last compiled version -- it's fixed now, just didn't want to generate another version.]

@HeloiseS -- I updated the user options documentation as requested (just waiting for pull request https://github.com/TeamCOMPAS/COMPAS/pull/688 to be checked/approved). Thank you for the excellent suggestions!

@HeloiseS -- that's been pulled in. I've now added a list of program options categorised by what they do as you suggested, to be included once https://github.com/TeamCOMPAS/COMPAS/pull/689 is approved. I think this addresses all of your must-implement comments so far -- hope I haven't missed anything!

Awesome! I can see all the changes :) Will try to finish the cosmic integration tutorials and reading the developer section by the end of tomorrow.

@ilyamandel hello again. What data am I expected to use for the Cosmic Integration tutorials?

Reason I'm asking is that i don't have a COMPAS output with 'BSE_Double_Compact_Objects' it seems. I used the output from the chirpmass distribution example (expecting that I'd defo have DCOs in there) and that didn't work.

I went looking through the manual hoping to figure out how to make this appear in my output (I think I'd need to amend the COMPAS_Definitions file?) but I can't find it in: https://compas.readthedocs.io/en/latest/pages/User%20guide/COMPAS%20output/standard-logfiles-record-specification-binary.html

I don't want to open an issue yet since it's entierly possible i missed something obvious

@HeloiseS You're right, there's not a lot. Did you see this:

https://compas.readthedocs.io/en/latest/pages/Developer%20guide/Services/Logging%20and%20debugging/Base-level%20logging/services-base-level-logging.html?highlight=class#log-debug-record-filtering

The guide is a work in progress - particularly the developer guide...

Initial Review

Alright I finished reading the manual and running (or trying to) your tutorials and utility codes. It's no surprise to me that COMPAS runs like clockwork and I'm very excited about all the data it records on the systems - so much to play with!

The manual is very exhaustive, the C++ code is a thing of absolute beauty - I can't code in C++ but the lay out, the comments and everything meant that I could navigate it easily as I was going through the Dev guide, locating the classes and functions mentioned. The jupyter notebooks have loads of info and some stupendous graphs to illustrate some of the tougher concepts, I love it.

Overall it's really amazing.

That's for the "good cop" part of this review, let's move on to the "bad cop" part.

Bring user facing code to the same standard as the rest of the code

As much as the core of compas is refined and polished, a lot of the python utilities feel like an afterthought or something that's not been revised with the same level of care. My main request for this section is the following:

[ ] Python utilities need to be brought to the same standard as the core of the simulation (more details below)

Now @ilyamandel I know that you're going to say "that it's not the most urgent development priority at the moment" but my question would then be: if not priority in a JOSS review when your referee is asking you, then when will it be? If you don't do it now you're going to be stuck with code that's on the side-lines of good practice, so I'm sorry but this is non-negotiable, you gotta to fix the python. Now for more details:

Your Python should be Pythonic

I am not being a PEP8 pedant, this is a case where The Principle of Least Astonishment really comes to mind - I have been genuinely confused by some of the constructions, and I don't believe I'm all that special. If I was confused someone else will be too ;)

[ ] Refactor scripts and modules to ensure OO doesn't add complexity without adding functionality For example take the pythonSubmit.py (I told you it'd be back), pythonProgramOptions has no need to be class: the code runs top to bottom, there is no inheritance or actual need for encapsulation since we're not importing it in any other code... it adds complexity without adding functionality - plus when I see a class I unconsciously expect to be making objects with it in other places, in short the construction is not coherent with the use case in the manual. The Dev guide has some fantastic sections about the principles of Object Oriented Programming. Just make sure those are followed throughout COMPAS.
[ ] Ensure names of modules, classes don't break conventions and are sufficiently descriptive. Some pretty ubiquitous naming conventions were ignored: for example ClassCOMPAS is a module but it looks like a class name in python, and then the class within it is COMPASData. An easy fix would be to call your class COMPASData and your module compas_data.py or compasdata.py. But even then, you want to rethink that name because that's confusing in itself: it's NOT the COMPASData, its a processing pipeline. The name needs to reflect that. (by the way I've noticed a lot of C like camel case naming conventions, even though that's not very python it's consistent with the rest of the code and so it'd be fine to adopt across the board).
[ ] Add all necessary Docstrings Docstrings need to be ubiquitous and also they need to show some consistency. Some functions have beautifully detailed docstrings (e.g. in FastComsicIntegration.py or selection_effects.py, the latter being a perfect example of what I mean by pythonic code), some have pretty rudimentary ones, some have just plain comments, some have none. For example in the tutorial on COMPASData you have a cell with a markdown description of the parameters but the class itself doesn't have the info in docstrings.
- [ ] Also if you have optional parameters give me the default in the description.
[ ] Be consistent with your docstring conventions You should pick a docstring style and stick to it. selections_effects.py uses numpy (my fave :relaxed:) which is excellent, but there are plenty. The Dev Guide has a whole section about style conventions but it's about the C++. Again, same level of care should go into your python, it could be a good time for you all to establish solid standards for your post processing / pre processing utilities.

Other things

[ ] Group your utilities under one directory at the same level as the src directory Your utilities need to be grouped together under an easy to find (and easy to test, see below) directory. I would strongly recommend making it a package that can be installed, but even if you don't want to do that, at least have all the python utilities in one place.
[ ] tests not every line of python needs testing but have some tests for the main utilities that i can run on my machine on install to make sure it works. It is a requirement for the JOSS publication to have tests and all the workflows I see on the github at the moment only check that COMPAS compiles (it's possible I'm missing something, please let me know). Now, running a stellar evolution simulation in continuous integration sounds pretty unhinged so I'm not asking you to do that, but you can give us some thing to run locally - good thing is you already have a lot of that stuff in the tutorials! (For more about the tutos see below.) I'm happy to let you use the tutorials as test, by just telling us that explicitly in the install or the Get started section, that if we want to make sure all the utilities run we can just run the notebooks. That is unless @christinahedges says there must be unitests, in which case do that.
[ ] no bare exceptions Errors aren't like pokemon, you don't want to catch them all ;) after the except keywords make sure you explicitly state which error you're catching, otherwise you might end up with silent bugs. It's genuinely bitten me in the arse before so I strongly recommend you abide by best practices on this one.

User Manual

[ ] Move the examples and the Tutorial: Simple COMPAS Run Streamline the example directory to only contain the necessary things for the tutorial and call it accordingly (something like tutorial_somethingsomething). Also move that tutorial earlier in the manual (maybe even at the end of the getting started) and you could also make it so it gives data we can use with the other tutorials (see below).
[ ] Tutorials could be much more locatable There are plenty of tutorials and they are a little scattered and also sometimes hidden. Now I totally get why they are where they are and maybe moving them would take to much restructuring of the manual, but it would be helpful to find a way to bring them up in the levels or flagging them clearly somewhere that people will see much sooner when they're reading.
[ ] Tutorials need to come with their data I have not been able to run the majority of your tutorials (I might have additional comments late on that once it's sorted), because i don't know what data to use and the one I made with the examples is doesn't have the right stuff in it (see earlier messages). It would be nice if it was though :) it would make everything fit together quite nicely but you might say it's not possible or realistic (e.g. if the simulation takes more computational resources than is reasonable). In that case give me a direct link to where I can download the data.
[ ] Can you tell me early in the tutorial that the logs are where the data go :) It took me way too long to realise that when you were talking about logfiles and logs you were talking about data files, and once again I'm not special, I'm sure a fair few astronomers will not find that intuitive. No need to change the whole naming convention but really sign post very explicitly and very early on that that the data files are the log files. In my mind logs are for like administrative info like errors etc, so quite distinct from data files. See for illustration this epiphany I found in my notes "OMG THE LOG FILE RECORD SPECIFICATION PROPERTIES ARE THE DAMN COLUMN IN THE DATA SET". I wrote that nearly a week after starting the review, it really took me that long.

Addendum: I'm reading the ApJ paper top to bottom only now and have realised that's pretty clear from section 2.8 ^^" so i guess what you do with this info depends on whether your expect your users to have read the ApJ paper before or after they start on the tutorial.

[ ] Some fundamental info should be flagged earlier "Since the seed is the unique identifier of each binary." I had to get to the middle of rewrite_5.py to see it written and that tutorial is quite buried but that seems like the kind of stuff you might want your user to know and understand quite earlier on. You do say in the Random Seed page that "each binary star will evolve using a unique random seed" but that didn't convey the notion that you use those as IDs, so maybe add a sentence there ;)
[ ] The random seeds are not always unique - the caveat needs to be flagged harder and best practice given So the random seeds are unique identifiers BUT at the bottom of the page on the random seeds you say:

"Unfortunately there is no reasonable way to protect the user against specifying duplicate random seeds – especially since the random seed increments for each star or binary star."

Sounds like maybe it should be flagged harder and bolder (and maybe repeated in the tutos where you say the random seeds are used as unique IDs, give us the caveat). How do you avoid this? It sounds like not mixing up the ways in which you call compas (like in the random seed example) would avoid most issues. If yes give us that directive at the top of the page, the rest of the page does a great job of explaining what's going on, but some people don't need the detail, they just need to know what's best practice.

Rapid Fire:

[ ] (docker from cl) "--number-of-binaries=5" is not a viable option for me, I think changed to --number-of-systems
[ ] still don't know what "--kick-mean-anomaly" means ^^"
[ ] h5copy.py page: can i have a use case at the start of the page before going into the nitty gritty so I know the context in which I'm reading this? (maybe link to the tuto as well).
[ ] step 4 in rewrite_H5.py has the comment 'DOUBLE CHECK ORDER :)' - does the order of these seeds matter?
[ ] in the COMPAS HDF5 tools is --filter a regex filter or what?
[ ] preProcessing is not mentioned in the user manual and it seems to have some old stuff in there - clean up needed.
[ ] tuto 2 in Cosmic Integration: redshift = np.array([2]) \ agesBirth = redshift. That makes redshift and agesBirth the same variable (they'll have the same memory address, if you change one you change the other) not a copy. Is that on purpose or where you just trying to give agesBirth the same kind of dummy value?
[ ] tuto 3 in Cosmic Integration reference should be flagged earlier? "If you use this pipeline we would appreciate it if you cite Selection effects https://arxiv.org/pdf/1711.06287" | that might be missed and is worth repeating - if you have a few applications like that which require different referencing, have a section of your read me and your readthedocs telling users what they need to reference depending on what they end up using. Get those citations yeah!
[ ] some page of the readthedocs should really be subsections and put together on the same page. That happens both in the user guide and the dev guide but the clearest examples are in the later, e.g. "Random number service" and other such pages where the functions and macros have individual pages when they really don't need to (it's not just easthetics, it makes that stuff harder to find when we need to dig through 4 or 5 levels), similarly the OOP section can be just one big page.
[x] Rooky question, what's the right arrow in the commands like LOGGING→Enabled(); is it a tab?
[x] SQUWAK :laughing: what's the difference in use btw SQUAWK and SAY ( also I see it's on the todo int he Log.cpp: clean up use of Squawk() vs SAY())
[x] Error handling service: "error marcros are in ErrorsMacros.h" -> i see many many more things in that .h than the 4 things on that page i'm a little confused but maybe it's because i dont c++
[ ] MAJOR RELEASES VS MINOR RELEASES. Do you also consider backwards compatibility? In some software minor releases ensure backwards compatibility .e.g. my pythonSubmit for v2.25 should run the same on 2.26 (even if the pythonSubmit for v2.26 might not run for v2.25 because the older version is missing a feature), whereas major releases indicate it might break your code. That's not always the case at all but i was just wondering.

Debug classes

Thanks for the reply Jeff, and for clarification i deleted the small post because I was going to post the question within this larger one, I'm putting it back here so things make sense

@jeffriley - finished reading the whole manual including the dev guide and there wasn't much more about debug classes so i jsut wanna make sure i udnerstand what you said ealier: Are the 'class strings' names of the classes within COMPAS, say I know my troubles are in BaseBinaryStar, then i give it the name of that class so i only get the debug statements from that class?

And to answer your question, yes I did, and since you're asking I'm guessing my interpretation is not correct lmao. And the dev guide is actually super good, I'm just a noob i that domain, so I'm afraid you're going to have to explain like I'm 5 :laughing: because it clearly went over my head.

some page of the readthedocs should really be subsections and put together on the same page.

That's probably true (i a number of places). As I said, a work in progress... The online documentation was put together fairly recently, and we still have to go through and rationalise it. You review will help us do that :-)

Rooky question, what's the right arrow in the commands like LOGGING→Enabled(); is it a tab?

In C/C++ (and other languages), dot-notation indicates object membership (i.e. 'a.x' means 'x' in object 'a'). The arrow notation (->) is used for pointers and derefences first: 'a->x' means x in the object pointed to by 'a'). See https://www.cplusplus.com/doc/tutorial/pointers/ for a discussion of pointers (a pointer variable doesn't contain the value of the variable, it contains the memory address of the value of the variable (it "points to" the variable)).

what's the difference in use btw SQUAWK and SAY

SQUAWK() writes to stderr; SAY() writes to stdout. Generally, the idea is that SAY() is for information; SQUAWK() is to complain or report an error condition.

Error handling service: "error marcros are in ErrorsMacros.h" -> i see many many more things in that .h than the 4 things on that page

There are 4 macros: SHOW_ERROR, SHOW_ERROR_IF, SHOW_WARN, and SHOW_WARN_IF - and then there are static versions of each.

Each of the 4 macros has 7 lines in ErrorsMacros.h - the multiple lines support calling the macros with a variable number of arguments (the '_#' indicates the number of arguments - the code to the right shows how the arguments are treated).

I'll add a more detailed description of debug classes later...

@HeloiseS :

What data am I expected to use for the Cosmic Integration tutorials?

Reason I'm asking is that i don't have a COMPAS output with 'BSE_Double_Compact_Objects' it seems. I used the output from the chirpmass distribution example (expecting that I'd defo have DCOs in there) and that didn't work.

Not sure about the chirp mass distribution example (sorry, I haven't tried it, though I agree with your surprise -- will check). In the meantime, you can use one of the recent COMPAS data sets from zenodo, http://zenodo.org/communities/compas/ , e.g., https://zenodo.org/record/5544170#.YYoihb1Bxp8 .

Ahh, I see that Lieke has a pending PR on the chirp mass distribution. ;-) https://github.com/TeamCOMPAS/COMPAS/pull/692/files

@HeloiseS -- update -- I don't think the pythonSubmit file in the chirpmass_distribution directory will produce any DCOs, or at least not enough to create a chirp mass distribution. We'll get that sorted (see issue above); in the meantime, please use one of the zenodo files.

@ilyamandel update -- my vaccine failed to properly install my 5G I think and I can't download the data set you recommended (I'm looking at DL speeds of <1GB/hour :cry: and it keeps giving me an error and failing ). If you have a recommendation for a smaller dataset I'll take it otherwise I'll just try to make a little simulation with all the things I need to run the tutos and play with them :)

Hi Everyone -- apologies for my radio silence so far! I've been swamped with other tasks but wanted to give a heads up that I will be focusing on finishing up my review next week. One quick thing I wanted to bring up for discussion is the need for 100% consistent docstrings in all the code. In the review criteria (https://joss.readthedocs.io/en/latest/review_criteria.html#api-documentation) the absolute requirement is that the API be documented, which I think is satisfied at the OK level at present (though I must confess I haven't gone 100% through the docs yet; next week though!). I agree with @HeloiseS that this is important, though, and would advocate for standardized docstrings. At the same time, I want to make sure that we aren't pushing too far past the JOSS requirements. Very happy to have a convo about it here though!

Apologies again for my slowness!

Agreed, the formatting standadarsation is a looser request - I do strongly recommend it because I can see COMPAS growing a lot in the future as more users get on it and the standards you establish now (and I am not just talking about docstrings) are things you're going to be carrying with you long term - the C++ has a solid and explicit set of standards and good practices followed throughout, and tbh these are of a higher standard than would be required with JOSS, but I still think it's worth the devs use this opportunity to think along similar lines for their post processing python pipelines. At the end of the day it's up to the dev team to decide whether they want to engage with that at the current time, and if they choose not to I won't make a fuss ;)

Going back to docstrings I do think there is a need to have complete docstrings in the core pipelines (and that is something I would consider a requirement, rather than a request) such as COMPASData though. Besides it's quick to copy what's in the notebook already into a dosctring, so it's a low "cost" / high impact improvement :)

update -- my vaccine failed to properly install my 5G I think and I can't download the data set you recommended (I'm looking at DL speeds of <1GB/hour 😢 and it keeps giving me an error and failing ). If you have a recommendation for a smaller dataset I'll take it otherwise I'll just try to make a little simulation with all the things I need to run the tutos and play with them :)

@HeloiseS: If I change number_of_systems = 5000 in the standard pythonSubmit file (say, the one in examples/methods_paper_plots/chirpmass_distribution), I get ~20 DCOs in the output file in a run that takes under 1 minute. That's not enough for pretty plots, but enough to get examples to run without failing.

The output file is still large enough (129 MB) that it will probably be faster for you to run it yourself than to download it, until your vaccine booster gives you superpowers or New Zealand broadband enters the 21st century. ;-)

@HeloiseS, @katiebreivik -- a lot of Heloise's comments [thank you for the very nice list!] are about the python scripts.

You are absolutely correct that they are not at a consistent level of code quality, maintenance, and documentation. Part of the problem is that many of the developers don't actually use these. :) Because the processing tools are very problem-specific, we tend to develop our own on an as-needed basis (sometimes not even in python -- e.g., personally, I do most of my processing in Matlab ;-) ).

We thought that making public some of the python scripts behind, say, the figures in the ApJS-submitted methods paper would be useful. However, these are being provided on an as-available basis. Realistically, we do not currently have the human resources to guarantee their quality and documentation at the requested level. And I certainly understand that providing bad/confusing/dysfunctional examples may be worse than providing none at all.

So, my question is, what is the best approach? I would be quite happy writing in big red letters that the only processing files we are providing as part of the core COMPAS distribution and take full responsibility for are h5copy.py, selection_effects.py, and FastCosmicIntegration.py. Everything else (including pythonSubmit.py) are just examples that the user may find helpful, but if not, they are not necessary for single or binary evolution or for processing the results. Would that be better? Would you prefer if we moved all other .py files into a different repository to avoid confusion?

Let's not throw the baby out with the bath water ;)

Because the processing tools are very problem-specific, so we tend to develop our own on an as-needed basis

So like you suggested it might be worth differentiating between the python processing utilities that you anticipate users will need ont he regular and have those in tutorials, and then the stuff that reproduces paper results and are very specific could go in a separate section where it's not flagged as a utility but as a "here is the code we used for this"?

(Sorry on mobile, hit enter too quick)

As for the pythonSubmit.py leave it in the tutorial and just streamline the file - Just removing the class structure would make it coherent with the use case, it doesn't need to be fancy ;) it really won't take that long and since that file is just a script there is not maintenance except for making sure that all the commands are in there and spelt right 😛 If keeping that maintained is too much, then yes put it in as an example and then tell people to add the extra commands by hand if it's broken when they get it.

@ilyamandel @jeffriley - is there an native option to make COMPAS use X cores that I missed or do i need to do that by calling it in a bash script or something (or maybe does it do it automatically but i'm just not running enough models for that to trigger).

@HeloiseS COMPAS is not threaded, but it doesn't need to be. You can just run X standalone runs on X cores, and provided you used appropriate starting random seeds (so there's no overlap between runs), you can just join all the output files together after the runs have completed. If you use HDF5 output files, the use h5copy.py to join the output files into a single HDF5 files.

I think we're testing a script now that will split a big run into several smaller runs - but I don't think it automatically joins the output files.

Hello everyone, just wanted to let you know that I am reading with interest, I appreciate all the effort going into these discussions. If anyone has any questions specifically about JOSS standards please do let me know.

"FINAL" UPDATE - tutorials ran perfectly

Alright! thanks @ilyamandel for the pointers and @jeffriley for the debugging assistance - I've played with making my own models and ran the tutorials on Cosmic Integration: they ran like clockwork!

I made some detailed models and some not-detailed models: the run time you mentioned was spot on :+1: Super fast! Suggestion: give instructions to make the models to run those tutorials as an alternative to having to DL a big data set? Plus to run the 6th tutorial you need several metallicities which gives an opportunity to demonstrate the use of the h5view.py and h5copy.py scripts in practice :) That would tie all of the examples and tutorials you have together to train your new users in all your utilities.

I have one more (it's more of a comment than a) question and then I'm done: i looked at the ZAMS of some of my model (just for fun :) ) compared it to BPASS and it all looks spotless, but there is that one dot a little lower than the rest ....

So I looked at the IMF and if you plot it weird (not loging the masses) you see ONE model at 8 Msol... which is weird because my minimum mass in my run is 10 Msol (which I double checked in the run log file). I can't imagine this would have any impact anywhere but since it's unexpected behaviour I thought I'd let you know :)

openjournals / joss-reviews

[REVIEW]: COMPAS: A rapid binary population synthesis suite #3838

Status

Reviewer instructions & questions

Review checklist for @katiebreivik

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @HeloiseS

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Further discussion on the `pythonSubmit.py` files

Program option list and default values

Initial Review

Bring user facing code to the same standard as the rest of the code

Your Python should be Pythonic

Other things

User Manual

Rapid Fire:

Debug classes

"FINAL" UPDATE - tutorials ran perfectly

openjournals / joss-reviews

[REVIEW]: COMPAS: A rapid binary population synthesis suite #3838

Status

Reviewer instructions & questions

Review checklist for @katiebreivik

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @HeloiseS

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Further discussion on the pythonSubmit.py files

Program option list and default values

Initial Review

Bring user facing code to the same standard as the rest of the code

Your Python should be Pythonic

Other things

User Manual

Rapid Fire:

Debug classes

"FINAL" UPDATE - tutorials ran perfectly

Further discussion on the `pythonSubmit.py` files