search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
697 stars 36 forks source link

[REVIEW]: PyXAB - A Python Library for X-Armed Bandit and Online Blackbox Optimization #6507

Open editorialbot opened 4 months ago

editorialbot commented 4 months ago

Submitting author: !--author-handle-->@WilliamLwj<!--end-author-handle-- (Wenjie Li) Repository: https://github.com/WilliamLwj/PyXAB Branch with paper.md (empty if default branch): paper Version: v0.3.0 Editor: !--editor-->@drvinceknight<!--end-editor-- Reviewers: @Otomisin, @KBodolai Archive: Pending

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/abceb86c9e7aa7419f3a29d6b64ceb7d"><img src="https://joss.theoj.org/papers/abceb86c9e7aa7419f3a29d6b64ceb7d/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/abceb86c9e7aa7419f3a29d6b64ceb7d/status.svg)](https://joss.theoj.org/papers/abceb86c9e7aa7419f3a29d6b64ceb7d)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@Otomisin & @KBodolai, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @drvinceknight know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @KBodolai

📝 Checklist for @Otomisin

editorialbot commented 4 months ago

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf
editorialbot commented 4 months ago

Software report:

github.com/AlDanial/cloc v 1.90  T=0.08 s (1445.7 files/s, 248507.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          72           1557           2390           3709
reStructuredText                33            603           1005            598
TeX                              1            124              2            439
Jupyter Notebook                 3              0           8598            329
Markdown                         2             82              0            320
YAML                             3             15             35             79
DOS Batch                        1              8              1             26
make                             1              4              7              9
-------------------------------------------------------------------------------
SUM:                           116           2393          12038           5509
-------------------------------------------------------------------------------

Commit count by author:

   130  WilliamLi
    94  Williamlwj
    49  talhz
     8  Giggfitnesse
     3  William
editorialbot commented 4 months ago

Paper file info:

📄 Wordcount for paper.md is 1475

✅ The paper includes a Statement of need section

editorialbot commented 4 months ago

License info:

✅ License found: MIT License (Valid open source OSI approved license)

editorialbot commented 4 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

editorialbot commented 4 months ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1007/s10208-015-9296-2 is OK
- 10.1145/1374376.1374475 is OK
- 10.1287/moor.2021.1220 is OK
- 10.48550/ARXIV.1807.02811 is OK
- 10.1109/TIT.2015.2409256 is OK
- 10.48550/ARXIV.2205.15268 is OK

MISSING DOIs

- No DOI given, and none found for title: Hyperband: A Novel Bandit-Based Approach to Hyperp...
- 10.1007/bf00941892 may be a valid DOI for title: Lipschitzian optimization without the Lipschitz co...
- No DOI given, and none found for title: Derivative-Free Order-Robust Optimisation
- No DOI given, and none found for title: Contextual Bandits with Linear Payoff Functions
- 10.1007/978-3-030-64228-0_4 may be a valid DOI for title: Asset Management in Electrical Utilities in the Co...
- No DOI given, and none found for title: Google vizier: A service for black-box optimizatio...
- No DOI given, and none found for title: Prediction, Learning, and Games
- No DOI given, and none found for title: Bandit algorithms
- No DOI given, and none found for title: General parallel optimization a without metric
- No DOI given, and none found for title: Online stochastic optimization under correlated ba...
- No DOI given, and none found for title: Empirical Bernstein bounds and sample variance pen...
- No DOI given, and none found for title: Mathematics of statistical sequential decision mak...
- 10.1016/j.tcs.2009.01.016 may be a valid DOI for title: Exploration–exploitation tradeoff using variance e...
- No DOI given, and none found for title: χ-Armed Bandits
- No DOI given, and none found for title: Black-box optimization of noisy functions with unk...
- No DOI given, and none found for title: A simple parameter-free and adaptive approach to o...
- No DOI given, and none found for title: Optimistic Optimization of a Deterministic Functio...
- 10.1613/jair.4742 may be a valid DOI for title: Global continuous optimization with error bound an...
- No DOI given, and none found for title: Stochastic Simultaneous Optimistic Optimization
- 10.1007/978-3-540-72927-3_33 may be a valid DOI for title: Improved Rates for the Stochastic Continuum-Armed ...
- No DOI given, and none found for title: Optimum-statistical Collaboration Towards General ...
- 10.2139/ssrn.2661896 may be a valid DOI for title: Online decision making with high-dimensional covar...
- No DOI given, and none found for title: Federated Bayesian Optimization via Thompson Sampl...
- 10.1609/aaai.v35i11.17156 may be a valid DOI for title: Federated Multi-Armed Bandits
- No DOI given, and none found for title:  Federated Multi-armed Bandits with Personalizatio...
- No DOI given, and none found for title: Federated Linear Contextual Bandits
- No DOI given, and none found for title: Communication-efficient learning of deep networks ...
- No DOI given, and none found for title: Federated Online Sparse Decision Making
- 10.1145/3543516.3453919 may be a valid DOI for title: Federated bandit: A gossiping approach
- 10.1109/isit44484.2020.9174297 may be a valid DOI for title: Federated recommendation system via differential p...
- No DOI given, and none found for title: Differentially Private Federated Bayesian Optimiza...
- No DOI given, and none found for title: Federated Hyperparameter Tuning: Challenges, Basel...
- 10.3386/w29180 may be a valid DOI for title: Testing fractional doses of COVID-19 vaccines
- No DOI given, and none found for title: Semi-Supervised Multitask Learning
- 10.1109/tit.2023.3312308 may be a valid DOI for title: Lipschitz Bandits with Batched Feedback
- 10.1609/aaai.v30i1.10212 may be a valid DOI for title: Algorithms for Differentially Private Multi-Armed ...
- No DOI given, and none found for title: Batched Large-scale Bayesian Optimization in High-...
- No DOI given, and none found for title: Differentially Private Contextual Linear Bandits
- No DOI given, and none found for title: Differentially-Private Federated Linear Bandits
- No DOI given, and none found for title: Differential Privacy Under Continual Observation
- 10.1109/jproc.2015.2494218 may be a valid DOI for title: Taking the Human Out of the Loop: A Review of Baye...
- No DOI given, and none found for title: An Optimal Algorithm for Bandit and Zero-Order Con...

INVALID DOIs

- None
drvinceknight commented 4 months ago

👋🏼 @WilliamLwj @Otomisin, @KBodolai this is the review thread for the paper. All of our communications will happen here from now on.

As a reviewer, the first step is to create a checklist for your review by entering

@editorialbot generate my checklist

as the top of a new comment in this thread.

These checklists contain the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. The first comment in this thread also contains links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#REVIEW_NUMBER so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 2-4 weeks. Please let me know if any of you require some more time. We can also use EditorialBot (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@drvinceknight) if you have any questions/concerns.

KBodolai commented 4 months ago

Review checklist for @KBodolai

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

drvinceknight commented 2 months ago

Hi @Otomisin @KBodolai how are things going?

Otomisin commented 2 months ago

Review checklist for @Otomisin

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Otomisin commented 2 months ago

Hi @Otomisin @KBodolai how are things going? Hi @drvinceknight. I should get back with full review in coming week. Apologies for the delay. I am left with checking the functionality

Otomisin commented 2 months ago

@drvinceknight I think I am done with the review, Regarding, A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?

The paper does not have a section explicitly titled "Statement of need," but the content relevant to this purpose is dispersed throughout the introduction and other sections

Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item. _There does not seem to be a need for data sharing, hence data was not provided, there is however a sample data for real life examle here _

Functionality: Have the functional claims of the software been confirmed? I made a test by recreating with the example (1-D Example) on my personal PC, from 1-D Example

Please let me know if anything else is needed from my side.

crvernon commented 1 month ago

:wave: @drvinceknight - could you please check in on this one and keep things going? Thanks!

crvernon commented 1 month ago

👋 @drvinceknight - Just raising this one to the top of your notifications...

👋 @drvinceknight - could you please check in on this one and keep things going? Thanks!

drvinceknight commented 3 weeks ago

Thanks for your review @Otomisin. @WilliamLwj could you let me know how revisions are going?

WilliamLwj commented 3 weeks ago

Thanks for your review @Otomisin. @WilliamLwj could you let me know how revisions are going?

Hi @drvinceknight, to be honest, we are a bit unsure what we should do next because we have't heard back for quite long, and this is our first time submitting to JOSS. We thought we should wait for the reviews from @KBodolai to be completed before we start prepare the revisions. Also, could you let us know how to upload the revisions?

In the meantime, we do not see a lot of changes requested by the reviewers. The two (potential) improvements we identify are:

  1. Add missing DOIs for the references
  2. Add a "statement of need" section that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work.

Please kindly let us know whether the above changes are needed, and what our next steps should be. Thank you very much.

drvinceknight commented 3 weeks ago

We thought we should wait for the reviews from @KBodolai to be completed before we start prepare the revisions.

@KBodolai would you be able to follow up on your review?

Also, could you let us know how to upload the revisions?

If you make revisions to the paper and commit them to the paper repository that would be good. Once you have done that you can ping the reviewers here to let them know.

In the meantime, we do not see a lot of changes requested by the reviewers. The two (potential) improvements we identify are:

  1. Add missing DOIs for the references
  2. Add a "statement of need" section that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work.

That looks right to me from @Otomisin's review.

WilliamLwj commented 3 weeks ago

We thought we should wait for the reviews from @KBodolai to be completed before we start prepare the revisions.

@KBodolai would you be able to follow up on your review?

Also, could you let us know how to upload the revisions?

If you make revisions to the paper and commit them to the paper repository that would be good. Once you have done that you can ping the reviewers here to let them know.

In the meantime, we do not see a lot of changes requested by the reviewers. The two (potential) improvements we identify are:

  1. Add missing DOIs for the references
  2. Add a "statement of need" section that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work.

That looks right to me from @Otomisin's review.

Hi @drvinceknight, thanks for getting back to us so quickly. Regarding the two areas to improve,

  1. We have found that some references in the bib file are actually unused in the paper and removed them. For the remaining references, some of the papers do not have DOIs, for example, Derivative-free order-robust optimisation (Ammar, H. et al 2020) so we cannot add any DOIs to them. We have added DOIs to the rest of the references.

  2. This point confuses us because we have a "Statement of need" section, which is our section 2 in the paper. You can see that from the pdf generated in editorialbot message on Mar 19. Therefore, we are quite confused by what @Otomisin mean when saying our paper does not have this section.

We have committed all of our changes to the paper and you can find the newest version in our Github actions.

@editorialbot generate pdf