search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
725 stars 38 forks source link

[REVIEW]: PyXAB - A Python Library for X-Armed Bandit and Online Blackbox Optimization #6507

Closed editorialbot closed 1 month ago

editorialbot commented 8 months ago

Submitting author: !--author-handle-->@WilliamLwj<!--end-author-handle-- (Wenjie Li) Repository: https://github.com/WilliamLwj/PyXAB Branch with paper.md (empty if default branch): paper Version: v0.3.0-paper Editor: !--editor-->@drvinceknight<!--end-editor-- Reviewers: @Otomisin, @KBodolai Archive: 10.5281/zenodo.13963754

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/abceb86c9e7aa7419f3a29d6b64ceb7d"><img src="https://joss.theoj.org/papers/abceb86c9e7aa7419f3a29d6b64ceb7d/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/abceb86c9e7aa7419f3a29d6b64ceb7d/status.svg)](https://joss.theoj.org/papers/abceb86c9e7aa7419f3a29d6b64ceb7d)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@Otomisin & @KBodolai, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @drvinceknight know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @KBodolai

📝 Checklist for @Otomisin

editorialbot commented 8 months ago

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf
editorialbot commented 8 months ago

Software report:

github.com/AlDanial/cloc v 1.90  T=0.08 s (1445.7 files/s, 248507.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          72           1557           2390           3709
reStructuredText                33            603           1005            598
TeX                              1            124              2            439
Jupyter Notebook                 3              0           8598            329
Markdown                         2             82              0            320
YAML                             3             15             35             79
DOS Batch                        1              8              1             26
make                             1              4              7              9
-------------------------------------------------------------------------------
SUM:                           116           2393          12038           5509
-------------------------------------------------------------------------------

Commit count by author:

   130  WilliamLi
    94  Williamlwj
    49  talhz
     8  Giggfitnesse
     3  William
editorialbot commented 8 months ago

Paper file info:

📄 Wordcount for paper.md is 1475

✅ The paper includes a Statement of need section

editorialbot commented 8 months ago

License info:

✅ License found: MIT License (Valid open source OSI approved license)

editorialbot commented 8 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

editorialbot commented 8 months ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1007/s10208-015-9296-2 is OK
- 10.1145/1374376.1374475 is OK
- 10.1287/moor.2021.1220 is OK
- 10.48550/ARXIV.1807.02811 is OK
- 10.1109/TIT.2015.2409256 is OK
- 10.48550/ARXIV.2205.15268 is OK

MISSING DOIs

- No DOI given, and none found for title: Hyperband: A Novel Bandit-Based Approach to Hyperp...
- 10.1007/bf00941892 may be a valid DOI for title: Lipschitzian optimization without the Lipschitz co...
- No DOI given, and none found for title: Derivative-Free Order-Robust Optimisation
- No DOI given, and none found for title: Contextual Bandits with Linear Payoff Functions
- 10.1007/978-3-030-64228-0_4 may be a valid DOI for title: Asset Management in Electrical Utilities in the Co...
- No DOI given, and none found for title: Google vizier: A service for black-box optimizatio...
- No DOI given, and none found for title: Prediction, Learning, and Games
- No DOI given, and none found for title: Bandit algorithms
- No DOI given, and none found for title: General parallel optimization a without metric
- No DOI given, and none found for title: Online stochastic optimization under correlated ba...
- No DOI given, and none found for title: Empirical Bernstein bounds and sample variance pen...
- No DOI given, and none found for title: Mathematics of statistical sequential decision mak...
- 10.1016/j.tcs.2009.01.016 may be a valid DOI for title: Exploration–exploitation tradeoff using variance e...
- No DOI given, and none found for title: χ-Armed Bandits
- No DOI given, and none found for title: Black-box optimization of noisy functions with unk...
- No DOI given, and none found for title: A simple parameter-free and adaptive approach to o...
- No DOI given, and none found for title: Optimistic Optimization of a Deterministic Functio...
- 10.1613/jair.4742 may be a valid DOI for title: Global continuous optimization with error bound an...
- No DOI given, and none found for title: Stochastic Simultaneous Optimistic Optimization
- 10.1007/978-3-540-72927-3_33 may be a valid DOI for title: Improved Rates for the Stochastic Continuum-Armed ...
- No DOI given, and none found for title: Optimum-statistical Collaboration Towards General ...
- 10.2139/ssrn.2661896 may be a valid DOI for title: Online decision making with high-dimensional covar...
- No DOI given, and none found for title: Federated Bayesian Optimization via Thompson Sampl...
- 10.1609/aaai.v35i11.17156 may be a valid DOI for title: Federated Multi-Armed Bandits
- No DOI given, and none found for title:  Federated Multi-armed Bandits with Personalizatio...
- No DOI given, and none found for title: Federated Linear Contextual Bandits
- No DOI given, and none found for title: Communication-efficient learning of deep networks ...
- No DOI given, and none found for title: Federated Online Sparse Decision Making
- 10.1145/3543516.3453919 may be a valid DOI for title: Federated bandit: A gossiping approach
- 10.1109/isit44484.2020.9174297 may be a valid DOI for title: Federated recommendation system via differential p...
- No DOI given, and none found for title: Differentially Private Federated Bayesian Optimiza...
- No DOI given, and none found for title: Federated Hyperparameter Tuning: Challenges, Basel...
- 10.3386/w29180 may be a valid DOI for title: Testing fractional doses of COVID-19 vaccines
- No DOI given, and none found for title: Semi-Supervised Multitask Learning
- 10.1109/tit.2023.3312308 may be a valid DOI for title: Lipschitz Bandits with Batched Feedback
- 10.1609/aaai.v30i1.10212 may be a valid DOI for title: Algorithms for Differentially Private Multi-Armed ...
- No DOI given, and none found for title: Batched Large-scale Bayesian Optimization in High-...
- No DOI given, and none found for title: Differentially Private Contextual Linear Bandits
- No DOI given, and none found for title: Differentially-Private Federated Linear Bandits
- No DOI given, and none found for title: Differential Privacy Under Continual Observation
- 10.1109/jproc.2015.2494218 may be a valid DOI for title: Taking the Human Out of the Loop: A Review of Baye...
- No DOI given, and none found for title: An Optimal Algorithm for Bandit and Zero-Order Con...

INVALID DOIs

- None
drvinceknight commented 8 months ago

👋🏼 @WilliamLwj @Otomisin, @KBodolai this is the review thread for the paper. All of our communications will happen here from now on.

As a reviewer, the first step is to create a checklist for your review by entering

@editorialbot generate my checklist

as the top of a new comment in this thread.

These checklists contain the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. The first comment in this thread also contains links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#REVIEW_NUMBER so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 2-4 weeks. Please let me know if any of you require some more time. We can also use EditorialBot (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@drvinceknight) if you have any questions/concerns.

KBodolai commented 8 months ago

Review checklist for @KBodolai

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

drvinceknight commented 6 months ago

Hi @Otomisin @KBodolai how are things going?

Otomisin commented 6 months ago

Review checklist for @Otomisin

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Otomisin commented 6 months ago

Hi @Otomisin @KBodolai how are things going? Hi @drvinceknight. I should get back with full review in coming week. Apologies for the delay. I am left with checking the functionality

Otomisin commented 6 months ago

@drvinceknight I think I am done with the review, Regarding, A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?

The paper does not have a section explicitly titled "Statement of need," but the content relevant to this purpose is dispersed throughout the introduction and other sections

Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item. _There does not seem to be a need for data sharing, hence data was not provided, there is however a sample data for real life examle here _

Functionality: Have the functional claims of the software been confirmed? I made a test by recreating with the example (1-D Example) on my personal PC, from 1-D Example

Please let me know if anything else is needed from my side.

crvernon commented 5 months ago

:wave: @drvinceknight - could you please check in on this one and keep things going? Thanks!

crvernon commented 5 months ago

👋 @drvinceknight - Just raising this one to the top of your notifications...

👋 @drvinceknight - could you please check in on this one and keep things going? Thanks!

drvinceknight commented 4 months ago

Thanks for your review @Otomisin. @WilliamLwj could you let me know how revisions are going?

WilliamLwj commented 4 months ago

Thanks for your review @Otomisin. @WilliamLwj could you let me know how revisions are going?

Hi @drvinceknight, to be honest, we are a bit unsure what we should do next because we have't heard back for quite long, and this is our first time submitting to JOSS. We thought we should wait for the reviews from @KBodolai to be completed before we start prepare the revisions. Also, could you let us know how to upload the revisions?

In the meantime, we do not see a lot of changes requested by the reviewers. The two (potential) improvements we identify are:

  1. Add missing DOIs for the references
  2. Add a "statement of need" section that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work.

Please kindly let us know whether the above changes are needed, and what our next steps should be. Thank you very much.

drvinceknight commented 4 months ago

We thought we should wait for the reviews from @KBodolai to be completed before we start prepare the revisions.

@KBodolai would you be able to follow up on your review?

Also, could you let us know how to upload the revisions?

If you make revisions to the paper and commit them to the paper repository that would be good. Once you have done that you can ping the reviewers here to let them know.

In the meantime, we do not see a lot of changes requested by the reviewers. The two (potential) improvements we identify are:

  1. Add missing DOIs for the references
  2. Add a "statement of need" section that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work.

That looks right to me from @Otomisin's review.

WilliamLwj commented 4 months ago

We thought we should wait for the reviews from @KBodolai to be completed before we start prepare the revisions.

@KBodolai would you be able to follow up on your review?

Also, could you let us know how to upload the revisions?

If you make revisions to the paper and commit them to the paper repository that would be good. Once you have done that you can ping the reviewers here to let them know.

In the meantime, we do not see a lot of changes requested by the reviewers. The two (potential) improvements we identify are:

  1. Add missing DOIs for the references
  2. Add a "statement of need" section that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work.

That looks right to me from @Otomisin's review.

Hi @drvinceknight, thanks for getting back to us so quickly. Regarding the two areas to improve,

  1. We have found that some references in the bib file are actually unused in the paper and removed them. For the remaining references, some of the papers do not have DOIs, for example, Derivative-free order-robust optimisation (Ammar, H. et al 2020) so we cannot add any DOIs to them. We have added DOIs to the rest of the references.

  2. This point confuses us because we have a "Statement of need" section, which is our section 2 in the paper. You can see that from the pdf generated in editorialbot message on Mar 19. Therefore, we are quite confused by what @Otomisin mean when saying our paper does not have this section.

We have committed all of our changes to the paper and you can find the newest version in our Github actions.

@editorialbot generate pdf

WilliamLwj commented 3 months ago

Hi @drvinceknight , are there any updates regarding the status of our paper? Kindly let us know, thank you!

drvinceknight commented 3 months ago

@KBodolai would you be able to take a look please (to confirm if you're happy with the changes)?

KBodolai commented 3 months ago

Hi everyone, apologies to have been bottlenecking this!

I've set aside some time tomorrow and friday to finish my review, I'll ping you here when it's done.

KBodolai commented 3 months ago

Apologies again for the delay. I've now completed my checklist and my review, I added a few comments and minor things on WilliamLwj/PyXAB#37.

WilliamLwj commented 3 months ago

Hi @KBodolai, thank you for your review! We have had an update for the paper and the repository to address your comments.

  1. Minor typo in the second paragraph of the statement of need: ``arms". is started with a double backtick and finishes with double quotes.

Addressed this in the paper. Thank you! You can find the newest version of the paper here (In the generated artifacts)

  1. Just after that, I believe there's a missing "be": just before "infeasible" in the following statement Therefore, directly applying multi-armed bandit algorithms to such problems would infeasible

Same as the above. Thank you for finding these typos!

  1. The Quick Example section in the Readme could be made a bit more specific and self-evident by including the specific imports used in it.

Thanks! We have updated the quick example in the README. There is only one thing to import which is the HOO algorithm. You can find the new README here

  1. Love the real world example - if I were to add anything here, it would be a comparison with other algorithms commonly used for hyperparameter tuning (time, optimality)

Thank you! But just to clarify, are you referring to the real-life example in our doc? We can provide some comparisons with other hyper-parameter tuning algorithms, but we would like to know what algorithms you think are the most important to compare with. Thank you!

We would love to see your PR. Many thanks in advance!

KBodolai commented 3 months ago

Fantastic, very happy with the changes, thanks for being so quick!

I was referring to the docs example, yes. At the end is a simple example, so I'd probably just go with some baseline implementation in sklearn, such as RandomizedSearchCV, you could also go "fancier" and compare it to other methods, but I feel like that would be more of a benchmark than an example, and in practice many people stick to these simple methods anyway!

I may not have time in the following couple of weeks - but I sugget we move on with the publication of the paper, since this last point is a minor, incremental improvement on an otherwise good documentation.

@drvinceknight , I think this completes my review, let me know if I need to do anything else.

WilliamLwj commented 2 months ago

Hi @drvinceknight, we believe we have addressed both reviewers’ comments. Could you kindly let us know what our next steps should be? Thank you!

drvinceknight commented 2 months ago

Thanks @WilliamLwj this all looks good! There's a few more things I need to do now (but this shouldn't take long).

drvinceknight commented 2 months ago

@editorialbot generate pdf

editorialbot commented 2 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

drvinceknight commented 2 months ago

@editorialbot check references

editorialbot commented 2 months ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.1145/1374376.1374475 is OK
- 10.1109/JPROC.2015.2494218 is OK
- 10.1109/TIT.2015.2409256 is OK
- 10.1609/aaai.v38i12.29267 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: Hyperband: A Novel Bandit-Based Approach to Hyperp...
- No DOI given, and none found for title: Derivative-Free Order-Robust Optimisation
- No DOI given, and none found for title: General parallel optimization a without metric
- No DOI given, and none found for title: Online stochastic optimization under correlated ba...
- No DOI given, and none found for title: χ-Armed Bandits
- No DOI given, and none found for title: Black-box optimization of noisy functions with unk...
- No DOI given, and none found for title: A simple parameter-free and adaptive approach to o...
- No DOI given, and none found for title: Optimistic Optimization of a Deterministic Functio...
- No DOI given, and none found for title: Stochastic Simultaneous Optimistic Optimization
- No DOI given, and none found for title: Optimum-statistical Collaboration Towards General ...
- No DOI given, and none found for title: Federated High-Dimensional Online Decision Making
- No DOI given, and none found for title: An Optimal Algorithm for Bandit and Zero-Order Con...
- No DOI given, and none found for title: Personalized Federated X-armed Bandit

❌ MISSING DOIs

- None

❌ INVALID DOIs

- None
drvinceknight commented 2 months ago

Thanks for your patience everyone. This looks good to me now.

@WilliamLwj could you make a tagged release and archive, and report the version number and archive DOI in the review thread here please.

WilliamLwj commented 2 months ago

Hi @drvinceknight, yes I have made a tagged release and archived the release to Zenodo, the version number is v0.3.0-paper, and the DOI is 10.5281/zenodo.13743085

WilliamLwj commented 2 months ago

Hi @drvinceknight, we had some last minute changes on the paper, which is already pushed to our repository. (see commit) Specifically, Prof. Jean Honorio (https://www.cs.purdue.edu/homes/jhonorio/) moved from Purdue University to The University of Melbourne in Australia earlier this year during the review period, and we have updated his affiliation information.

Also, we tried to add one footnote saying that this work was done before Wenjie joined Amazon (required by Amazon), but that didn't show up in the generated pdf so we just decided to leave it like that.

Please let us know whether this is ok and if you have any concerns/questions. Many thanks in advance.

WilliamLwj commented 2 months ago

Hi @drvinceknight @crvernon, could you let us know whether there are any updates? Thanks!

drvinceknight commented 2 months ago

Hi @drvinceknight, we had some last minute changes on the paper, which is already pushed to our repository. (see commit) Specifically, Prof. Jean Honorio (https://www.cs.purdue.edu/homes/jhonorio/) moved from Purdue University to The University of Melbourne in Australia earlier this year during the review period, and we have updated his affiliation information.

Also, we tried to add one footnote saying that this work was done before Wenjie joined Amazon (required by Amazon), but that didn't show up in the generated pdf so we just decided to leave it like that.

Please let us know whether this is ok and if you have any concerns/questions. Many thanks in advance.

Thanks for this, could you up date the release and the doi?

WilliamLwj commented 2 months ago

Thanks for this, could you up date the release and the doi?

Hi @drvinceknight, yes, I have updated the release and the new DOI is DOI: 10.5281/zenodo.13835534

WilliamLwj commented 1 month ago

Hi @drvinceknight, could you let us know whether there are any updates?

drvinceknight commented 1 month ago

@editorialbot set 10.5281/zenodo.13835534 as archive

editorialbot commented 1 month ago

Done! archive is now 10.5281/zenodo.13835534

drvinceknight commented 1 month ago

@editorialbot recommend-accept

editorialbot commented 1 month ago 
Attempting dry run of processing paper acceptance...
editorialbot commented 1 month ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.1145/1374376.1374475 is OK
- 10.1109/JPROC.2015.2494218 is OK
- 10.1109/TIT.2015.2409256 is OK
- 10.1609/aaai.v38i12.29267 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: Hyperband: A Novel Bandit-Based Approach to Hyperp...
- No DOI given, and none found for title: Derivative-Free Order-Robust Optimisation
- No DOI given, and none found for title: General parallel optimization a without metric
- No DOI given, and none found for title: Online stochastic optimization under correlated ba...
- No DOI given, and none found for title: χ-Armed Bandits
- No DOI given, and none found for title: Black-box optimization of noisy functions with unk...
- No DOI given, and none found for title: A simple parameter-free and adaptive approach to o...
- No DOI given, and none found for title: Optimistic Optimization of a Deterministic Functio...
- No DOI given, and none found for title: Stochastic Simultaneous Optimistic Optimization
- No DOI given, and none found for title: Optimum-statistical Collaboration Towards General ...
- No DOI given, and none found for title: Federated High-Dimensional Online Decision Making
- No DOI given, and none found for title: An Optimal Algorithm for Bandit and Zero-Order Con...
- No DOI given, and none found for title: Personalized Federated X-armed Bandit

❌ MISSING DOIs

- None

❌ INVALID DOIs

- None
editorialbot commented 1 month ago

:warning: Error preparing paper acceptance. The generated XML metadata file is invalid. 

ID tabU003AU0020summary already defined
drvinceknight commented 1 month ago

@danielskatz do you have any idea what I might be missing here?

danielskatz commented 1 month ago

Just from the error message, it looks like a table label might be multiply defined. I would search the .md file for that ID, or perhaps just "summary"

WilliamLwj commented 1 month ago

Hi @danielskatz @drvinceknight, I just checked our .md file and I only found three places where I used the word "summary",

  1. The section Summary
  2. The sentence In Table \ref{tab: summary}, we provide the comparison among some of the algorithms implemented in PyXAB...
  3. The label \label{tab: summary}

I'm actually baffled, where exactly is the table label multiply-defined? Please find our paper .md file here

Thank you very much.

danielskatz commented 1 month ago

The issue might be the space in the label - I would change both instances of tab: summary to tab:summary and see if that fixes the problem.

WilliamLwj commented 1 month ago

The issue might be the space in the label - I would change both instances of tab: summary to tab:summary and see if that fixes the problem.

Thank you @danielskatz! We have removed the spaces in both instances and published a new release after the change. @drvinceknight Could you use 10.5281/zenodo.13835537 to try again? Thank you!

danielskatz commented 1 month ago

@WilliamLwj - JOSS doesn't require the paper to be in the repo, so that wasn't strictly needed, but I'll see how it looks...

danielskatz commented 1 month ago

@editorialbot set 10.5281/zenodo.13835537 as archive