Closed editorialbot closed 1 year ago
editorialbot
commented
1 year ago Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.
For a list of things I can do to help you, just type:
@editorialbot commandsFor example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:
@editorialbot generate pdfSoftware report:
github.com/AlDanial/cloc v 1.88 T=0.09 s (908.1 files/s, 112361.7 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
Python 52 1035 1350 2198
Jupyter Notebook 6 0 3218 1192
Markdown 7 79 0 291
reStructuredText 10 166 166 142
Bourne Shell 1 31 9 141
YAML 4 10 4 117
TeX 1 9 0 112
TOML 1 13 4 73
Solidity 1 0 0 13
make 1 4 7 9
-------------------------------------------------------------------------------
SUM: 84 1347 4758 4288
-------------------------------------------------------------------------------
gitinspector failed to run statistical information for the repositoryReference check summary (note 'MISSING' DOIs are suggestions that need verification):
OK DOIs
- 10.1007/978-1-4419-1665-5_13 is OK
- 10.1007/s10732-018-9377-x is OK
- 10.1287/trsc.1050.0135 is OK
- 10.1016/j.cor.2022.105903 is OK
- 10.48550/arxiv.2211.00759 is OK
- 10.1016/j.cor.2022.106089 is OK
- 10.1016/j.cor.2021.105643 is OK
- 10.1006/jcph.1993.1010 is OK
- 10.1007/s12532-021-00209-7 is OK
- 10.1016/j.ejor.2021.05.042 is OK
MISSING DOIs
- None
INVALID DOIs
- NoneWordcount for paper.md is 739
editorialbot
commented
1 year ago :point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
skadio
commented
1 year ago skadio
commented
1 year ago I played with this library, and first things first. I LOVE it! Kudos to @N-Wouda and the team!
The summary of my JOSS review, minimum action items needed for publication, and my detailed commentary is below.
At a minimum, we need:
The notebooks and documentations are great! Someone with an OR background should not have any problem using the library. That said, we need a bit more on the example usage for outsiders. With a quick low-hanging effort, the library can target a larger audience --which I believe is where largest potential is. To inspire the authors, I took a stab at this below. More on this in details below.
See the JOSS required item above to please add "State of the Field" to cover how this package compares to other packages. If none such exists (which I doubt it, since stochastic local search is well-known), please make the contribution explicit. If there are other tools, please cover mainstream ones, if any and mention where each would benefit the most (e.g., genetic algorithms for multi-point solutions vs. alns)
See the JOSS required item above to please add Community Guidelines, if not already.
skadio
commented
1 year ago Detailed comments:
My main concern is that (for a non-OR user), the "minimum" set of requirements is not clearly defined. What does the user need to bring to the table to benefit from this library.
The current README only provides a quick comment, and while the Documentations is quite nice/detailed, it does not provide a crisp answer to a newcomer.
Any optimization expert should be able to consume the library but with JOSS, there is a potential for a larger audience (who are technical experts, say a Kaggler who wants to use this library in a competition, just not familiar with optimization know-how/jargon).
What would help the most is a high-level, problem independent boilerplate implementation with TODO statements to guide the user to fill for their first ALNS model. Specific problem implementations in the documents can then become extremely useful for detailed applications.
Here is one example boiler plate. I am sharing it here, so that the authors can, first of all, correct me if my understanding of the library is incorrect.
I believe, at a MINIMUM, the library requires the user to define:
Is this correct? If so, here is my boiler plate:
skadio
commented
1 year ago import numpy.random as rnd
from alns import ALNS, State
from alns.select import RouletteWheel
from alns.accept import HillClimbing
from alns.stop import MaxRuntime
class MyState(State):
"""
Problem specific solution state
"""
def __init__(self, my_parameter):
self.my_parameter = my_parameter
def objective(self) -> float:
"""
This function calculates the objective of the state.
Remember, ALNS works minimization problems!
"""
# TODO implement the objective function
return NotImplementedError()
def create_initial_state() -> MyState:
# TODO implement the creation of an initial state and return it
# Create a state and setup the initial solution
state = MyState("My problem specific state")
return NotImplementedError()
def my_state_destroy(state: MyState) -> MyState:
# TODO: implement how destroy a given state and return the destroyed state
# Remember to destroy a copy!
destroyed = copy.deepcopy(state)
return NotImplementedError()
def my_state_repair(state: MyState) -> MyState:
# TODO: implement how repair a given state and return repaired state
return NotImplementedError()
# Create ALNS
alns = ALNS(rnd.RandomState(seed=42))
# Add one or more destroy and repair operators to alns
alns.add_destroy_operator(my_state_destroy)
alns.add_repair_operator(my_state_repair)
# Configurate ALNS
select = RouletteWheel() # See alns.select for others
accept = HillClimbing() # See alns.accept for others
stop = MaxRuntime(60) # See alns.stop for others
# Create the initial solution
initial_solution = create_initial_state()
print("Initial solution objective is ", initial_solution.objective()")
# Run ALNS
result = alns.iterate(initial_solution, select, accept, stop)
# Retrieve the final solution
solution = result.best_state
objective = solution.objective()
print("Best heuristic objective is:", objective)
You can correct this as needed, hope the idea is clear. I tried to add useful commentary, type hints etc. to guide the user. Please extend from there.
In the README:
In the PAPER:
skadio
commented
1 year ago Detailed Comments Continued:
Some minor code/low-level commentaries, mostly for my curiosity:
skadio
commented
1 year ago Finally, one high-level Research Comment, completely independent of the JOSS publication here.
How you considered the integration of ALNS library with Optimization Modeling/Solving technology? For example, ALNS is well-known to be great complement to Constraint Programming, see e.g., the famous P. Shaw paper from 1998 https://link.springer.com/chapter/10.1007/3-540-49481-2_30
These ideas have found concrete implementations in libraries, e.g.,
Here, the ALNS part is "generic" whereas the problem specific details are left to the user. Beyond vanilla problems, where there are a lot of side constraints, the user might have trouble "modeling" the state, in the first place.
What would be amazing is IF one could take ALNS, and then in the STATE description to use an off-the-shelf solver to model and solve the problem. This is probably easier said than done, but I believe there is a tremendous potential if we could build out the path for such integration.
Let's take Google OR-Tools for instances, since it is a great performing solver, has a high-level modeling language with support for many interesting constraints and available in Python. I imagine the STATE can be the initial Constraint model. Notice that, by solving it (potentially without an objective function), IMMEDIATELY generates an initial solution. So this initial construction comes for free (alleviating the user further). In some problems, especially with many constraints, even finding this solution can be tricky. In the Documentation I noticed, some simple/clever tricks are used like setting the first variable to 1, for instance. Then the relax/repair operators need to work off of the solvers variables.
I think this should be doable, and would give best of both worlds: ability to solve optimization problems at scale (thanks to local search) as opposed to exact solutions (which might not scale, e.g., mixed integer programs) BUT with the modeling support (as opposed to tradition local search where the user must implement everything from scratch).
I think there is vital gap in the Optimization community for this, especially within the Python tech stack which also is the main technology powering Data Science/ML/AI community. Feel free to connect with me later on, if this research directions sounds interesting. I would be happy to brainstorm.
(btw, feel free to use these resources in the State of Field, if relevant)
skadio
commented
1 year ago To summarize: great library/paper for publication at JOSS. Only a few minimum items to revise, which should be easy for the team.
Thank you for the great work! @N-Wouda
N-Wouda
commented
1 year ago Dear @skadio, many thanks for your kind words, and your thorough and thoughtful review! You raise a lot of good and actionable points that I am going to discuss with my co-author @leonlan. I will respond more fully here later this week.
Again, thank you!
skadio
commented
1 year ago Glad to hear that you find it useful, and thank you both again for the library!
Inspired by your alpha-UCB policy in ALNS, let me add one potential integration opportunity for future. This comment is not related to publication at JOSS, so feel free to discard.
===
Similar to the UCB policy, other Bandit policies can be considered when selecting operators, for instance Thompson Samling.
In that regard, the MABWiser library that offers a comprehensive set of policies would neatly work as the Selection operator. Disclaimer: the library is built within our group in the AI Center at Fidelity.
The idea is to enable the "select" parameter to be MABWiser object which implements a variety of Bandit policies.
So the current UCB policy
select = AlphaUCB(scores=[5, 2, 1, 0.5], alpha=0.05), num_repair, num_destroy)becomes
select = AlphaUCB(scores=[5, 2, 1, 0.5], MABWiser(arms=num_repair * num_destroy, LearningPolicy.UCB1(alpha=0.05)Under the hood, ALNS leverages the expected reward and arm prediction computations from MABWiser.
In principle, the user can select any available bandit policy. In that sense, AlphaUCB becomes BanditALNS(scores, MABWiser)
Btw, it might be worthwhile considering Thompson Sampling (TS) Bandits which is a common counterpart to UCB policies, and often outperforms it.
It would look like this:
select = BanditALNS(scores=[1, 1, 1, 0], MABWiser(arms=num_repair * num_destroy, LearningPolicy.ThompsonSampling())The reward for TS is binary. Here I used 1 for best, better, and accepted solutions and 0 for rejected solutions. The 0/1 scoring schema for TS would be easier to tune than UCB where it is not clear how to pick reasonable defaults.
Finally, both UCB and LS are non-contextual policies. Moving beyond them, one could also consider contextual policies, where a set of features that describes that current state (neighborhood) help decide the arm selection. I am not aware of any research that considers contextual bandit policies for ALNS --this would be super cool! MABWiser supports all of these, and more.
N-Wouda
commented
1 year ago @editorialbot generate pdf
N-Wouda
commented
1 year ago @editorialbot check references
editorialbot
commented
1 year ago Reference check summary (note 'MISSING' DOIs are suggestions that need verification):
OK DOIs
- 10.1007/978-1-4419-1665-5_13 is OK
- 10.1007/s10732-018-9377-x is OK
- 10.1287/trsc.1050.0135 is OK
- 10.1016/j.cor.2022.105903 is OK
- 10.48550/arxiv.2211.00759 is OK
- 10.1016/j.cor.2022.106089 is OK
- 10.1006/jcph.1993.1010 is OK
- 10.1007/s12532-021-00209-7 is OK
- 10.1016/j.ejor.2021.05.042 is OK
- 10.1145/3319619.3326865 is OK
- 10.1016/j.advengsoft.2011.05.014 is OK
- 10.1145/3449726.3463276 is OK
- 10.21105/joss.04723 is OK
- 10.1007/s00500-011-0754-8 is OK
- 10.21105/joss.02448 is OK
- 10.21105/joss.00433 is OK
MISSING DOIs
- None
INVALID DOIs
- None:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
N-Wouda
commented
1 year ago @skadio best wishes for 2023! 🎉
I'll respond in three separate parts, addressing first your minimum requested changes, then your detailed comments, and finally some of the more tangentially related items.
The notebooks and documentations are great! Someone with an OR background should not have any problem using the library. That said, we need a bit more on the example usage for outsiders. With a quick low-hanging effort, the library can target a larger audience --which I believe is where largest potential is. To inspire the authors, I took a stab at this below. More on this in details below.
This could be very exciting! We have adopted your code template (here in the documentation, along with some notes), and link to it from our paper and the repository readme.
See the JOSS required item above to please add "State of the Field" to cover how this package compares to other packages. If none such exists (which I doubt it, since stochastic local search is well-known), please make the contribution explicit. If there are other tools, please cover mainstream ones, if any and mention where each would benefit the most (e.g., genetic algorithms for multi-point solutions vs. alns)
Thank you for pointing this out: we had initially missed this requirement. We now discuss several commonly used libraries in our “statement of need” section in our paper, and link to a recent review article for readers wishing to learn more.
See the JOSS required item above to please add Community Guidelines, if not already.
These were a bit buried in the documentation. We have several pages in the documentation detailing 1) how to contribute to the software, and 2) report issues/seek support. Links to these pages have been added to the repository readme, so they are easier to find for users. We have also set-up issue and pull request templates in the repository, to guide people opening issues or pull requests.
I believe, at a MINIMUM, the library requires the user to define:
- a STATE
- at least one DESTROY and one REPAIR operator, and
- an INITIAL STATE.
This is correct. We have made this more explicit in our paper (under “Features”), the repository readme, and in the documentation.
The pydoc in
State.pyclass reads "you are encouraged to subclass it". My understanding is, doing is MANDATORY not encourage. If so, please re-word. Same everywhere to be explicit about what is mandatory.
This is not required, but is something we used to do to satisfy the type checker. A better way to encode the actual constraint this is meant to satisfy---that a state object must have an objective() member function---is the use of the Protocol type. But that was only added to Python in version 3.8, and the alns package supports Python 3.7 until its end-of-life (in June 2023).
Luckily, none of this is problematic to the user. It will work with and without explicitly subclassing alns.State. To remove some of the confusion, we have removed this text from the State.py file, and also similarly rephrased the base classes for acceptance criteria, operator selection schemes, and stopping criteria.
In the Destroy() operator is it mandatory to work with the deep copy object? It seems so to me, but if correct, please be explicit about it since this is critical, no? Again, this is where the boiler plate comes quite handy. I explicitly mentioned this in the code TODO above
The destroy operator is given the current solution maintained by the ALNS algorithm. It is indeed critical that this solution itself is not modified, but instead a copy is changed/constructed. Since some solution states can be quite big, a deep copy might be very expensive – and there could be a problem-specific way to avoid having to deep copy a lot of data. As such, the alns package does not deep copy by default, but instead leaves this to the user.
We agree this was not phrased clearly. We have added a warning about this to the API documentation (here), and link to it from the quick start template. We have also kept the relevant TODO in the quick start code.
If I am following correctly, the library is built for minimization. Again, this is quite critical and needs to be explicitly stated. See my TODO comments about this and aim for something similar
This is correct. We have added a note about this to the quick start template in the documentation, and explain there how one could also do maximisation.
I saw some use of @DataClass annotation in some of the STATE definitions. Why is this needed/useful? and when? Documentation does not say much about its use. Please add some color
We mostly use the @dataclass decorator to avoid having to be explicit about the class’ data fields and the __init__ method. This is not really something specific to alns, but rather something built into Python since 3.7. In the example notebooks, we have added a few notes here and there to explain how this can be helpful in simplifying the state implementation a bit.
In the source code /examples folder, if you create a /data folder, then you can move all data files there. This would clean-up the structure and keep the notebooks (the most interesting bit) much more visible to library users.
This is a great idea! We have done exactly what you suggested, and this has really cleaned up the examples/ directory.
On that note, here is nice to have comment, and completely not necessary, that you are also probably aware of: there is A LOT of great code used in the documentation. At the same time, there is an opportunity to have them built-in within the library. I am thinking of sklearn and other ML library that comes ready with data loaders, plotting classes, that can be imported and used directly. If these data elements were moved into the library, then you could drop all that code from the documentation and hugely simplify the ALNS presentation with nice imports (say draw_tsp()), and the purpose is clear from the context. This would also hide a lot non-ALNS related imports/configs from sight as well. Again, this is not critical at all. Just mentioning, and I am sure you already considered.
Thank you for your suggestion! One of our selling points is that our alns package is problem-agnostic, and could in principle be used to solve any problem having an objective and some constraints. As such, we have opted not to already include problem-specific data loaders or related tools. If, however, a community of users working on related problems (e.g., TSP or VRP) should emerge, we are very willing to support such uses better ‘out of the box’, from an alns.tsp or alns.vrp submodule. Such support could include data loaders, plotting facilities, and standard solution state classes, along with some destroy and repair operators.
How you considered the integration of ALNS library with Optimization Modeling/Solving technology?
We had not considered this before, but indeed, solving general mathematical programming problems should also be possible. There has been some work on this in recent years (e.g., Hendel (2022)), but more can be done to simplify/improve the integration of heuristic methods for general optimization problems.
We think this is an interesting direction, and have copied your response to a new issue in our repository.
Inspired by your alpha-UCB policy in ALNS, let me add one potential integration opportunity for future. This comment is not related to publication at JOSS, so feel free to discard.
Thank you for pointing us to the MABWiser library. Using MABs to select operators is a very exciting direction for ALNS, and integrating with your group’s work at Fidelity seems like a great way to very quickly get a large range of MABs going. We have copied your comment to a new issue in our repository, and will look into this more in the (near) future!
hugoledoux
commented
1 year ago @editorialbot add @kenohori as reviewer
editorialbot
commented
1 year ago @kenohori added to the reviewers list!
kenohori
commented
1 year ago kenohori
commented
1 year ago Hello! First of all, congrats @N-Wouda on the nice library. Even as someone outside OR (but who implements a lot of optimisation problems), it was relatively easy to get it to create sample solutions to some simple problems. The paper, the code repository and the documentation are all easy to follow.
Mainly, I would like to second @skadio's comments on improving the help for non-OR users to bring their own problems into ALNS. This could make the library much more accessible to a wider audience. The code quickstart suggested by @skadio and the ALNS features page are good places to start, but I would have really appreciated an introduction to ALNS and some more general help (eg what problems it is good for, how to apply ALNS to your own problem, what is necessary to implement, how important different parts are).
Minor comments:
N-Wouda
commented
1 year ago Hello! First of all, congrats @N-Wouda on the nice library. Even as someone outside OR (but who implements a lot of optimisation problems), it was relatively easy to get it to create sample solutions to some simple problems. The paper, the code repository and the documentation are all easy to follow.
Hello @kenohori! 👋 We are glad you managed to get started so quickly, and are grateful for your positive comments!
Mainly, I would like to second @skadio's comments on improving the help for non-OR users to bring their own problems into ALNS. This could make the library much more accessible to a wider audience. The code quickstart suggested by @skadio and the ALNS features page are good places to start, but I would have really appreciated an introduction to ALNS and some more general help (eg what problems it is good for, how to apply ALNS to your own problem, what is necessary to implement, how important different parts are).
Thank you for this comment. I will write another page in the documentation explicitly introducing ALNS, explaining what type of problems it can be used to solve, and briefly describe what a minimal implementation should contain (in terms of ideas). This complements the quick start template, which is really about "so how do I turn those ideas into code". I expect I will have this finished later today, and will report back then.
- A list of dependencies would be nice, even if it can be handled automatically by pip. From what I gathered, they are only matplotlib and numpy?
Yes, it's just matplotlib and numpy. We have added a short description of these dependencies to the repository readme and the documentation.
- Maybe I missed it, but is there a way to test that the library is working correctly? Unit tests would be best, but a script that uses all the main components of the library would be good as well.
We have both! The tests are embedded in the alns/ submodules: each submodule has a tests/ directory that contains the (unit) tests related to that submodule. These are executed using poetry run pytest in the repository root: the contributing page in the documentation explains how to set up poetry to get this running locally. The tests are also run for every commit to a pull request or to the master branch, as the "Run pytest" step in the GitHub Actions workflow.
The test coverage is reported here: codecov.
Finally, the examples in the examples/ directory are executable notebooks that each use all the main alns components. These can be run locally as well, and how to do that is explained here.
N-Wouda
commented
1 year ago I expect I will have this finished later today, and will report back then.
@kenohori I had less time yesterday than I had hoped, but the new Introduction to ALNS page is finished, and available here in the documentation. The page explains the kind of problems ALNS can solve, and how ALNS works. We also link to a handbook chapter that provides further details, should a user be interested in exploring these methods further.
We have also updated the quick start page here a little to provide suggestions for a first destroy and repair operator pair.
kenohori
commented
1 year ago Great! Thanks, @N-Wouda. The page has all the basics and reads well.
After some numpy issues with Python 3.10, I managed to get the test suite working in Python 3.9 and everything looks good.
As far as I'm concerned, it's ready to be accepted 🎉 .
N-Wouda
commented
1 year ago @kenohori great to hear, and thank you for the review!
hugoledoux
commented
1 year ago @editorialbot generate pdf
hugoledoux
commented
1 year ago @editorialbot check references
editorialbot
commented
1 year ago Reference check summary (note 'MISSING' DOIs are suggestions that need verification):
OK DOIs
- 10.1007/978-1-4419-1665-5_13 is OK
- 10.1007/s10732-018-9377-x is OK
- 10.1287/trsc.1050.0135 is OK
- 10.1016/j.cor.2022.105903 is OK
- 10.48550/arxiv.2211.00759 is OK
- 10.1016/j.cor.2022.106089 is OK
- 10.1006/jcph.1993.1010 is OK
- 10.1007/s12532-021-00209-7 is OK
- 10.1016/j.ejor.2021.05.042 is OK
- 10.1145/3319619.3326865 is OK
- 10.1016/j.advengsoft.2011.05.014 is OK
- 10.1145/3449726.3463276 is OK
- 10.21105/joss.04723 is OK
- 10.1007/s00500-011-0754-8 is OK
- 10.21105/joss.02448 is OK
- 10.21105/joss.00433 is OK
MISSING DOIs
- None
INVALID DOIs
- None:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
hugoledoux
commented
1 year ago First, let me congratulate you on this nice submission @N-Wouda. It's not often that: reviewers are so enthusiastic, that the paper is clear and concise and typo-free (I couldn't find any faults, I just proof-read it), and that the authors implement so swiftly the comments (I like very much the intro to ALNS you added).
Second, I'll start the acceptance of the paper since @skadio stated that he had only the beginning of january to work on this, and I see you did respond and implemented all his suggestions (and also those of @kenohori).
hugoledoux
commented
1 year ago At this point could you:
I can then move forward with recommending acceptance of the submission.
N-Wouda
commented
1 year ago @hugoledoux thanks for you kind words! We are happy to hear this is ready for publication. In response to your checklist:
v5.0.4 of the package, which will also be uploaded to PyPI in a few minutes.skadio
commented
1 year ago @hugoledoux I confirm that revisions look good and address my minor points raised earlier --thank you @N-Wouda!
In particular, the Quick Start Template is super useful! Thanks for including this, and hopefully it serve all users (OR-savy or not) well in the future.
Not only a great submission for JOSS, and an important tool for the community, we also identified several follow-up research directions. Look forward seeing progress/collaboration on these in future.
Thank you all! Serdar
N-Wouda
commented
1 year ago @hugoledoux I have just updated the version and DOI above, because I had made a mistake in the package versioning :).
Look forward seeing progress/collaboration on these in future.
@skadio much the same for us, thank you for your comments and review! I have copied the follow-up research directions to new issues in the ALNS repository, and expect we will follow up on these later in 2023!
hugoledoux
commented
1 year ago @editorialbot set 10.5281/zenodo.7551649 as archive
editorialbot
commented
1 year ago Done! Archive is now 10.5281/zenodo.7551649
hugoledoux
commented
1 year ago @editorialbot set v5.0.4 as version
editorialbot
commented
1 year ago Done! version is now v5.0.4
hugoledoux
commented
1 year ago @editorialbot recommend-accept
editorialbot
commented
1 year ago Attempting dry run of processing paper acceptance...Reference check summary (note 'MISSING' DOIs are suggestions that need verification):
OK DOIs
- 10.1007/978-1-4419-1665-5_13 is OK
- 10.1007/s10732-018-9377-x is OK
- 10.1287/trsc.1050.0135 is OK
- 10.1016/j.cor.2022.105903 is OK
- 10.48550/arxiv.2211.00759 is OK
- 10.1016/j.cor.2022.106089 is OK
- 10.1006/jcph.1993.1010 is OK
- 10.1007/s12532-021-00209-7 is OK
- 10.1016/j.ejor.2021.05.042 is OK
- 10.1145/3319619.3326865 is OK
- 10.1016/j.advengsoft.2011.05.014 is OK
- 10.1145/3449726.3463276 is OK
- 10.21105/joss.04723 is OK
- 10.1007/s00500-011-0754-8 is OK
- 10.21105/joss.02448 is OK
- 10.21105/joss.00433 is OK
MISSING DOIs
- None
INVALID DOIs
- None:wave: @openjournals/csism-eics, this paper is ready to be accepted and published.
Check final proof :point_right::page_facing_up: Download article
If the paper PDF and the deposit XML files look good in https://github.com/openjournals/joss-papers/pull/3886, then you can now move forward with accepting the submission by compiling again with the command @editorialbot accept
hugoledoux
commented
1 year ago Thanks @skadio and @kenohori for the prompt reviews.
Congrats @N-Wouda and co-author!
(@N-Wouda you might have to change a few minor things after one EiC goes over everything)
skadio
commented
1 year ago Congrats @N-Wouda and team! 🥳🎉
kenohori
commented
1 year ago Congrats, @N-Wouda! It was a pleasure to review this.
danielskatz
commented
1 year ago @N-Wouda - As the track editor, I'll be proofreading this, likely later tonight, and seeing what else (if anything) is needed before we can publish it
danielskatz
commented
1 year ago @N-Wouda - I've only found a few minor issues related to cases in the bib, as proposed changes in https://github.com/N-Wouda/ALNS/pull/138
N-Wouda
commented
1 year ago @editorialbot generate pdf
Submitting author: !--author-handle-->@N-Wouda<!--end-author-handle-- (Niels Wouda) Repository: https://github.com/N-Wouda/ALNS Branch with paper.md (empty if default branch): joss-paper Version: v5.0.4 Editor: !--editor-->@hugoledoux<!--end-editor-- Reviewers: @skadio, @kenohori Archive: 10.5281/zenodo.7551649
Status
Status badge code:
Reviewers and authors:
Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)
Reviewer instructions & questions
@skadio, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:
The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @hugoledoux know.
✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨
Checklists
📝 Checklist for @skadio
📝 Checklist for @kenohori