search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
696 stars 36 forks source link

[REVIEW]: Benchmarking Hierarchical Reasoning with HierarchyCraft #6468

Open editorialbot opened 4 months ago

editorialbot commented 4 months ago

Submitting author: !--author-handle-->@MathisFederico<!--end-author-handle-- (Mathis Federico) Repository: https://github.com/IRLL/HierarchyCraft Branch with paper.md (empty if default branch): Version: v1.2.4 Editor: !--editor-->@logological<!--end-editor-- Reviewers: @lwu9, @Christopher-Henry-UM Archive: Pending

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/8cb63e5284d648614ab711bb52099988"><img src="https://joss.theoj.org/papers/8cb63e5284d648614ab711bb52099988/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/8cb63e5284d648614ab711bb52099988/status.svg)](https://joss.theoj.org/papers/8cb63e5284d648614ab711bb52099988)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@lwu9 & @Christopher-Henry-UM, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @logological know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @Christopher-Henry-UM

📝 Checklist for @lwu9

editorialbot commented 4 months ago

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf
editorialbot commented 4 months ago

Software report:

github.com/AlDanial/cloc v 1.90  T=0.12 s (1032.4 files/s, 205378.3 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          79           2128           1576           8445
SVG                              5              5              5           7817
HTML                            14            686             28           1470
Jinja Template                   5             27             39            527
CSS                              4            108             48            523
Markdown                         4            133              0            268
TeX                              1             22              0            225
YAML                             7             21             12            209
TOML                             1             19              7             89
JavaScript                       1              4              6             20
PowerShell                       2              0              0              2
-------------------------------------------------------------------------------
SUM:                           123           3153           1721          19595
-------------------------------------------------------------------------------

Commit count by author:

   856  Mathïs Fédérico
   134  MathisFederico
     3  SWang848
     2  metaylor
editorialbot commented 4 months ago

Paper file info:

📄 Wordcount for paper.md is 1449

✅ The paper includes a Statement of need section

editorialbot commented 4 months ago

License info:

🟡 License found: GNU General Public License v3.0 (Check here for OSI approval)

editorialbot commented 4 months ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- None

MISSING DOIs

- No DOI given, and none found for title: The Malmo Platform for Artificial Intelligence Exp...
- No DOI given, and none found for title: Unified Planning Framework
- No DOI given, and none found for title: Benchmarking the Spectrum of Agent Capabilities
- 10.1016/s0004-3702(99)00052-1 may be a valid DOI for title: Between MDPs and semi-MDPs: A framework for tempor...
- No DOI given, and none found for title: Minimalistic Gridworld Environment for Gymnasium
- 10.1613/jair.5699 may be a valid DOI for title: Revisiting the arcade learning environment: Evalua...
- No DOI given, and none found for title: The NetHack learning environment
- No DOI given, and none found for title: Retrospective Analysis of the 2019 MineRL Competit...
- No DOI given, and none found for title: The MineRL 2020 Competition on Sample Efficient Re...
- No DOI given, and none found for title: Mastering Diverse Domains through World Models
- No DOI given, and none found for title: Leveraging procedural generation to benchmark rein...
- 10.1613/jair.3912 may be a valid DOI for title: The arcade learning environment: An evaluation pla...
- No DOI given, and none found for title: DeepMind Lab
- No DOI given, and none found for title: OpenAI Gym
- No DOI given, and none found for title: PDDLGym: Gym Environments from PDDL Problems
- No DOI given, and none found for title: PDDL - The Planning Domain Definition Language
- No DOI given, and none found for title: Interval-based relaxation for general numeric plan...
- No DOI given, and none found for title: Stable-Baselines3: Reliable Reinforcement Learning...
- No DOI given, and none found for title: The ANML Language
- No DOI given, and none found for title: Insights From the NeurIPS 2021 NetHack Challenge

INVALID DOIs

- None
editorialbot commented 4 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

logological commented 3 months ago

@lwu9 and @Christopher-Henry-UM, thanks for agreeing to review this submission. In case you didn't receive the notification from editorialbot, please visit this issue on GitHub and review the Reviewer Instructions at the top of the thread. Let me know if you have any questions about the review process.

Christopher-Henry-UM commented 3 months ago

Thanks for the reminder Tristan! I will get to the review this week.

lwu9 commented 3 months ago

Thanks, Tristan!

On Mon, 18 Mar 2024 at 19:36, Christopher-Henry-UM @.***> wrote:

Thanks for the reminder Tristan! I will get to the review this week.

— Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/6468#issuecomment-2005299407, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG4SWUJB5OYO22SM6DDXE33YY527XAVCNFSM6AAAAABERATZ6CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBVGI4TSNBQG4 . You are receiving this because you were mentioned.Message ID: @.***>

logological commented 3 months ago

I just heard from Chris Henry that he won't be able to start his review for a few more days yet. But don't let that stop you, @lwu9, from starting your review once you're ready.

logological commented 3 months ago

Hey there @lwu9 and @Christopher-Henry-UM! I just wanted to ping you to ask whether you'll be able to start your reviews of this submission soon.

lwu9 commented 3 months ago

Thanks for the reminder. I plan to work on it this weekend.

Best, Lili

On Mon, Apr 8, 2024 at 6:55 PM Tristan Miller @.***> wrote:

Hey there @lwu9 https://github.com/lwu9 and @Christopher-Henry-UM https://github.com/Christopher-Henry-UM! I just wanted to ping you to ask whether you'll be able to start your reviews of this submission soon.

— Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/6468#issuecomment-2043769811, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG4SWUJ3JV5QNQ6FP3CPVMLY4MN57AVCNFSM6AAAAABERATZ6CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBTG43DSOBRGE . You are receiving this because you were mentioned.Message ID: @.***>

Christopher-Henry-UM commented 3 months ago

Thank you for bearing with me. I will get to it this week.

Christopher-Henry-UM commented 3 months ago

Review checklist for @Christopher-Henry-UM

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

lwu9 commented 3 months ago

Review checklist for @lwu9

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

lwu9 commented 3 months ago
  1. There are two authors in the paper but there are three contributors in GitHub.
  2. I was able to install the package, but when I ran "action = get_human_action(env)", it got stuck.
  3. I didn't find clear community guidelines.
logological commented 2 months ago

@lwu9 Thanks for the review so far. Is your second issue blocking you from performing other steps in the review? @MathisFederico, perhaps you could comment on the three issues?

Christopher-Henry-UM commented 2 months ago

I have performed most of the review except I am not comfortable checking the Functionality or the Performance boxes. The software is used to create arbitrary hierarchical environments that are compatible with the OpenAI Gym Reinforcement Learning framework or the AIPlan4EU Unified Planning Framework. This library enables users to easily create complex hierarchical structures that can be used to test and develop various reinforcement learning or planning algorithms.

While I am familiar with the concept of reinforcement learning, I do not have the experience with the type of programming this library was created for. As a result, I have no way to test it other than to carry out the Quickstart steps listed in the documentation.

logological commented 2 months ago

Thanks, @Christopher-Henry-UM. @MathisFederico, are there any further reviewers you could nominate with the requisite knowledge of OpenAI Gym Reinforcement Learning or AIPlan4EU Unified Planning Framework to evaluate the functional claims of your submission? Also, could you please respond to my query upthread concerning the issues raised by @lwu9?

MathisFederico commented 2 months ago

Thanks for the reviews so far !

Sorry @logological for being long to answer, it has been two very busy weeks...

About @lwu9 issues:

  1. There are two authors in the paper but there are three contributors in GitHub.

I was not sure if I could add @SWang848 and @liyuxuan-academic because most of their contributions where non-code ones (testing, reviewing documentation, discussing ideas, ...) but re-reading JOSS authorship guidelines it seems that I could indeed ! I asked them if they want to be added as author (and not just being in acknowledgments as now) and will update according to their answers.

2. I was able to install the package, but when I ran "action = get_human_action(env)", it got stuck.

When using human actions, the program will pause and wait for the action of the human, meaning you ! A pygame window should pop (be sure to have gui optional requirements as indicated in the readme):

Initial pygame window for the readme example)

You can select action be either using arrows and pressing enter, or by just clicking on available actions. Selecting wood block would result in this:

image

3. I didn't find clear community guidelines.

There was a CONTRIBUTING.md file but the readme didn't mention it indeed ! It is now added !

About @Christopher-Henry-UM issues:

There are tests that checks those facts automatically both for planning (see planning tests) and for gym API (see gym tests) also for the gym compatibility this library has been heavily used with stable-baselines3 that requires the gym API to be valid.

I still asked around in the planning community to know if someone would like to review, I will update here if I find someone !

MathisFederico commented 2 months ago

After their approval, I've added both non-code co-authors @logological

Need any help regarding issue 2 @lwu9 ?

logological commented 2 months ago

Thanks for this, @MathisFederico. Have you made any progress in finding prospective reviewers? Even with automated tests, I think it would be better if the functionality claims were verified by a reviewer who has experience with the relevant APIs. I don't personally know of anyone with this experience but have been on the lookout for someone who does.

MathisFederico commented 2 months ago

I agree it would be better but its hard to convince people to spend a bit of time on this, I'm still on the lookout too

logological commented 1 month ago

@lwu9 It's been a few weeks since we've heard from you. Could you please review the MathisFrederico's responses to your issues? If everything is now to your satisfaction, you could check off the remaining items in your review; otherwise, please let us know what matters remain unresolved.

logological commented 1 month ago

I agree it would be better but its hard to convince people to spend a bit of time on this, I'm still on the lookout too

I've contacted a few more potential reviewers and will of course keep everyone apprised.

lwu9 commented 1 month ago

Sounds good, thanks for the heads up. I will finish the check list today.

On Sun, May 26, 2024 at 11:43 PM Tristan Miller @.***> wrote:

I agree it would be better but its hard to convince people to spend a bit of time on this, I'm still on the lookout too

I've contacted a few more potential reviewers and will of course keep everyone apprised.

— Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/6468#issuecomment-2132586493, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG4SWUJQCPBSRALNS7OZDRLZEKTWBAVCNFSM6AAAAABERATZ6CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZSGU4DMNBZGM . You are receiving this because you were mentioned.Message ID: @.***>

lwu9 commented 1 month ago

My previous questions have been resolved, thanks for the clarification. When I ran the snippets in Readme, I ran into two more issues.

  1. 'from hcraft.examples import TowerHcraft, RecursiveHcraft, RandomHcraft' It gave me import errors, but if I changed it to 'from hcraft.examples import TowerHcraftEnv, RecursiveHcraftEnv, RandomHcraftEnv', it worked.

  2. 'problem.solve()' in the last section It gave me 'FileNotFoundError' in [\lib\site-packages\unified_planning\engines\pddl_planner.py:194], in PDDLPlanner._solve(self, problem, heuristic, timeout, output_stream)

logological commented 1 month ago

@yg211 Hope all is well with you! Would you be willing and able to review this submission for the Journal of Open Source Software? We're particularly looking for a reviewer who has familiarity with the OpenAI Gym or AIPlan4EU platforms.

lwu9 commented 1 month ago

I posted some comments on the github last week, but they didn't show up in this email thread, so I just copied them here just in case:

My previous questions have been resolved, thanks for the clarification, Mathïs! When I ran the snippets in Readme, I ran into two more issues.

1.

'from hcraft.examples import TowerHcraft, RecursiveHcraft, RandomHcraft' It gave me import errors, but if I changed it to 'from hcraft.examples import TowerHcraftEnv, RecursiveHcraftEnv, RandomHcraftEnv', it worked. 2.

'problem.solve()' in the last section It gave me 'FileNotFoundError' in [\lib\site-packages\unified_planning\engines\pddl_planner.py:194], in PDDLPlanner._solve(self, problem, heuristic, timeout, output_stream)

On Tue, 4 Jun 2024 at 15:29, Tristan Miller @.***> wrote:

@yg211 https://github.com/yg211 Hope all is well with you! Would you be willing and able to review this submission for the Journal of Open Source Software? We're particularly looking for a reviewer who has familiarity with the OpenAI Gym or AIPlan4EU platforms.

— Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/6468#issuecomment-2148250411, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG4SWUIAPKI47QNRJJXTRX3ZFYISXAVCNFSM6AAAAABERATZ6CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBYGI2TANBRGE . You are receiving this because you were mentioned.Message ID: @.***>

MathisFederico commented 3 weeks ago 
1. 'from hcraft.examples import TowerHcraft, RecursiveHcraft, RandomHcraft'
   It gave me import errors, but if I changed it to 'from hcraft.examples import TowerHcraftEnv, RecursiveHcraftEnv, RandomHcraftEnv', it worked.

Indeed thanks for spotting it, fixed now.

2. 'problem.solve()' in the last section
   It gave me 'FileNotFoundError' in [\lib\site-packages\unified_planning\engines\pddl_planner.py:194], in PDDLPlanner._solve(self, problem, heuristic, timeout, output_stream)

I can't seem to reproduce that error, may have more informations about your system and the pip list result for package versions ?

logological commented 3 weeks ago

I haven't checked in for a while so I just wanted to report that I'm continuing to contact potential reviewers by e-mail.

logological commented 1 week ago

@MathisFederico, I apologize for the long delay in finding an additional reviewer for your manuscript. I have been continuing to reach out to candidates by e-mail but have not yet had any positive responses. In the meantime, if you or your colleagues could suggest any potential reviewers for me to approach, please let me know.

logological commented 1 week ago

@maltehelmert @roeger Would you be willing and able to review this submission for the Journal of Open Source Software? We're particularly looking for a reviewer who has familiarity with the OpenAI Gym or AIPlan4EU platforms.

maltehelmert commented 1 week ago

@maltehelmert @roeger Would you be willing and able to review this submission for the Journal of Open Source Software? We're particularly looking for a reviewer who has familiarity with the OpenAI Gym or AIPlan4EU platforms.

I'm sorry I'm currently overcommitted. :-( I think it's the same for @roeger. But perhaps you can recommend someone from the AIPlan4EU contributors, Gabi?

logological commented 15 hours ago

@alvalentini @arbimo @mikand Would you be willing and able to review this submission for the Journal of Open Source Software? We're particularly looking for a reviewer who has familiarity with the OpenAI Gym or AIPlan4EU platforms.