[PRE REVIEW]: Generic reinforcement learning codebase in TensorFlow

[whedon]

Submitting author: @alexanderimanicowenrivers (Alexander I. Cowen-Rivers) Repository: https://github.com/for-ai/rl Version: v2.0 Editor: @mbobra Reviewers: @desilinguist, @paragkulkarni11

Author instructions

Thanks for submitting your paper to JOSS @alexanderimanicowenrivers. Currently, there isn't an JOSS editor assigned to your paper.

@alexanderimanicowenrivers if you have any suggestions for potential reviewers then please mention them here in this thread. In addition, this list of people have already agreed to review for JOSS and may be suitable for this submission.

Editor instructions

The JOSS submission bot @whedon is here to help you find and assign reviewers and start the main review. To find out what @whedon can do for you type:

@whedon commands

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@whedon commands

What happens now?

This submission is currently in a pre-review state which means we are waiting for an editor to be assigned and for them to find some reviewers for your submission. This may take anything between a few hours to a couple of weeks. Thanks for your patience :smile_cat:

You can help the editor by looking at this list of potential reviewers to identify individuals who might be able to review your submission (please start at the bottom of the list). Also, feel free to suggest individuals who are not on this list by mentioning their GitHub handles here.

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

Failed to discover a valid open source license.

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[arfon]

:wave: @mbobra - would you be willing to edit this submission for JOSS?

[arfon]

Failed to discover a valid open source license.

Not sure why @whedon isn't happy - GitHub seems to think this is MIT-licensed (https://github.com/for-ai/rl/blob/master/LICENSE).

[mbobra]

@arfon Sure, I'm happy to edit!

[arfon]

@whedon assign @mbobra as editor

[whedon]

OK, the editor is @mbobra

[arfon]

@arfon Sure, I'm happy to edit!

Great, thanks @mbobra!

[alexanderimanicowenrivers]

Thanks @mbobra

Out of interest, do you know what the average length of a review process is roughy 😄 ?

[mbobra]

@alexanderimanicowenrivers I'm sorry for the delay -- I'm still working on finding reviewers for this submission. I don't know the average review time for a JOSS submission, but it can be short (less than one week) once the handling editor finds reviewers. I've messaged a ton of potential reviewers and it looks like a busy time -- but I'll get there. Thank you!

[mbobra]

@alexanderimanicowenrivers Can you please suggest reviewers from this list -- or from elsewhere?

[alexanderimanicowenrivers]

Hey, will anyone else review our Deep RL codebase, written in Python, specifically TF :) ? @desilinguist , @rougier, @sealhuang, @ahurriyetoglu, @betatim, @jkahn, @nirum, @arokem, @deniederhut, @effigies, @stsievert, @oesteban.

Much appreciated!

[mbobra]

@alexanderimanicowenrivers Thank you for the suggestions. To those mentioned above: Are you available and willing to review this submission?

[effigies]

Sorry, no TF experience.

[desilinguist]

I am familiar with TF but not very familiar with reinforcement learning. If you don’t find anyone else better suited, I’ll take a look.

On Thu, Jun 20, 2019 at 5:56 AM Chris Markiewicz notifications@github.com wrote:

Sorry, no TF experience.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/1502?email_source=notifications&email_token=AABHPEALLM6G3SR62XWRTQDP3NH3NA5CNFSM4HX2CRI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYE5VCA#issuecomment-503962248, or mute the thread https://github.com/notifications/unsubscribe-auth/AABHPEF23W3J5T2NHW7ZN3LP3NH3NANCNFSM4HX2CRIQ .

-- Linguist, Desi Linguist http://www.desilinguist.org

[rougier]

I'm a bit busy right now (and I'm just the opposite of @desilinguist, familiar with RL but no TL). One of our PhD student here (@cervere) might be able to review if needed.

[mbobra]

@desilinguist There is no need to be an expert -- according to the JOSS docs, a reviewer only needs "sufficient expertise" in the field of submission. The reviewer checklist covers general software, documentation, and functionality (i.e. confirming the functional claims of the software). I think you'd be more than qualified to do this job. Do you mind reviewing this submission?

@cervere Do you have time to review this submission?

[cervere]

@mbobra Sorry, cannot make time at this moment. (Had a quick run through, so if I am allowed make some unsolicited remarks : I would have liked to spend some time on certain aspects - particularly because the work claims 'generic' - for e.g, envs outside OpenAI? more complex agents with multiple objectives? etc.. And as observed in other submissions, I hope the authors have addressed the important question well : What is the claim? 'Generic' nature from RL pov or from a utility for tensorflow pov..)
Cheers

[alexanderimanicowenrivers]

Sorry, no TF experience.

Hey @effigies , I don't think TF experience is necessary, the most important aspects are the way its structured and the algorithms we provide, thus if you know RL that should me more to suffice a review :)

[alexanderimanicowenrivers]

@cervere Sorry, cannot make time at this moment. (Had a quick run through, so if I am allowed make some unsolicited remarks : I would have liked to spend some time on certain aspects - particularly because the work claims 'generic' - for e.g, envs outside OpenAI? more complex agents with multiple objectives? etc.. And as observed in other submissions, I hope the authors have addressed the important question well : What is the claim? 'Generic' nature from RL pov or from a utility for tensorflow pov..) Cheers

So generic in both terms, 1 we provide integration to external gyms (CoinRun, Atari and OpenAI GYM) and its generic in terms of a RL point of view, abstracting out the relevant methods allowing re-use across both model-free and model-based algorithms.

It was also generic in the sense that when we could, we wrote non differentiable functions in numpy (e.g memory), which allowed me recently to port the whole codebase over to PyTorch with minimal effort (just changing the models, train.py and algos folder).

[jkahn]

currently without time to contribute here. Wish I could; I'm excited to see the sort of work described by the title.

On Wed, Jun 19, 2019 at 9:50 PM Alexander Cowen-Rivers < notifications@github.com> wrote:

@desilinguist https://github.com/desilinguist , @rougier https://github.com/rougier, @sealhuang https://github.com/sealhuang, @ahurriyetoglu https://github.com/ahurriyetoglu, @betatim https://github.com/betatim, @jkahn https://github.com/jkahn, @nirum https://github.com/nirum, @arokem https://github.com/arokem

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/1502?email_source=notifications&email_token=AAARW7HFDOQWKE2POFENLTLP3MD7TA5CNFSM4HX2CRI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYED2AQ#issuecomment-503856386, or mute the thread https://github.com/notifications/unsubscribe-auth/AAARW7CFQ5IXOEVCOSIBS7LP3MD7TANCNFSM4HX2CRIQ .

[alexanderimanicowenrivers]

@mbobra can we assign the review to @desilinguist please?

https://github.com/openjournals/joss-reviews/issues/1502#issuecomment-503985440

[mbobra]

@whedon assign @desilinguist as reviewer

[whedon]

OK, the reviewer is @desilinguist

[mbobra]

@whedon add @paragkulkarni11 as reviewer

[whedon]

OK, @paragkulkarni11 is now a reviewer

[mbobra]

@whedon start review magic-word=bananas

[whedon]

OK, I've started the review over in https://github.com/openjournals/joss-reviews/issues/1524. Feel free to close this issue now!