Support environments on rlberry?

[mmcenta]

Currently, rlberry supports some environments, but we would like to add other environments developed by the SCOOL group to the repository. This raises the question of whether we should have environments as part of rlberry or not. An alternative would be to create another repository (like an RL Baselines Zoo for rlberry) in which we would put code for the environments (and possibly other things such as default hyperparameters).

I think it would be cleaner to separate the two repositories so that we can keep rlberry concise. In that case, we would still need to:

1. Decide what things will go into this new repository (trained agents, default hyperparameters, etc..)
2. Give the new repository a name
3. Create issues to remove what isn't needed here

[omardrwch]

Since one of the principles of the library is to help people design their agents, and environments are a key part of algorithmic design, I think it is important to keep environments in rlberry, but not all environments.

For instance, grid worlds (continuous or discrete) are very useful to debug agents, and I think that environments for debugging/analysis of agents should not only be kept, but developed.

However, I think that application-specific environments (as some that are developed by the team to simulate crops, for instance) should be kept in another repository, because their purpose is to solve a specific applied problem, rather than helping people to debug/develop general agents.

To summarize:

• I suggest to keep and develop simple environments for debugging;
• and to put application-specific environments (and that are more complex) in another repository.

Some other points that are useful in rlberry environments:

• rlberry provides tools for users to easily render their own 2D environments (rlberry.rendering, which we can extend to 3D), and visualization is very useful for debugging;
• rlberry provides an interface for environments that represents a generative model, that is, with a method sample(state, action) to collect a transition from any (state, action) pair; in addition to a step(action) that only takes an action. Some RL algorithms require a generative model, and might be implemented in rlberry, so we need to keep this environment interface in the library anyway.

[mmcenta]

You're right, I hadn't thought about the use case of debugging agents. Currently, we have:

• benchmarks: these seem to be toy problems for exploration and generalization, which I think is fine to keep;
• bullet3: these are implementations of the PyBullet pendulum environments with some additional features. I would prefer to wrap the PyBullet environment instead, because that way we don't need to keep up our version up to date with PyBullet's version.
• classic_control: these are implementations of the Gym classic control environments that work with rlberry.redering. I think these are nice and maybe it's fine that they are reimplemented since they are not so complex.
• finite: simple finite environments for debugging. I think these should be kept since their implementations are pretty straightforward and they are useful.

Should all these environments be kept? Also, I think it would be better to avoid reimplementing environments from other libraries unless they're simple - the less complex code we need to maintain, the better.