CartPole notebooks and refactoring

[ntoxeg]

This adds the notebook for CartPole learning from the WIP PR that I will close, so that the intermediate results are available to all.

Additionally, a I did a bunch of refactoring:

• agents and env wrappers take an Atomspace now so that there is no ambiguity as to what owns which Atomspace (it has to always be one created beforehand).
• Env wrappers return (observation, reward, done) to match the Gym convention.
• Agents just return done from step instead of its negation for some reason.
• Training loops fixed to restart environments each epoch of training.
• Rendering is not hardcoded anymore into step of wrappers. (just invoke render in the training loop)
  • A nice outcome from this is that we can run environments like CartPole in CI — I added a unit test that runs CartPole.
• More info in README.md
• More type annotations and some refactoring of variables / methods.
• The notebooks now use TensorBoard to store information about received rewards — you can use it to see them as a nice chart now.
• I added some additional logging of my own because it works better for me than using OpenCog’s loggers but I can remove them if that’s annoying.

[ntoxeg]

@ngeiswei 🔔

[ntoxeg]

Rebuilding my Docker images as PLN now needs spacetime and I had to adjust the build order for that.

[ngeiswei]

Thanks @ntoxeg

[ngeiswei]

@ntoxeg, I've got the feeling that OpencogAgent.log_level isn't used anywhere. If so, unless you intend to have it used somewhere, I'd like to remove it. Let me know.

[ngeiswei]

I also don't understand why you construct the atomspace outside of OpencogAgent construction. For instance I want to introduce multiple atomspaces, one containing percepta, another one containing cognitive schematics and a third one for working memory, it makes little sense to construct them outside of OpencogAgent, especially given that the user can always access them via OpencogAgent attributes.

So unless you can rationalize that decision I'm gonna move atomspace construction back inside OpencogAgent.

[ngeiswei]

So unless you can rationalize that decision

OK, I get it, it's because the action space requires an atomspace, isn't it? Otherwise the example agent constructs an atomspace that is different than that constructed inside OpencogAgent.

I need to think more about it now.

[ngeiswei]

My current decision is to keep the atomspace constructed outside of OpencogAgent as working atomspace, and construct percepta and cognitive schematics atomspaces inside OpencogAgent. It requires a minimal amount of change and will not alter the current API. We may decide to alter the API later on, but it does not have to be today.

[ntoxeg]

So unless you can rationalize that decision

OK, I get it, it's because the action space requires an atomspace, isn't it? Otherwise the example agent constructs an atomspace that is different than that constructed inside OpencogAgent.

I need to think more about it now.

Yes, my primary intention there was to deal somehow with potentially ambiguous AtomSpace allocation and ownership.

I guess one simple change could be to not have an AtomSpace for the action space, as it doesn’t seem to be really needed, of course that means that the agent itself has to add necessary info to its own AtomSpace.