Procgen wrapper

[Bam4d]

Procgen also has it's own inbuilt parallelization which we can make use of. I'm not entirely sure how it works off the top of my head in terms of threads and CPU affinity. We should be able to either replace ParallelEnvs or EnvList and use the env parameter which I think is "num_envs".

[cswinter]

Braindump:

• current integration is quite inefficient, i'm getting ~4K SPS but majority of time is spent on env, probably we could get at least 10K-20K
  • one gym3 vectorized env is created for every environment, instead we should directly implement an entity-gym VecEnv and push all parallelization into procgen
  • deserializer could be a lot faster, it should never materialized any dataclasses, write directly into numpy arrays, skip all fields which we aren't using, skip tilemap unless it's used, deserializes multiple contiguous fields of the same type in one go
• at least three more environments which have only entities would be very easy to add support for: starfighter, boss fight, leaper, maybe also plunder
  • entities are identical across all environments
  • environments write out a little bit of additional info at the end which is specific to the environment (e.g. BigFish::serialize) which may contain important additional features
• other environments include a tilemap as well which doesn't map nicely to anything in entity-gym currently
  • running python deserialize.py will print out the tilemap
  • some of these could probably be converted into entities, e.g. coinrun tilemap below (truncated, full tilemap is padded to 64x64)
  • G is goal, W is wall, L is lava, "B" is invisible barrier used by enemy AI, when enemy collides with barrier it reverses direction.
  • could just have one entity for each horizontal position that has a feature for the height of the platform

    WW              WWWWWWWWWWWWWWWWWWWWWW
    WW              WWWWWWWWWWWWWWWWWWWWWW
    WW       B     GWWWWWWWWWWWWWWWWWWWWWW
    WW        WWWWWWWWWWWWWWWWWWWWWWWWWWWW
    WW   B    WWWWWWWWWWWWWWWWWWWWWWWWWWWW
    WW    WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
    WW    WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
    WW    WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
    xWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW

[cswinter]

Getting good performance turned out to be a bit harder than I thought, the get_state method turns out to be a bottleneck. I've eliminated some inefficiencies in the parsing, which gives a nice speedup, but still far from what Procgen is capable of. There's not further easy gains though without optimizing/parallelizing the get_state method and Procgen serialization logic. Since get_state isn't multithreaded afaik, using the in-built vectorized environment and parallelization doesn't really help, and so sticking with the current design where we have one single-env vectorized env for every environment and do parallelization on our side is both simpler and more performant.

[cswinter]

Baselines: https://wandb.ai/entity-neural-network/enn-ppo/reports/Procgen-Baselines--VmlldzoxNzUxNDcy