[Feature discussion] Improving model save format

[Miffyli]

Continuation on the discussion in #312 related to improving save file format.

Issue with current way of storing/loading models

• Cloudpickle can have issues with different version of Python (mentioned here)
• Cloudpickle itself mentions that it shouldn't be used for long-term storage (something RL people might want).
• Since everything is stored into one serialized file, a single corrupted part prevents easily loading rest of the things. Example: My saved model had Tensorflow code that did not work on different Python version, thus resulting error upon loading. I only needed the model parameters, but I couldn't load them because one of the class items broke deserialization.
• A minor point, but cloudpickle is Python-specific, so accessing anything inside these files requires going through Python+ cloudpickle.

Suggestion Using Python-native Zipfile to manually construct a zipped archive. Each entry in the zip file would be a separately serialized object. E.g. A2C would have one entry for "gamma", another for "n_steps" and so on. Finally the zip file would include one more entry for the model parameters, serialized with numpy's savez or just with pickle as is done now.

Since many learning algorithms store policies, obs/action spaces, functions and other custom Python objects, it might make sense to store these using pickle. However now if these become non-deserializable (e.g. new Python version), we can still load other items. This works especially well with load_parameters, which could directly access the parameters in zip file rather than trying to deserialize the whole file.

An alternative would be to use JSON to serialize class parameters, but this may not play well with custom functions and the likes. JSON would offer very human-readable and long-lasting files, though. I also took a look at jsonpickle, but I am not sure if relying on another library is a better idea than sticking with pickle for serializing things.

[araffin]

Hello,

I will be on holidays soon, so I'll let @AdamGleave @erniejunior @hill-a discuss this issue (if they have time), otherwise, I'll try to come by to you at the end of august.

[ernestum]

I am a big fan of storing the class parameters in json files. Its easy to parse from other context and will last long enough for all practical purposes. Not so big fan of the one file per parameter within a zip approach. It seems a bit convoluted to me.

[araffin]

I'm back ;) after encountering some issues due to different package version, i'm definitely for that feature.

I would favor yaml over json (easier to write and it can handle some python objects (cf rl zoo)) and maybe use zip format to have only one file ar the end? (containing npy arrays and serialized params)

The main issue i see is ensuring that we can serialize all python objects.

Last thing: i would keep the cloudpickle format as an option for backward compat at first.

[Miffyli]

@araffin Welcome back, hopefully all refreshed! ^^

JSON vs YAML: YAML indeed is more human-readable, but I would go with JSON as it seems more commonly supported (and still seems to require 3rd party libraries for Python). With JSON files, somebody could parse them in other languages/scripts easier. We can use the indent argument of json.dump to make it prettier for human consumption.

Zipping: I agree with @erniejunior "one file per parameter" approach is quite convoluted. How about .zip archive with one JSON file and one file with the parameters (e.g. numpy file)?

Unserializable objects: I figure still will rely on cloudpickle/pickle. How about serializing them into bytes/base64 and storing in the JSON file much like other parameters? This would be tested with json.dump to see which objects are serializable, and if not then use cloudpickle. This will result in messier JSON file, but still readable by other languages.

Backward compatibility: Yes, this would definitely be implemented for loading and as optional setting for saving (just in case).

[araffin]

still seems to require 3rd party libraries for Python

Ok, good point.

We can use the indent argument of json.dump to make it prettier for human consumption.

yes, good idea.

How about .zip archive with one JSON file and one file with the parameters (e.g. numpy file)?

Yes, that was my idea + maybe pickle file for what cannot be properly put into one or the other

How about serializing them into bytes/base64 and storing in the JSON file much like other parameters?

This would prevent from using/loading it from another language no? Btw I think the first step would be to reference/identify what cannot be put into a JSON/numpy file and figure out how to change that, no?

[Miffyli]

If we encode the bytes of non-JSON-able objects into a string and save it in the JSON, then other languages can still read the JSON file all nice but can't really do much with these byte-strings (they do not have any meaning outside Python/Cloudpickle/Pickle).

Did a quick run of what can be JSONiable and what needs pickling. As expected, spaces and policy are the main issue as they contain classes and functions. However, in my opinion, these can stay byte-serialized as they only have proper meaning in Python. These are also the parts that will likely break something when changing Python or library versions.

One thing we can do with this much like Keras does: When loading model, we can replace some of the items from a file with something else. E.g. A2C.load(..., custom_objects={'policy': other_policy}) would skip reading policy from the file and instead uses the provided one other_policy. This is already in load function but I would move it bit deeper so that it could prevent loading invalid objects (and thus crashing).

``` gamma JSON n_steps JSON vf_coef JSON ent_coef JSON max_grad_norm JSON learning_rate JSON alpha JSON epsilon JSON lr_schedule JSON verbose JSON policy Cloudpickle observation_space Cloudpickle action_space Cloudpickle n_envs JSON _vectorize_action JSON policy_kwargs JSON gamma JSON n_steps JSON q_coef JSON ent_coef JSON max_grad_norm JSON learning_rate JSON lr_schedule JSON rprop_alpha JSON rprop_epsilon JSON replay_ratio JSON replay_start JSON verbose JSON policy Cloudpickle observation_space Cloudpickle action_space Cloudpickle n_envs JSON _vectorize_action JSON policy_kwargs JSON gamma JSON nprocs JSON n_steps JSON vf_coef JSON ent_coef JSON vf_fisher_coef JSON max_grad_norm JSON learning_rate JSON kfac_clip JSON lr_schedule JSON verbose JSON policy Cloudpickle observation_space Cloudpickle action_space Cloudpickle n_envs JSON _vectorize_action JSON policy_kwargs JSON checkpoint_path JSON param_noise JSON learning_starts JSON train_freq JSON prioritized_replay JSON prioritized_replay_eps JSON batch_size JSON target_network_update_freq JSON checkpoint_freq JSON prioritized_replay_alpha JSON prioritized_replay_beta0 JSON prioritized_replay_beta_iters JSON exploration_final_eps JSON exploration_fraction JSON learning_rate JSON gamma JSON verbose JSON observation_space Cloudpickle action_space Cloudpickle policy Cloudpickle n_envs JSON _vectorize_action JSON policy_kwargs JSON gamma JSON n_steps JSON vf_coef JSON ent_coef JSON max_grad_norm JSON learning_rate JSON lam JSON nminibatches JSON noptepochs JSON cliprange JSON cliprange_vf JSON verbose JSON policy Cloudpickle observation_space Cloudpickle action_space Cloudpickle n_envs JSON _vectorize_action JSON policy_kwargs JSON gamma JSON timesteps_per_batch JSON max_kl JSON cg_iters JSON lam JSON entcoeff JSON cg_damping JSON vf_stepsize JSON vf_iters JSON hidden_size_adversary JSON adversary_entcoeff JSON expert_dataset JSON g_step JSON d_step JSON d_stepsize JSON using_gail JSON verbose JSON policy Cloudpickle observation_space Cloudpickle action_space Cloudpickle n_envs JSON _vectorize_action JSON policy_kwargs JSON ```

[araffin]

Great, I agree on both points ;). @hill-a @erniejunior @adamgleave could you comment on that so we can start the implementation?

[AdamGleave]

Things being human-readable where possible seems useful for debugging. I don't think there's any need to support languages other than Python, though: even if we're careful with the serialization format, it'd be very hard to make use of this outside of Stable Baselines, since the TensorFlow graph would have to be reconstructed exactly. (If this was a desiderata, we should probably save the graph and weights together.)

This is unrelated, but one thing that continually bugs me when saving models: if you're using VecNormalize, you have to separately save the normalization statistics. It might be nice to have functionality to save both a policy and the set of transformations it used?

[Miffyli]

@AdamGleave

I agree on both points. Hence I would just cloudpickle/pickle the non-JSON-able objects and leave it at that.

One open point is the storing of model parameters: Should this be Numpy savez object or something more universal? I figure savez (and save) format is long-lasting and easy enough for people to use the values somewhere else-

I did quick experiments with storing objects with this new format, and here is the JSON (class parameters) part of the saved model. Objects that are serialized with cloudpickle (:serialized:) also include first-level members of the object so human reader can get some idea what the serialization contains. These are not used in any way when reading the file (only :serialized: is read and deserialized).

Example JSON file of class parameters

``` { "gamma": 0.99, "timesteps_per_batch": 1024, "max_kl": 0.01, "cg_iters": 10, "lam": 0.98, "entcoeff": 0.0, "cg_damping": 0.01, "vf_stepsize": 0.0003, "vf_iters": 3, "hidden_size_adversary": 100, "adversary_entcoeff": 0.001, "expert_dataset": null, "g_step": 1, "d_step": 1, "d_stepsize": 0.0003, "using_gail": false, "verbose": 0, "policy": { ":type:": "", ":serialized:": "gASVMgAAAAAAAACMIHN0YWJsZV9iYXNlbGluZXMuY29tbW9uLnBvbGljaWVzlIwJTWxwUG9saWN5lJOULg==", "__module__": "stable_baselines.common.policies", "__doc__": "\n Policy object that implements actor critic, using a MLP (2 layers of 64)\n\n :param sess: (TensorFlow session) The current TensorFlow session\n :param ob_space: (Gym Space) The observation space of the environment\n :param ac_space: (Gym Space) The action space of the environment\n :param n_env: (int) The number of environments to run\n :param n_steps: (int) The number of steps to run for each environment\n :param n_batch: (int) The number of batch to run (n_envs * n_steps)\n :param reuse: (bool) If the policy is reusable or not\n :param _kwargs: (dict) Extra keyword arguments for the nature CNN feature extraction\n ", "__init__": "", "__abstractmethods__": "frozenset()", "_abc_registry": "<_weakrefset.WeakSet object at 0x7feb24c340f0>", "_abc_cache": "<_weakrefset.WeakSet object at 0x7feb24c34128>", "_abc_negative_cache": "<_weakrefset.WeakSet object at 0x7feb24c34198>", "_abc_negative_cache_version": 52 }, "observation_space": { ":type:": "", ":serialized:": "gASVTgsAAAAAAACME2d5bS5zcGFjZXMuZGlzY3JldGWUjAhEaXNjcmV0ZZSTlCmBlH2UKIwBbpRLCowFc2hhcGWUKYwFZHR5cGWUjAVudW1weZSMBWR0eXBllJOUjAJpOJRLAEsBh5RSlChLA4wBPJROTk5K/////0r/////SwB0lGKMCW5wX3JhbmRvbZSMFG51bXB5LnJhbmRvbS5fcGlja2xllIwSX19yYW5kb21zdGF0ZV9jdG9ylJOUjAdNVDE5OTM3lIWUUpR9lCiMDWJpdF9nZW5lcmF0b3KUaBSMBXN0YXRllH2UKIwDa2V5lIwVbnVtcHkuY29yZS5tdWx0aWFycmF5lIwMX3JlY29uc3RydWN0lJOUaAiMB25kYXJyYXmUk5RLAIWUQwFilIeUUpQoSwFNcAKFlGgKjAJ1NJRLAEsBh5RSlChLA2gOTk5OSv////9K/////0sAdJRiiULACQAAW6DUsAzWsNOFcUrEouywwBF3V4lPOLX1Xz38oPU+Ft9cmSdH+UdKVn9ZtIUnq6RP6Mn1mqpv9eamqbx1OvrRab59kHcR+kFSd8mlTnwjWpsL797ktjG68OFSTa52PM4iqJpJOdzQFK7Ya4YlvS238Xav/nWTBJgNoxEQ82f0pka7olRWNoR54gIVszlaFWckVM0PArDWId+aJ+TWDqHpOkHom217lPxCqiHJvG/jZ3CyOLA5ID7nwjQRQ2x/6FLJnW2hAPwje7eDUmmSxhm3J2OAuzqZPI30/o9P7OiFAN6aM4wb8Mg3fVua1xIxsgI/88AkcSMhpYXgHnxhf6UkXb/c3pwQ6HQhCWrPEaMgonDwGDMazfV2oSaUmhzWxVP27GpNKRX2OryVeJoKAcCvsXRXkrigucUQiESyksdsTE1HQYmxZeAM2PhNL3hG8jwaUZqsIADZk1vUGMQXX5C9ZMUkHg/ESKviQwacFSKZtt/7RF8iSFB/0zFF/JwQ+xtwueDgwWRB2MnS8q/JvkC7uriGXG1qDEfjaNAM751mZy/X4PQv0AH+YE4o5pYplmO/+hwix/xec3whdl4+KAinMUpLGJMGKERO/i174h8YAmhMepSEN6yQPuTszOCwDJOZGdfZ+ap3SYzC6PIG3ELgXf69Ek++j0zvpRkLP1rCLprf76pd9hgSLrX2kaT1jAbUgnJhdJdEhyJy+4JbXSxc9P6hPLiHZeicFHzXSWTE0hJ0H8PuP8RNEEJe+oL9E4xgSJLLNXFhQd63zVkg6v9bxR0X1pzag8kC1UtF6oH9xhBUoAavFA2J6J+PsHzTsZE5YOGw1YhKv8camBtB+InSXLeL56wwXYc/m+D+fWJeL1P8/hHDB5VuqIajBaEHppbTtH3osXyd9ewomecdRmLYaI6GP+Vy2wlu8/AlCp7fcZChr4JOwcuIqfhU+Ij063ALAQWwglAuHzgO/BAL6Wqw36LCkiARi4WbSROuj3CRYLqhk4AizvA4zDEz7cUCjWm4tS73WkDVeN0U4sa3PDY2uNpwF7Nj9jl6+IwSCv2HDqHvPkOcjwORfQMaJx7i990yst8p+naj13pZCQB9semWWGxbi3LY21UVG8lg5lvmizJBHFpCYTBV4AlaHnk9BBXnmnsZftuFtHvDDXAV3nvvuouP2qQhy3VE3dbmH9xazRQRy4dENoB4ydClla5yAzv6G3fxMs1uxOf5Vo+MZrlmWxCUsitH6A0p2qq1ulMixe73W51u/bcb6JquDCgtFF8xIm8ZGzP+jyvQSi0oPCWH+dIKipQlc7thg8fJ3zxQ0GvNAzmXqoOXr5+OkClJnGA+WWGYEl4Jy5y5UKOKtTmv91BZbciXByuGnGEDk4z/aWrdHBRzLrIqRpo8rvu/XRDpSt1Crh/woK4l1CtohhHfgybR0+dZzHoew5m9Dd0slOdkUFB0YJDFAouVDhbaKfmmxZO14ZS3Lt43yBEiv67U5cHwNdudZP7bnQV8bP2JF0a9atXC+h/iHBZgMJSYvByflIEFMkZUAUyeq3mDeudXTj6f6EIOduOuqGLNgZui9Sy303WUpyD+IHZQt+PwbFyae+gaGQ4wFN7zp0B2fnNYwb7K0RQImf4/4gkABgQp/J5h4JVQXxt+f06dJLHroswT2MhUR2b/7go0O/D7GewQaB4SWLmRpzmCFsOHp7akEiRPFLQrOXvg0jgiUUptf0cW+LuyeLo3bYbs9k8bkvOX/VgbdSujE6SvGA6oSe6ztUT11dvw8+aft0IWMa9EnJzILXa1NShJVM6nTu2r2cZRG7vaqQE4OJTr8XQZw8qMgBbStI/NRLfgbSA+XJIVV/L6ixpRYQCC5lHlbHcnSMoJi0pSYZ5fr71eWjDijhULcH6csQnIO/YNZKKJF3Q0xbxaqP6aq7C3RsIRoQsls2sSv9/uZ4IQ4D/ioe8m397X/Zs6QJZ/hYK0nXZn1c6dDSOaDJibVtkKBUTdWhbo6e93y1bwMWC+PYkUkxAmEx2Y1SnntwoUmiJYH179BCQLignL1omY+HagH5ZP5odULYNdU/A4oJHoSEgPxYDtnta4VRsu21fN5a9Tx/+0rA08cghouw32yH4A7qjboJdpX3b7YhKinhhzjFtNLmjy79k6o5KDLM9vb6oswJFAp82SEFkLCiycVvUTTE7pwOB3PpRzcHUdHTBfnPF0FuPp//hjNv7srG5XjzuiH9wvmK1oYxsnvm2vqcomlBE0gjiW/nsqXybwJtqpGxYu8ipbqsYpqDN+QzRtu+S+jRfnwdkcAazG9cpasrAoUDhWTYzVbgNJBdraafYa6WxJNXdlB26/CkgXQEAJ28qZDLccx7h20V+0NH/Mj9fMd2+jOq9Ksd50k5MQqzWosk4dynbUhbGM/atehHVf10fxTWftyXBL1aI+FvaGKJj/NxcxnhWhNwFn5zd0tBgI6BpXZup/vdsMrc5S+ZrRafbQi5Rhhjr88gZHmOcBMljLe7cOLtmXhqk/HrI+47ftamQ8Oy6MGY+F6RcxSOWlQpU4w3MjhfnKqw9NEc18+Bjg9O/+60oj9SNNteMW8xL/d2D6+PS0k8R18oFQl/zOk9yBbgeqYkYTZ39pFygk1KdbbgybFkj/VMwkWEKGy+qIFWP2NWbVHf0nqd/U8rVfvo9PHcDF92HphtkYGqqMgbwfZhQWJfXO8Ijw7f+vvWZzgpt2khJSwYaBiJH9czE2rg5ps4CDlQ64nKnGWE/USQCAfyicP7MesC2+wE1gmLMR4WCDLHq7YhaRHTpUPmYqWx9xuB9AVIIL147OZPeYTfqTHDI53kbVnw4nqds4pAa64MvQtsZT/r46alHXeWJDvIwF3S2E1hHoGsXSt62RPtZpbcZ1hl4wZqaM3tkuX6ZIWsFJ7+Q8pGrPrvWdaFRMg/g/QDE0SmWu706c3q0yPYoEdqxQldbeEveY81V0sR6rRftLnWWGJ637ujgDNYujxwCZJpJ52hy8gdhTgli78HKd9nhoqx2wo3BS403A9CCAcSOCTFR8zCde6foFoPCp0pUp6DAtNLwvpmdNBTRlfF4cLh38FUvfM8r6gwNs3WFQ63pyy9fIHFPI6ApUFcgZ1VbmnA9jUOiJ6KKhQygLFZ/q91ZCqCYTgDfUUj8UqWXfpa4Ucmljc6KX2JDgDhKFa5wvYUiIHO6LdMClSU6sc5FxVpMJOx+jFgD2O7HII9kueQM9QjKjFo7H4ZxX1uX88a+oV04JkJDIDSd3iTFqqfOn//1KrCutmxdLoyHJ0PeizgDmcQbFsrGi0qya1LsW2PdIuwXscMRJ7ZRJlHSUYowDcG9zlEsDdYwJaGFzX2dhdXNzlEsAjAVnYXVzc5RHAAAAAAAAAAB1YnViLg==", "n": 10, "shape": [], "dtype": "int64", "np_random": "RandomState(MT19937)" }, "action_space": { ":type:": "", ":serialized:": "gASVTgsAAAAAAACME2d5bS5zcGFjZXMuZGlzY3JldGWUjAhEaXNjcmV0ZZSTlCmBlH2UKIwBbpRLCowFc2hhcGWUKYwFZHR5cGWUjAVudW1weZSMBWR0eXBllJOUjAJpOJRLAEsBh5RSlChLA4wBPJROTk5K/////0r/////SwB0lGKMCW5wX3JhbmRvbZSMFG51bXB5LnJhbmRvbS5fcGlja2xllIwSX19yYW5kb21zdGF0ZV9jdG9ylJOUjAdNVDE5OTM3lIWUUpR9lCiMDWJpdF9nZW5lcmF0b3KUaBSMBXN0YXRllH2UKIwDa2V5lIwVbnVtcHkuY29yZS5tdWx0aWFycmF5lIwMX3JlY29uc3RydWN0lJOUaAiMB25kYXJyYXmUk5RLAIWUQwFilIeUUpQoSwFNcAKFlGgKjAJ1NJRLAEsBh5RSlChLA2gOTk5OSv////9K/////0sAdJRiiULACQAAW6DUsAzWsNOFcUrEouywwBF3V4lPOLX1Xz38oPU+Ft9cmSdH+UdKVn9ZtIUnq6RP6Mn1mqpv9eamqbx1OvrRab59kHcR+kFSd8mlTnwjWpsL797ktjG68OFSTa52PM4iqJpJOdzQFK7Ya4YlvS238Xav/nWTBJgNoxEQ82f0pka7olRWNoR54gIVszlaFWckVM0PArDWId+aJ+TWDqHpOkHom217lPxCqiHJvG/jZ3CyOLA5ID7nwjQRQ2x/6FLJnW2hAPwje7eDUmmSxhm3J2OAuzqZPI30/o9P7OiFAN6aM4wb8Mg3fVua1xIxsgI/88AkcSMhpYXgHnxhf6UkXb/c3pwQ6HQhCWrPEaMgonDwGDMazfV2oSaUmhzWxVP27GpNKRX2OryVeJoKAcCvsXRXkrigucUQiESyksdsTE1HQYmxZeAM2PhNL3hG8jwaUZqsIADZk1vUGMQXX5C9ZMUkHg/ESKviQwacFSKZtt/7RF8iSFB/0zFF/JwQ+xtwueDgwWRB2MnS8q/JvkC7uriGXG1qDEfjaNAM751mZy/X4PQv0AH+YE4o5pYplmO/+hwix/xec3whdl4+KAinMUpLGJMGKERO/i174h8YAmhMepSEN6yQPuTszOCwDJOZGdfZ+ap3SYzC6PIG3ELgXf69Ek++j0zvpRkLP1rCLprf76pd9hgSLrX2kaT1jAbUgnJhdJdEhyJy+4JbXSxc9P6hPLiHZeicFHzXSWTE0hJ0H8PuP8RNEEJe+oL9E4xgSJLLNXFhQd63zVkg6v9bxR0X1pzag8kC1UtF6oH9xhBUoAavFA2J6J+PsHzTsZE5YOGw1YhKv8camBtB+InSXLeL56wwXYc/m+D+fWJeL1P8/hHDB5VuqIajBaEHppbTtH3osXyd9ewomecdRmLYaI6GP+Vy2wlu8/AlCp7fcZChr4JOwcuIqfhU+Ij063ALAQWwglAuHzgO/BAL6Wqw36LCkiARi4WbSROuj3CRYLqhk4AizvA4zDEz7cUCjWm4tS73WkDVeN0U4sa3PDY2uNpwF7Nj9jl6+IwSCv2HDqHvPkOcjwORfQMaJx7i990yst8p+naj13pZCQB9semWWGxbi3LY21UVG8lg5lvmizJBHFpCYTBV4AlaHnk9BBXnmnsZftuFtHvDDXAV3nvvuouP2qQhy3VE3dbmH9xazRQRy4dENoB4ydClla5yAzv6G3fxMs1uxOf5Vo+MZrlmWxCUsitH6A0p2qq1ulMixe73W51u/bcb6JquDCgtFF8xIm8ZGzP+jyvQSi0oPCWH+dIKipQlc7thg8fJ3zxQ0GvNAzmXqoOXr5+OkClJnGA+WWGYEl4Jy5y5UKOKtTmv91BZbciXByuGnGEDk4z/aWrdHBRzLrIqRpo8rvu/XRDpSt1Crh/woK4l1CtohhHfgybR0+dZzHoew5m9Dd0slOdkUFB0YJDFAouVDhbaKfmmxZO14ZS3Lt43yBEiv67U5cHwNdudZP7bnQV8bP2JF0a9atXC+h/iHBZgMJSYvByflIEFMkZUAUyeq3mDeudXTj6f6EIOduOuqGLNgZui9Sy303WUpyD+IHZQt+PwbFyae+gaGQ4wFN7zp0B2fnNYwb7K0RQImf4/4gkABgQp/J5h4JVQXxt+f06dJLHroswT2MhUR2b/7go0O/D7GewQaB4SWLmRpzmCFsOHp7akEiRPFLQrOXvg0jgiUUptf0cW+LuyeLo3bYbs9k8bkvOX/VgbdSujE6SvGA6oSe6ztUT11dvw8+aft0IWMa9EnJzILXa1NShJVM6nTu2r2cZRG7vaqQE4OJTr8XQZw8qMgBbStI/NRLfgbSA+XJIVV/L6ixpRYQCC5lHlbHcnSMoJi0pSYZ5fr71eWjDijhULcH6csQnIO/YNZKKJF3Q0xbxaqP6aq7C3RsIRoQsls2sSv9/uZ4IQ4D/ioe8m397X/Zs6QJZ/hYK0nXZn1c6dDSOaDJibVtkKBUTdWhbo6e93y1bwMWC+PYkUkxAmEx2Y1SnntwoUmiJYH179BCQLignL1omY+HagH5ZP5odULYNdU/A4oJHoSEgPxYDtnta4VRsu21fN5a9Tx/+0rA08cghouw32yH4A7qjboJdpX3b7YhKinhhzjFtNLmjy79k6o5KDLM9vb6oswJFAp82SEFkLCiycVvUTTE7pwOB3PpRzcHUdHTBfnPF0FuPp//hjNv7srG5XjzuiH9wvmK1oYxsnvm2vqcomlBE0gjiW/nsqXybwJtqpGxYu8ipbqsYpqDN+QzRtu+S+jRfnwdkcAazG9cpasrAoUDhWTYzVbgNJBdraafYa6WxJNXdlB26/CkgXQEAJ28qZDLccx7h20V+0NH/Mj9fMd2+jOq9Ksd50k5MQqzWosk4dynbUhbGM/atehHVf10fxTWftyXBL1aI+FvaGKJj/NxcxnhWhNwFn5zd0tBgI6BpXZup/vdsMrc5S+ZrRafbQi5Rhhjr88gZHmOcBMljLe7cOLtmXhqk/HrI+47ftamQ8Oy6MGY+F6RcxSOWlQpU4w3MjhfnKqw9NEc18+Bjg9O/+60oj9SNNteMW8xL/d2D6+PS0k8R18oFQl/zOk9yBbgeqYkYTZ39pFygk1KdbbgybFkj/VMwkWEKGy+qIFWP2NWbVHf0nqd/U8rVfvo9PHcDF92HphtkYGqqMgbwfZhQWJfXO8Ijw7f+vvWZzgpt2khJSwYaBiJH9czE2rg5ps4CDlQ64nKnGWE/USQCAfyicP7MesC2+wE1gmLMR4WCDLHq7YhaRHTpUPmYqWx9xuB9AVIIL147OZPeYTfqTHDI53kbVnw4nqds4pAa64MvQtsZT/r46alHXeWJDvIwF3S2E1hHoGsXSt62RPtZpbcZ1hl4wZqaM3tkuX6ZIWsFJ7+Q8pGrPrvWdaFRMg/g/QDE0SmWu706c3q0yPYoEdqxQldbeEveY81V0sR6rRftLnWWGJ637ujgDNYujxwCZJpJ52hy8gdhTgli78HKd9nhoqx2wo3BS403A9CCAcSOCTFR8zCde6foFoPCp0pUp6DAtNLwvpmdNBTRlfF4cLh38FUvfM8r6gwNs3WFQ63pyy9fIHFPI6ApUFcgZ1VbmnA9jUOiJ6KKhQygLFZ/q91ZCqCYTgDfUUj8UqWXfpa4Ucmljc6KX2JDgDhKFa5wvYUiIHO6LdMClSU6sc5FxVpMJOx+jFgD2O7HII9kueQM9QjKjFo7H4ZxX1uX88a+oV04JkJDIDSd3iTFqqfOn//1KrCutmxdLoyHJ0PeizgDmcQbFsrGi0qya1LsW2PdIuwXscMRJ7ZRJlHSUYowDcG9zlEsDdYwJaGFzX2dhdXNzlEsAjAVnYXVzc5RHAAAAAAAAAAB1YnViLg==", "n": 10, "shape": [], "dtype": "int64", "np_random": "RandomState(MT19937)" }, "n_envs": 1, "_vectorize_action": true, "policy_kwargs": {} } ```

[araffin]

Should this be Numpy savez object or something more universal?

Using numpy format sounds reasonable.

I did quick experiments with storing objects with this new format, and here is the JSON

Nice ;)