search

Distributive-Network / Bifrost

An SHM/memmap based python and node communication/evaluation tool.
Other
7 stars 2 forks source link

Cloudpickle version conflict in Google Colab #13

Open ccmcintyre-kingsds opened 2 years ago

ccmcintyre-kingsds commented 2 years ago

In the current default Google Colab environment, cloudpickle is pre-installed as version 1.3.0, which is required by the pre-installed gym library (Google's AI Gym), as of gym version 0.17.3, which claims not-compatibility with cloudpickle version 2.0.0.

The pip depenency manager requires that gym be uninstalled, cloudpickle upgraded or reinstalled, and will still insist that the Google Colab runtime be restarted in order to recognize the upgraded installation. This is not a viable UX.

The newer features of cloudpickle version 2.0.0 are required for the serialization of some more complex jobs, when using our buildless Pyodide API via Bifrost. Until this issue is addressed on our end, or Google upgrades its package versions, those jobs (and for the short term, the buildless Pyodide API) will be restricted to non-Colab environments.

ccmcintyre-kingsds commented 2 years ago

Pyodide jobs are now able to be deployed from Colab environments, and Bifrost is now installing correctly in Colabs.

At present, cloudpickle's version conflicts are deeper than with the Colab's gym module; the Python version (3.7 in default Colab) is potentially introducing pickling errors when used with the updated Cloudpickle version. Further investigation is required before the full cloudpickle funtionality will be supported for Colabs. Fortunately, most of our Python jobs no longer rely on this feature (and NodeJS jobs have of course never required it).