Upgrading Colab host environment to Ubuntu LTS 20.04

[drew-parsons]

Dear Colab team,

the recent system upgrade from python 3.6 to python 3.7 is creating some issues for libraries packaged in PPAs, especially those which have a C++ backend wrapped to python with tools like pybind11, swig or Cython. The issue is that the C++ backend is compiled for python 3.6, and this is not necessarily compatible with python 3.7.

See for instance https://github.com/googlecolab/colabtools/issues/1878, https://github.com/googlecolab/colabtools/issues/1877, https://github.com/googlecolab/colabtools/issues/1873, https://fenicsproject.discourse.group/t/installing-fenics-from-ppa-on-colab/5148 .

The upgrade came as a surprise to software developers and PPA maintainers, and each development team is currently dealing with it on their own. Unfortunately, it is not possible to ask PPA maintainers to support packaging a special python 3.7 version targeted for Google Colab, because python 3.7 has been explicitly excluded from the list of supported python versions by Ubuntu 18.04 LTS, see https://launchpad.net/ubuntu/+source/python3-defaults/3.6.7-1~18.04 . Software developers will then have to cope by differently packaging such Colab specific versions and releasing them in some other way.

Considering that Ubuntu 18.04 LTS is now almost three years old, what are the plans to upgrade Colab to Ubuntu 20.04 LTS in future? This would probably fix in a very natural way many of the packaging issues introduced by the recent Colab upgrade.

If an upgrade to Ubuntu 20.04 LTS is planned, please keep this ticket open to inform us on the progress.

Thanks, Drew Parsons @drew-parsons Francesco Ballarin @francesco-ballarin

[dhyeythumar]

Hi @drew-parsons, How to check what changes are made at the kernel level on Colab? Because I used to train reinforcement learning agents & live streaming the process to twitch using X server virtual screen & ffmpeg on Colab. But now the same streaming code has stopped working and, I am not able to figure out why this is happening.

[drew-parsons]

Hi @Dhyeythumar , you can check the Colab python version with

import sys; print(sys.version)

which would match the identity of the python3 command.

!ls -l /usr/bin/python3

On Colab the symlink for this command has been changed from the original python3.6 for Ubuntu 18.04, to /etc/alternatives/python3, which !ls -l /etc/alternatives/python3 shows pointing at /usr/bin/python3.7

You can check the version of the host system with

!lsb_release -a

The linux kernel shouldn't be an issue here, but you can check it with !uname -a.

[colaboratory-team]

Upgrading to 20.04 had been blocked on upstream availability of some dependencies but should now be unblocked. This issue will track that update (and internally this is b/181338315). We don't have a timeline to share yet, sorry.

[soonho-tri]

• In Ubuntu 20.04 LTS, the python included in the base system is Python 3.8. Does that mean we will have Python3.8 runtime when b/181338315 is resolved?
• Is there a place (e.g. github / blog / mailing-list / twitter) where breaking changes are announced in advance so that users can prepare?

[craigcitro]

• In Ubuntu 20.04 LTS, the python included in the base system is Python 3.8. Does that mean we will have Python3.8 runtime when b/181338315 is resolved?

Yes -- at least, that's the plan.

• Is there a place (e.g. github / blog / mailing-list / twitter) where breaking changes are announced in advance so that users can prepare?

Unfortunately, no.

[drew-parsons]

Upgrading to 20.04 had been blocked on upstream availability of some dependencies but should now be unblocked. This issue will track that update (and internally this is b/181338315).

Thanks Colab Team. Our users are looking forward to the upgrade.

[vak]

Today is exactly "3 months later", so, I thought it would be OK to ask:

is there any news/updates on new Google Colab image?

[segatrade]

Today is exactly "3 months later", so, I thought it would be OK to ask:

is there any news/updates on new Google Colab image?

Not sure, but probably on Dec 26, 2021 drop support for Python 3.7 and update to Ubuntu 20.04 Python 3.8. NEP 29 deprecation policy: https://numpy.org/neps/nep-0029-deprecation_policy.html

[vak]

any update on this issue?

[cperry-goog]

Thanks, no updates, this is on our radar.

[ogrisel]

For information scikit-learn is also anticipating to stop supporting Python 3.7 in its next major release (Q2 2022) in accordance to NEP 29.

[demining]

Hello everybody! Help me to solve this problem. I try to install the fpylll package onpython3 but I get a refusal.

apt install python3-fpylll

Reading package lists ... Done
Building dependency tree
Reading state information ... Done
E: Unable to locate package python3-fpylll

The fact is that Google Colab uses Ubuntu 18.04.5 LTS, but judging by the answer, to fully install package python3-fpylll, me need Ubuntu 21.04 Is it possible to solve this problem in Google Colab itself?

[jasonkhadka]

Repeating the question again in 2 months but: Any update? Do you know when can we expect this?

Thanks, no updates, this is on our radar.

[matteobachetti]

I guess this means that Google Colab will never update to Ubuntu 20.04? :(

[ElDavoo]

I made a silent block to update from 18 to 20. Unfortunately, that takes around 30 minutes and probably (90%) breaks a lot of stuff, so use at your risk.

!apt-mark unhold libnccl-dev libnccl2 libcudnn8 > /dev/null
!apt update > /dev/null 2>&1 
!apt -y upgrade > /dev/null 2>&1 
!apt -y full-upgrade > /dev/null 2>&1 
!apt -y autoremove > /dev/null 2>&1
!apt install ubuntu-release-upgrader-core
!rm /usr/bin/python3
!ln -sf /usr/bin/python3.6 /usr/bin/python3 > /dev/null 2>&1 
!do-release-upgrade -f DistUpgradeViewNonInteractive; > /dev/null 2>&1 

[vak]

I guess this means that Google Colab will never update to Ubuntu 20.04? :(

actually it is already quite a time for Ubuntu 22.04 ;)

[ElDavoo]

actually it is already quite a time for Ubuntu 22.04 ;)

What I would like to have is a variety of different base images, for example when you choose "runtime type" (cpu, gpu, tpu), you can also choose "base image" (18 deprecated, 20, 22 beta)

[EwoutH]

Upgrading to 20.04 had been blocked on upstream availability of some dependencies but should now be unblocked. This issue will track that update (and internally this is b/181338315). We don't have a timeline to share yet, sorry.

@colaboratory-team We are now a full year further, could you provide any update?

[fhk]

[Danc2050]

@colaboratory-team or anyone: I made a notebook that upgrades a Colab VM in about an hour (if you interactively select no to remove packages, don't restart).

It may be a workaround for those that need Ubuntu 20.04 without a restart and who don't mind wasting ~1 hour upgrading their notebook VM each time.

Here are the commands:

!sudo apt update && sudo apt upgrade && sudo apt-get install libcudnn8 libcudnn8-dev libnccl-dev libnccl2 --allow-change-held-packages && sudo apt dist-upgrade
!sudo apt autoremove
!sudo apt install update-manager-core
!sudo ln -sf /usr/bin/python3.6 /usr/bin/python3
!export DEBIAN_FRONTEND=noninteractive # or sudo apt-get install dialog && sudo apt-get install whiptail
!sudo do-release-upgrade -f DistUpgradeViewNonInteractive

[ElDavoo]

I made a silent block

update:

!apt install -y iptables > /dev/null 2>&1 
!apt-mark unhold libnccl-dev libnccl2 libcudnn8 > /dev/null
!apt update > /dev/null 2>&1 
!apt -y upgrade > /dev/null 2>&1 
!apt -y full-upgrade > /dev/null 2>&1 
!apt -y autoremove > /dev/null 2>&1
!apt --fix-broken install > /dev/null 2>&1
!apt install ubuntu-release-upgrader-core > /dev/null 2>&1
!apt-mark unhold libnccl-dev libnccl2 libcudnn8 > /dev/null
!apt update > /dev/null 2>&1 
!apt -y upgrade > /dev/null 2>&1 
!apt -y full-upgrade > /dev/null 2>&1 
!apt -y autoremove > /dev/null 2>&1
!apt --fix-broken install > /dev/null 2>&1
!apt install ubuntu-release-upgrader-core > /dev/null 2>&1
!rm /usr/bin/python3
!ln -sf /usr/bin/python3.6 /usr/bin/python3 > /dev/null 2>&1 
!do-release-upgrade -f DistUpgradeViewNonInteractive; > /dev/null 2>&1 
!ln -sf /usr/bin/python3.8 /usr/bin/python3 > /dev/null 2>&1 
!apt install --reinstall -y python3-pip > /dev/null 2>&1 

[chadbrewbaker]

Bump. I have a C++17 dependency that is horribly broken on the 18 compile chain. This needs to be prioritized.

[EwoutH]

At this point I would suggest going straight to Ubuntu 22.04 LTS (Jammy Jellyfish). That will also directly solve #2165, when using the default Python version of Ubuntu 22.04, which is Python 3.10.

[fhk]

+1

[RyuuOujiXS]

Upgrading to 20.04 had been blocked on upstream availability of some dependencies but should now be unblocked. This issue will track that update (and internally this is b/181338315).

Thanks Colab Team. Our users are looking forward to the upgrade.

Thanks for thanking the Colab Team. Your thanks are appreciated.

Today is exactly "3 months later", so, I thought it would be OK to ask: is there any news/updates on new Google Colab image?

Not sure, but probably on Dec 26, 2021 drop support for Python 3.7 and update to Ubuntu 20.04 Python 3.8. NEP 29 deprecation policy: https://numpy.org/neps/nep-0029-deprecation_policy.html

Thanks for replying to a question your don't have an answer to. Your lack of knowledge has been noted.

I guess this means that Google Colab will never update to Ubuntu 20.04? :(

actually it is already quite a time for Ubuntu 22.04 ;)

...and?

actually it is already quite a time for Ubuntu 22.04 ;)

What I would like to have is a variety of different base images, for example when you choose "runtime type" (cpu, gpu, tpu), you can also choose "base image" (18 deprecated, 20, 22 beta)

Then submit a feature request, kid

At this point I would suggest going straight to Ubuntu 22.04 LTS (Jammy Jellyfish). That will also directly solve #2165, when using the default Python version of Ubuntu 22.04, which is Python 3.10.

Please don't reply to issues you don't understand. If you knew how to fix this, you would have fixed it already. The fact that you're here proves your suggestions are meaningless.

[RyuuOujiXS]

Also, to all the kids crying: This issue is triaged with no one assigned to it. What more proof do you need to believe that this issue doesn't matter to the devs?

[petrasovaa]

It looks like it's happening: #3327. I just got Ubuntu 20.04 for my runtime.

[francesco-ballarin]

@metrizable #3327 closes this

[metrizable]

@drew-parsons Thanks for filing the request. Colab's runtime has been upgraded to Ubuntu 20.04 with plans to upgrade to 22.04 in the future.

[drew-parsons]

Thanks @metrizable

[drew-parsons]

Mind you, gotta say, it's still messed up. Ubuntu 20.04 comes with python3.8, but you've overwritten that with python3.9. That means that colab is still incompatible with the python packages built for the system that it's running on. Which sort of defeats the point of the upgrade. It's packages with C extensions built using cython that are affected.

When you do upgrade to 22.04, please don't undermine the system that you're running it on.

[smartibase]

Now with the upgrade to Ubuntu 20.04, all scripts that have code:

from sage.all import *

you give an error:

!sage-python3 script.py

Traceback (most recent call last): File "/script.py", line 1, in from sage.all import * File "/usr/lib/python3/dist-packages/sage/all.py", line 77, in from cysignals.signals import (AlarmInterrupt, SignalError, File "/usr/lib/python3/dist-packages/cysignals/init.py", line 1, in from .signals import AlarmInterrupt, SignalError, init_cysignals # noqa ModuleNotFoundError: No module named 'cysignals.signals'sage-python3 script.py

Has anyone been able to resolve this issue?

[demining]

@c0de91101 This obviously should help you "SageMath-in-Google-Colab"

[drew-parsons]

demining refers to a Russian bitcoin scam. Pulling hacked code from https://cryptodeeptech.ru is not going to help @c0de91101 .