Update ubuntu version from 18.04 to 22.04

[eugeneteoh]

Changes

• Update ubuntu version from 18.04 to 22.04

Checklist

• [x] Read the contribution guide and accept the code of conduct
• [x] Readme (updated or not needed)

[github-actions[bot]]

[webbertakken]

This by itself doesn't close #215 there are multiple steps (as described) and a somewhat elaborate release process.

As the base image is meant to be relatively small, could you show the differences in size when changing dependencies? i.e. ubuntu-22 with same deps vs ubuntu-22 with the new deps.

It would also be helpful if you could elaborate on why each dependency is needed and which one it replaces, for future reference.

[eugeneteoh]

Old image size:

❯ docker inspect -f "{{ .Size }}" base | numfmt --to=si && docker history base
467M
IMAGE          CREATED              CREATED BY                                      SIZE      COMMENT
8f38a0d933be   About a minute ago   ENV UNITY_PATH=/opt/unity                       0B        buildkit.dockerfile.v0
<missing>      About a minute ago   RUN /bin/sh -c echo "576562626572264761624c6…   33B       buildkit.dockerfile.v0
<missing>      About a minute ago   RUN /bin/sh -c /bin/echo -e 'pcm.!default {\…   52B       buildkit.dockerfile.v0
<missing>      About a minute ago   RUN /bin/sh -c apt-get -q update  && apt-get…   4.54MB    buildkit.dockerfile.v0
<missing>      2 minutes ago        RUN /bin/sh -c apt-get -q update  && apt-get…   93.2MB    buildkit.dockerfile.v0
<missing>      2 minutes ago        RUN /bin/sh -c apt-get -q update  && apt-get…   305MB     buildkit.dockerfile.v0
<missing>      5 hours ago          RUN /bin/sh -c echo 'debconf debconf/fronten…   911kB     buildkit.dockerfile.v0
<missing>      5 hours ago          ENV LC_ALL=C.UTF-8                              0B        buildkit.dockerfile.v0
<missing>      5 hours ago          ENV LANG=C.UTF-8                                0B        buildkit.dockerfile.v0
<missing>      5 weeks ago          /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B        
<missing>      5 weeks ago          /bin/sh -c #(nop) ADD file:3c74e7e08cbf9a876…   63.2MB    
<missing>      5 weeks ago          /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B        
<missing>      5 weeks ago          /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B        
<missing>      5 weeks ago          /bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH     0B        
<missing>      5 weeks ago          /bin/sh -c #(nop)  ARG RELEASE                  0B

New image size:

❯ docker inspect -f "{{ .Size }}" base | numfmt --to=si && docker history base
702M
IMAGE          CREATED              CREATED BY                                      SIZE      COMMENT
92532541d024   About a minute ago   ENV UNITY_PATH=/opt/unity                       0B        buildkit.dockerfile.v0
<missing>      About a minute ago   RUN /bin/sh -c echo "576562626572264761624c6…   33B       buildkit.dockerfile.v0
<missing>      About a minute ago   RUN /bin/sh -c /bin/echo -e 'pcm.!default {\…   52B       buildkit.dockerfile.v0
<missing>      About a minute ago   RUN /bin/sh -c apt-get -q update  && apt-get…   192MB     buildkit.dockerfile.v0
<missing>      About a minute ago   RUN /bin/sh -c apt-get -q update  && apt-get…   82MB      buildkit.dockerfile.v0
<missing>      About a minute ago   RUN /bin/sh -c apt-get -q update  && apt-get…   350MB     buildkit.dockerfile.v0
<missing>      5 hours ago          RUN /bin/sh -c echo 'debconf debconf/fronten…   993kB     buildkit.dockerfile.v0
<missing>      5 hours ago          ENV LC_ALL=C.UTF-8                              0B        buildkit.dockerfile.v0
<missing>      5 hours ago          ENV LANG=C.UTF-8                                0B        buildkit.dockerfile.v0
<missing>      10 days ago          /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B        
<missing>      10 days ago          /bin/sh -c #(nop) ADD file:140fb5108b4a2861b…   77.8MB    
<missing>      10 days ago          /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B        
<missing>      10 days ago          /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B        
<missing>      10 days ago          /bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH     0B        
<missing>      10 days ago          /bin/sh -c #(nop)  ARG RELEASE                  0B

[eugeneteoh]

It would also be helpful if you could elaborate on why each dependency is needed and which one it replaces, for future reference.

Added comments to the changes. Let me know if this is sufficient.

[webbertakken]

Thank you @eugeneteoh!

Old image size: 467 MB New image size: 702 MB Size increase of base image: 51%

Everyone ok with the image sizes becoming quite a bit bigger, or does anyone see room for any optimisations?

[cloudymax]

Thank you @eugeneteoh!

Old image size: 467 MB New image size: 702 MB Size increase of base image: 51%

Everyone ok with the image sizes becoming quite a bit bigger, or does anyone see room for any optimisations?

At the cost of some readability you can prune it down to 667 by combining a few layers, and clearing the python cache + deleting some icons

docker inspect -f "{{ .Size }}" test2 | numfmt --to=si && docker history test2
667M
IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT
5637a3906eb2   2 minutes ago   ENV UNITY_PATH=/opt/unity                       0B        buildkit.dockerfile.v0
<missing>      2 minutes ago   RUN /bin/sh -c echo 'debconf debconf/fronten…   589MB     buildkit.dockerfile.v0
<missing>      2 minutes ago   ENV LC_ALL=C.UTF-8                              0B        buildkit.dockerfile.v0
<missing>      2 minutes ago   ENV LANG=C.UTF-8                                0B        buildkit.dockerfile.v0
<missing>      11 days ago     /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B
<missing>      11 days ago     /bin/sh -c #(nop) ADD file:140fb5108b4a2861b…   77.8MB
<missing>      11 days ago     /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B
<missing>      11 days ago     /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B
<missing>      11 days ago     /bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH     0B
<missing>      11 days ago     /bin/sh -c #(nop)  ARG RELEASE                  0B

ARG ubuntuImage="ubuntu:22.04"
FROM $ubuntuImage

# Fixes a Gradle crash while building for Android on Unity 2019 when there are accented characters in environment variables
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8

# Set frontend to Noninteractive in Debian configuration.
# https://github.com/phusion/baseimage-docker/issues/58#issuecomment-47995343
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections\
# Global dependencies
 && apt-get -q update \
 && apt-get -q install -y --no-install-recommends apt-utils \
 && apt-get -q install -y --no-install-recommends --allow-downgrades \
 ca-certificates \
 libasound2 \
 libc6-dev \
 libcap2 \
 libgconf-2-4 \
 libglu1 \
 libgtk-3-0 \
 libncurses5 \
 libnotify4 \
 libnss3 \
 libxtst6 \
 libxss1 \
 cpio \
 lsb-release \
 python3-pip \
 xvfb \
 xz-utils \
 atop \
 curl \
 git \
 git-lfs \
 jq \
 openssh-client \
 wget \
 zip \
 unzip \
 build-essential \
 libstdc++6 \
 && git lfs install --system --skip-repo \
 && apt-get autoremove \
 && apt-get clean \
 && rm -rf /var/lib/apt/lists/* \
 && rm -rf /usr/share/icons/* \
 && rm -rf /usr/lib/python3.10/__pycache__/* \
 # Disable default sound card, which removes ALSA warnings
 && /usr/bin/echo -e 'pcm.!default {\n\
    type plug\n\
    slave.pcm "null"\n\
}' > /etc/asound.conf \
# Support forward compatibility for unity activation
&& /usr/bin/echo "576562626572264761624c65526f7578" > /etc/machine-id && mkdir -p /var/lib/dbus/ && ln -sf /etc/machine-id /var/lib/dbus/machine-id

# Used by Unity editor in "modules.json" and must not end with a slash.
ENV UNITY_PATH="/opt/unity"

[webbertakken]

Thank you @eugeneteoh! Old image size: 467 MB New image size: 702 MB Size increase of base image: 51% Everyone ok with the image sizes becoming quite a bit bigger, or does anyone see room for any optimisations?

At the cost of some readability you can prune it down to 667 by combining a few layers, and clearing the python cache + deleting some icons

In my opinion readability is quite important if we want people to keep contributing and in case some deps become obsolete.

Clearing python cache and icons sounds like it's worth it though!

[AndrewKahr]

I think the more interesting thing before we start comparing image sizes is seeing what dependencies are needed/not needed anymore as currently, it looks like most of the test builds failed. Maybe I'm reading it wrong but according to dockerhub, Ubuntu 18.04 was 24.5MB while 22.04 is 28.17MB so the real deciding factor for the size seems to be all the dependencies.

[eugeneteoh]

Clearing python cache and icons sounds like it's worth it though!

Will do this.

I think the more interesting thing before we start comparing image sizes is seeing what dependencies are needed/not needed anymore as currently, it looks like most of the test builds failed. Maybe I'm reading it wrong but according to dockerhub, Ubuntu 18.04 was 24.5MB while 22.04 is 28.17MB so the real deciding factor for the size seems to be all the dependencies.

Agreed. It is likely because 22.04 includes much newer versions of the dependencies, which makes the sizes bigger. But I can look into which dependencies are not needed anymore. Will also try to fix the CI.

[cloudymax]

I think the more interesting thing before we start comparing image sizes is seeing what dependencies are needed/not needed anymore as currently, it looks like most of the test builds failed. Maybe I'm reading it wrong but according to dockerhub, Ubuntu 18.04 was 24.5MB while 22.04 is 28.17MB so the real deciding factor for the size seems to be all the dependencies.

I'm not sure, but at least on desktop ubuntu 22.04, there was (maybe still is?) an issue where Unity Hub fails to launch due to Ubuntu 22.04 using an updated version of libssl that Unity Hub does not yet support, thus requiring an OLD version of ssl be installed:

• https://forum.unity.com/threads/workaround-for-libssl-issue-on-ubuntu-22-04.1271405/

  #---installing older version of SSL as workaround for Unity Hub failing to launch---
  wget http://security.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2_amd64.deb
  sudo dpkg -i libssl1.1_1.1.1f-1ubuntu2_amd64.deb
  rm libssl1.1_1.1.1f-1ubuntu2_amd64.deb

I moved all of my custom images back to 18.04 to make them compatible with the existing base images so i don't have a good way to test this right now

[eugeneteoh]

Clearing python cache and icons sounds like it's worth it though!

Will do this.

I think the more interesting thing before we start comparing image sizes is seeing what dependencies are needed/not needed anymore as currently, it looks like most of the test builds failed. Maybe I'm reading it wrong but according to dockerhub, Ubuntu 18.04 was 24.5MB while 22.04 is 28.17MB so the real deciding factor for the size seems to be all the dependencies.

Agreed. It is likely because 22.04 includes much newer versions of the dependencies, which makes the sizes bigger. But I can look into which dependencies are not needed anymore. Will also try to fix the CI.

Ok so I cleared the python cache and icons and removed build-essentials. Got the size down to 485M.

❯ docker inspect -f "{{ .Size }}" base | numfmt --to=si && docker history base
485M
IMAGE          CREATED          CREATED BY                                      SIZE      COMMENT
2f8ceb64b6f2   20 seconds ago   ENV UNITY_PATH=/opt/unity                       0B        buildkit.dockerfile.v0
<missing>      20 seconds ago   RUN /bin/sh -c echo "576562626572264761624c6…   33B       buildkit.dockerfile.v0
<missing>      21 seconds ago   RUN /bin/sh -c /bin/echo -e 'pcm.!default {\…   52B       buildkit.dockerfile.v0
<missing>      21 seconds ago   RUN /bin/sh -c apt-get -q update  && apt-get…   0B        buildkit.dockerfile.v0
<missing>      17 minutes ago   RUN /bin/sh -c apt-get -q update  && apt-get…   82MB      buildkit.dockerfile.v0
<missing>      17 minutes ago   RUN /bin/sh -c apt-get -q update  && apt-get…   324MB     buildkit.dockerfile.v0
<missing>      2 days ago       RUN /bin/sh -c echo 'debconf debconf/fronten…   993kB     buildkit.dockerfile.v0
<missing>      2 days ago       ENV LC_ALL=C.UTF-8                              0B        buildkit.dockerfile.v0
<missing>      2 days ago       ENV LANG=C.UTF-8                                0B        buildkit.dockerfile.v0
<missing>      12 days ago      /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B        
<missing>      12 days ago      /bin/sh -c #(nop) ADD file:140fb5108b4a2861b…   77.8MB    
<missing>      12 days ago      /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B        
<missing>      12 days ago      /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B        
<missing>      12 days ago      /bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH     0B        
<missing>      12 days ago      /bin/sh -c #(nop)  ARG RELEASE                  0B        

Turns out build-essentials is the one that's huge.

I won't touch the other dependencies for now as I'm not familiar with what they're for.

[eugeneteoh]

Quick question, what's the best way to replicate the CI locally? I'm trying to fix the failing builds.

[AndrewKahr]

Local runs are usually a pain. If you really want to do it, you'll need to copy the dist folder from Unity builder then copy the docker run command from a GitHub action run and replace all the paeans as needed. Usually I'd use an env file for that. I'd recommend just letting GitHub actions run the builds as that'll require no setup and runs on a matrix of all versions. It should be running in your repo on every commit if I'm not mistaken. If it's not, you might consider tweaking the workflow to run.

[eugeneteoh]

Sounds too complicated... I'll stick to gh actions. But it needs approval for every run.

[AndrewKahr]

It shouldn't require approvals on your own fork. Does your fork not have actions enabled?

[eugeneteoh]

Didn't think of that... Thanks!

[AndrewKahr]

More specifically this is the workflow you need to enable: https://github.com/eugeneteoh/docker/blob/main/.github/workflows/test.yml

Not sure why it's not running, maybe simplify the run condition to just push and comment out the additional path conditions in case it doesn't like that for some reason.

[eugeneteoh]

Should be running now, it just wasn't enabled.

[eugeneteoh]

Hi all, I fixed most of the builds. Only ones that have failed are 2021.1.28f1-webgl (timed out), 2021.1.28f1-android (failed to install). Can you review again?

[AndrewKahr]

Looks like a step failed during hub installation based on the most recent run.

[eugeneteoh]

All jobs have passed after re-running!

New base image size:

❯ docker inspect -f "{{ .Size }}" base | numfmt --to=si
474M

[webbertakken]

Agreed.

This'll need to be a major bump. Then people can use ˋcustomImageˋ until we rollout the next major version of builder (which includes a few other breaking changes, including the CLI).

[AndrewKahr]

Should we hold off on merging until builder is ready then? I suspect we will get a lot of questions/complaints if users have to start using custom image for new Unity versions.

[webbertakken]

Sure, let's do that.

For context: Whatever is merged to main also publishes the new versions of Unity Editor. So after we merge this we do also need to update unity-builder and unity-test-runner, as they use image version 1.*, and new Editor versions would no longer be added to that.

An alternative is to have the build pipeline checkout a tag. But that comes with other challenges as well, as we'd have to make a tag for every change to be included in the newest versions, which means many more TBs of images - at which point we're going to have to start removing older images from Dockerhub.

[eugeneteoh]

Hi guys, any updates for this? Anything I can help to get this merged?

[cloudymax]

Hi guys, any updates for this? Anything I can help to get this merged?

I'm not sure... @webber @AndrewKahr is there a open issue on builder that needs help getting over the line? I'm happy to lend a hand if needed

[webbertakken]

We've worked ourselves into a little bit of a pickle.

Basically we have a few breaking changes, and we said the CLI would be part of the next release too. The CLI isn't yet finished. We did make the decision to move away from Deno and back into Node.

This PR itself is also a breaking change as we'd like to do a major bump for it, which won't automatically work in builder as it's using the latest ^0 image. (Which the point of doing a major release for this PR).

The most forward way would be to finish the Unity-builder part of the CLI, and use it in the next release of unity-builder.

[eugeneteoh]

I think a lot of people (at least for my use case) use the docker images outside of the Unity builder. IMO it makes sense to first have a major release for the docker images, then slowly adopt the major release to other tools like the Unity builder & cli.

[webbertakken]

We could do that, but it would also mean that we can no longer make changes to the older version of the images (because we only have a single release branch: main and no next branch), which are still in use for thousands of projects. It's quite risky if you ask me.

Reference: Unity builder has 3,5k projects using it. That does not include private projects (that we can't see at all) and other VCS services like GitLab.

[eugeneteoh]

Can we set up a v2 release branch?

[webbertakken]

It's possible. However, that would require adopting the versioning backend build scheduling.

It would be much lighter to just finish the builder part of the CLI. Not to mention the amount of extra work for maintainers and increased risk of Dockerhub becoming unhappy from us uploading more version. If I remember correctly, each version in this repo accounts for ~4TB in images.

[webbertakken]

In the meantime, if just one person makes forks the repo and publishes a single Unity Editor version (or all the newest ones) then that would effectively resolve anyone who is blocked here.

[webbertakken]

Still discussing our options among maintainers. We might go with a double version bump and simply skip a version in the docs (because of potentially rapid major bumps and needless duplication of all docs)

[webbertakken]

We've agreed to move forward with a unity-builder major bump that will include the change to the docker image major bump.

That means

• this can be merged and bumped.
• builder needs a PR with the docker version bump
• once that's also merged we can bump builder (and test-runner etc)
• docs need to be bumped to v3

Then later for the CLI

• we'll do another version bump for builder (and test runner etc)
• docs v3 turns into v4 (v3 does not persist as a separate version)

That way it should be manageable enough.

[eugeneteoh]

Thanks, this is super helpful! Keep up the good work!

[webbertakken]

Here's the PR that will be needed before bumping major. https://github.com/game-ci/unity-builder/pull/556. Before we merge that, we need the images to be published for all editor versions.

You can follow the progress in our Discord channel #new-versions and on the documentation website under docker image versions.