Improve devcontainer build speed

[corona10]

@brettcannon The 35-minute release period is pretty painful. We may need to find out whether we can reduce the build time. https://github.com/python/cpython-devcontainers/actions/runs/11038928730

[AA-Turner]

The below is extracted from the logs, the main dnf step takes 2,038 seconds (34 minutes).

Notably, no output is produced for ~12 minutes (730 seconds). "Fedora 40 - aarch64 - Updates" then takes another 6 minutes (370 seconds). In the second builddep command, another six minutes (350 seconds) elapse from "enabling fedora-source repository" to the first "Package [...] is already installed.".

cc @hroncok for any advice (as author of https://github.com/python/cpython/pull/103283)

dnf install ...:

[linux/arm64 2/4] RUN dnf -y --nodocs --setopt=install_weak_deps=False install /usr/bin/{blurb,clang,curl,git,ln,tar,xz} 'dnf-command(builddep)' &&     dnf -y --nodocs --setopt=install_weak_deps=False builddep python3 &&     dnf -y clean all
729.3 Fedora 40 openh264 (From Cisco) - aarch64       476  B/s | 2.1 kB     00:04    
785.2 Fedora 40 - aarch64 - Updates                   1.1 MB/s |  38 MB     00:33    
1156.0 Last metadata expiration check: 0:00:23 ago on Wed Sep 25 18:39:54 2024.
1241.3 Package [...] is already installed. [x4]
1250.7 Dependencies resolved.
1251.6 ================================================================================
1251.6  Package                     Arch     Version                    Repo      Size
1251.6 ================================================================================
1251.6 Installing:
...
1251.6 Transaction Summary
1251.6 ================================================================================
1251.6 Install  47 Packages
1251.6 
1251.7 Total download size: 140 M
1251.7 Installed size: 562 M
1251.7 Downloading Packages:
...
1267.9 --------------------------------------------------------------------------------
1267.9 Total                                           8.7 MB/s | 140 MB     00:16     
1293.7 Running transaction check
1295.7 Transaction check succeeded.
1295.7 Running transaction test
1305.3 Transaction test succeeded.
1305.3 Running transaction
1313.6   Preparing        :                                                        1/1 
1315.1   Installing       : systemd-libs-255.12-1.fc40.aarch64                    1/47 
...
1344.5   Running scriptlet: dnf-plugins-core-4.9.0-1.fc40.noarch                 47/47 
...
1348.7 Complete!

dnf builddep python3:

1358.6 enabling fedora-source repository
1358.6 enabling fedora-cisco-openh264-source repository
1358.6 enabling updates-source repository
1370.6 Fedora 40 - Source                              1.2 MB/s | 6.6 MB     00:05    
1433.9 Fedora 40 openh264 (From Cisco) - aarch64 - Sou  30  B/s | 134  B     00:04    
1452.2 Fedora 40 - Updates Source                      359 kB/s | 1.7 MB     00:04    
1544.5 Last metadata expiration check: 0:00:08 ago on Wed Sep 25 18:51:10 2024.
1708.0 Package [...] is already installed. [x12]
1722.1 Dependencies resolved.
1724.3 ================================================================================
1724.3  Package                    Arch    Version                       Repo     Size
1724.3 ================================================================================
1724.3 Installing:
...
1724.3 Transaction Summary
1724.3 ================================================================================
1724.3 Install  233 Packages
1724.3 Upgrade   11 Packages
1724.3 
1724.7 Total download size: 123 M
1724.8 Downloading Packages:
...
1746.9 --------------------------------------------------------------------------------
1746.9 Total                                           5.6 MB/s | 123 MB     00:21     
1861.1 Running transaction check
1867.8 Transaction check succeeded.
1867.8 Running transaction test
1893.0 Transaction test succeeded.
1893.0 Running transaction
1914.6   Preparing        :                                                        1/1 
1917.6   Upgrading        : zlib-ng-compat-2.1.7-2.fc40.aarch64                  1/255 
...
1989.6   Running scriptlet: expat-2.6.0-1.fc40.aarch64                         255/255 
...
2025.9 Complete!

dnf clean:

2036.9 37 files removed
DONE 2037.9s

[hroncok]

Notably, no output is produced for ~12 minutes (730 seconds). "Fedora 40 - aarch64 - Updates" then takes another 6 minutes (370 seconds). In the second builddep command, another six minutes (350 seconds) elapse from "enabling fedora-source repository" to the first "Package [...] is already installed.".

That's pretty bad, but I have no idea why would it take so long. Perhaps it's the emulation/virtualization that is there that is so slow?

[trallard]

I was having a quick nosey here 👋🏽 but looking at the Actions runs, the build is taking < 3mins and the 35 mins + was an outlier.

[corona10]

@trallard

FYI, 3 mins running releases were about Autoconf :) For the devcontainer it takes a long time, but thanks to your comment, I've written up a new PR for distinguishing which packages were released :)

Please take a look if you are interested :)

https://github.com/python/cpython-devcontainers/pull/16

[trallard]

Ah that makes sense. Will have a look over there and maybe do some digging on why this is taking so long then.

[corona10]

Last release by Brett took 28 mins. (36mins -> 28mins!!) https://github.com/python/cpython-devcontainers/actions/runs/11245912749

The PR by @hroncok enhanced the speed a lot!

[hroncok]

We have a container job at github actions that runs dnf on 4 architectures. The arm64 one (third in the screenshot) is regularly extremely slow: