Python 3.9 rebuild post mortem

[lazka]

• staging download every 30 minutes created a lot of traffic
• fetching all release assets every time didn't scale well with 3500 files
• having more than 2 jobs per arch made us run into the api rate limit, 3/4 jobs per arch would have made things faster. This was mainly due to iterating assets.
• the regex filename based selection for packages to rebuild wasn't very exact. Basing it on actual file content would make it better.
• PKGBUILD parsing takes too long for CI

[jeremyd2019]

I don't know if it's appropriate here or not, but I've found on raspberry pi invoking repo-add for each package seems quite slow. I believe it would be faster to invoke repo-add for many packages at once. It would probably need some xargs-style logic to keep to a maximum command line length in a degenerate case with tons of packages though :wink:

[lazka]

sounds good. Since it only adds packages already in the queue and only if they are a real dependency I think the amount should be quite limited anyway.

[lazka]

staging download every 30 minutes created a lot of traffic

this is now fixed, it only downloads when things change now. -> https://github.com/msys2/msys2-main-server/commit/28363c4107aad5543e73f4c02f9ff2aeb0742a8a

[lazka]

I don't know if it's appropriate here or not, but I've found on raspberry pi invoking repo-add for each package seems quite slow. I believe it would be faster to invoke repo-add for many packages at once. It would probably need some xargs-style logic to keep to a maximum command line length in a degenerate case with tons of packages though wink

Improved in 842072fe5530ea26588153e4e1e02df56d8df71b and 10fdc3ec57d6577a53f65261a4ff37c3c4a0c901

[lazka]

I've split up the upload release tag for each build type, which should help a bit with reducing API calls when there are many assets. (the code will create the releases automatically when needed btw)

[lazka]

PKGBUILD parsing takes too long for CI

while hacky, this should improve it by a factor of 3-4: https://github.com/msys2/MINGW-packages/commit/22ad01c8ac826951541e5a86f31e356dcbed1277

[Biswa96]

Wouldn't removing the lint_pkgbuild function allow wrong PKGBUILD file?

[lazka]

yes, but ideally that would be caught in CI first when building the package.

[lazka]

the regex filename based selection for packages to rebuild wasn't very exact. Basing it on actual file content would make it better.

This is now solved by looking for DLL uses like here: https://github.com/lazka/package-grokker/runs/3578861455?check_suite_focus=true in combination with grepping for *.pyc and python3.9/site-packages

[lazka]

I've added a braindump for the next rebuild here: https://www.msys2.org/dev/python/

I think with all those improvements we can close this, and re-evaluate on the next rebuild.

[jeremyd2019]

the regex filename based selection for packages to rebuild wasn't very exact. Basing it on actual file content would make it better.

This is now solved by looking for DLL uses like here: https://github.com/lazka/package-grokker/runs/3578861455?check_suite_focus=true in combination with grepping for *.pyc and python3.9/site-packages

This made me decide to improve that script for the use case of finding DLL dependents of a scripting language. (jeremyd2019/package-grokker#7). For perl, this reduced the runtime of the script from 17 minutes to 3, with also probably a good reduction of packages downloaded to check.