What is the bus factor for MINGW-packages?

[ArekPiekarz]

If you're not familiar with this term, the bus factor describes in a tongue-in-cheek way how many people have to be run over by a bus to bring a project to a halt.

I have read that all packages here are built manually by one person in his spare time . This doesn't seem like a scalable solution to me for the long term, not only because of a limited capacity, but also a potential of burnout. Have there been considerations about automating the process? If for some reason it cannot be done, are there any plans to extend the group of trusted "builders" for the project, to achieve better redundancy and speed?

[Alexpux]

@Fazer2, If anyone wants to automate the process of building packages, let him present his solution. I have no time and resources to do it myself. I want to see how will be solve problems with fixing "bad" packages that users push very often. Also want to see how to rebuild all needed packages after update some package where dll name will be changed or API incompatibility. I can move all work about packages creating to someone who have more time and resources to do it.

[Alexpux]

@Fazer2 , building packages under Windows take lot of time. For example, Qt5 building only when you have all dependencies installed take on my work PC about 13 hours.

[peterbud]

We had a discussion on this earlier, but no conclusion ultimately. The need is there, but its a complex problem domain. Nevertheless I#d be happy to contribute and continue this discussion.

[wirx6]

Todo board can be created in Projects tab. People(trusted, we don't want nsa to include their addons) who want to help can just build requested packages and upload.

[lazka]

I've asked appveyor yesterday to increase the max build time from 60 to 90 minutes and they said they have changed it now. I'd guess, if pull requests don't get too big, that would cover many packages.

[peterbud]

• Are the artifacts from an appveyor build are used in the repo?
• Or are those packages are re-built offline and then uploaded to the repo?

[Alexpux]

All packages builded and signed by me on my computer. Nothing used from appveyour

[mingwandroid]

IMHO, if you want to try to move towards more automation (which isn't a bad thing if done right) you should tackle this in stages, first by creating tooling (but also ensuring Alexey is on-board with using the tools of they are useful to him; hopefully they would be).

The problem of DLL names changing for example could be checked automatically. ntldd will tell you what is used. But beyond that, you need to also check that symbols needed are still present and correct. On way would be to either employ Dependency Walker (if it has a non GUI mode) or add better Windows support to diff-o-scope (from the reproducible builds project). The next stage would involve something more complicated at the compiler/linker level to ensure function definitions (and structure layouts) have not changed (a GCC plugin would be most appropriate for this unless we switch to clang).

This tooling could be and tailored to handling the MSYS2 repositories (perhaps linking to libalpm and optionally calling it after builds have been completed).

Once that's in place and proven to be catching lots of issues and saving lots of time it would be interesting for someone to try to build CI. Currently I regard the existing solutions for CI inadequate though (but this is an area we're looking at at my day job).

What do you think Alexey? Could any of this be useful to you?

[elieux]

The problem of DLL names changing [...] check that symbols needed are still present and correct

Last year I heard a presentation about libabigail being successfully used in Fedora packaging to detect such mistakes. Maybe it could work for us as well.

CI

If we go with the trusted package builders concept, we don't necessarily need a public CI capable of building every package. We could also try private build slaves, if someone is able to provide one (I could try to get something approved at work for example).

[elieux]

I'll also leave a link to our "to do" item regarding package building I wrote after discussing it long ago with Alexey: https://github.com/msys2/msys2/wiki/Devtopics#off-load-package-building-and-uploading

[emmenlau]

On 22.06.2017 14:58, David Macek wrote:

If we go with the /trusted package builders/ concept, we don't necessarily need a public CI capable of building every package. We could also try private build slaves, if someone is able to provide one (I could try to get something approved at work for example).

I also have a build bot available that I use for my MSYS2 CI.

Cheers,

Mario

[lazka]

On the topic of automation, here are some scripts I've been using to automate msys2 related things. I've cleaned them up now and put them in a repo:

https://github.com/lazka/msys2-helpers

• msys2-check-updates.py: This compares either all installed packages or all available packages against the Arch repos and reports out of date ones.
• msys2-dll-check.py: This goes through all installed executables/libraries and checks if the required dlls are available.

Maybe someone finds them useful.

[Ede123]

Nice! msys2-dll-check.py output on my installation if somebody want to check it out:

MISSING: C:/msys64/mingw64/bin/cmake-gui.exe -> qt5core.dll
MISSING: C:/msys64/mingw64/bin/cmake-gui.exe -> qt5gui.dll
MISSING: C:/msys64/mingw64/bin/cmake-gui.exe -> qt5widgets.dll
MISSING: C:/msys64/mingw64/bin/doxywizard.exe -> qt5core.dll
MISSING: C:/msys64/mingw64/bin/doxywizard.exe -> qt5gui.dll
MISSING: C:/msys64/mingw64/bin/doxywizard.exe -> qt5widgets.dll
MISSING: C:/msys64/mingw64/bin/doxywizard.exe -> qt5xml.dll
MISSING: C:/msys64/mingw64/bin/libpoppler-qt5-1.dll -> qt5core.dll
MISSING: C:/msys64/mingw64/bin/libpoppler-qt5-1.dll -> qt5gui.dll
MISSING: C:/msys64/mingw64/bin/libpoppler-qt5-1.dll -> qt5xml.dll
MISSING: C:/msys64/mingw64/lib/bin/libenchant_zemberek.dll -> libdbus-glib-1-2.dll

I guess that script can also be easily adjusted to check dependencies for other projects (i.e. to determine what to package and/or if everything needed is packaged.

[lazka]

There is a good chance that the reported missing things are optional dependencies and to be expected. Feel free to open an issue with ideas/feedback at the repo as this is getting a bit too off topic I guess.

[lazka]

Here is another one:

https://github.com/lazka/msys2-helpers/blob/master/msys2-check-builds.py

Passing it the path to the MINGW or MSYS2 repo it will compare the version numbers of the PKGBUILD files to the repo and report packages which need to be build/added. It's quite slow due to makepkg-mingw taking 4+ seconds to parse one PKGBUILD file, so this uses some hacky caching file which is also in the repo.

Here is the current output:

• MINGW-packages: https://bpaste.net/raw/e716b3d3c8d6
• MSYS2-packages: https://bpaste.net/raw/92d2e0a954b2

[mati865]

@lazka I think the last tool is similar to https://github.com/renatosilva/pactoys/tree/master/source/updpkgver

[lazka]

@mati865 I didn't know about pactoys, thanks. The last one is something different than updpkgver, it compares the package version in GIT with the one from the repo and gives you packages which where updated but not build/published yet. Basically a todo list for @Alexpux ;)

updpkgver is similar to msys2-check-updates.py as far as I can see, but the later checks against the Arch repos instead of trying to bump the version.

[mati865]

@lazka it must have been different tool then.

[lazka]

(Sorry for the noise) Here is another one reporting the PKGBUILD files which need to be build in the right order according to their dependencies.

``` $ python -u msys2-check-builds.py ../M --buildorder Searching for PKGBUILD files in C:/msys64/home/xy/M Found 1059 PKGBUILD files C:/msys64/home/xy/M/mingw-w64-http-parser/PKGBUILD C:/msys64/home/xy/M/mingw-w64-lfcbase/PKGBUILD C:/msys64/home/xy/M/mingw-w64-cego/PKGBUILD C:/msys64/home/xy/M/mingw-w64-mpc/PKGBUILD C:/msys64/home/xy/M/mingw-w64-xapian-core/PKGBUILD C:/msys64/home/xy/M/mingw-w64-db/PKGBUILD C:/msys64/home/xy/M/mingw-w64-live-media/PKGBUILD C:/msys64/home/xy/M/mingw-w64-cppunit/PKGBUILD C:/msys64/home/xy/M/mingw-w64-lcov/PKGBUILD C:/msys64/home/xy/M/mingw-w64-ocaml-camlp4/PKGBUILD C:/msys64/home/xy/M/mingw-w64-openblas/PKGBUILD C:/msys64/home/xy/M/mingw-w64-diffutils/PKGBUILD C:/msys64/home/xy/M/mingw-w64-nss/PKGBUILD C:/msys64/home/xy/M/mingw-w64-python-bsddb3/PKGBUILD C:/msys64/home/xy/M/mingw-w64-quantlib/PKGBUILD C:/msys64/home/xy/M/mingw-w64-aria2/PKGBUILD C:/msys64/home/xy/M/mingw-w64-rust-git/PKGBUILD C:/msys64/home/xy/M/mingw-w64-mesa/PKGBUILD C:/msys64/home/xy/M/mingw-w64-creduce/PKGBUILD C:/msys64/home/xy/M/mingw-w64-rust/PKGBUILD C:/msys64/home/xy/M/mingw-w64-glib2/PKGBUILD C:/msys64/home/xy/M/mingw-w64-dbus/PKGBUILD C:/msys64/home/xy/M/mingw-w64-crypto++-git/PKGBUILD C:/msys64/home/xy/M/mingw-w64-ag/PKGBUILD C:/msys64/home/xy/M/mingw-w64-innoextract/PKGBUILD C:/msys64/home/xy/M/mingw-w64-shiboken-qt4/PKGBUILD C:/msys64/home/xy/M/mingw-w64-librest/PKGBUILD C:/msys64/home/xy/M/mingw-w64-libvirt-glib/PKGBUILD C:/msys64/home/xy/M/mingw-w64-qemu/PKGBUILD C:/msys64/home/xy/M/mingw-w64-pyqt4/PKGBUILD C:/msys64/home/xy/M/mingw-w64-pyside-qt4/PKGBUILD C:/msys64/home/xy/M/mingw-w64-suitesparse/PKGBUILD C:/msys64/home/xy/M/mingw-w64-pyside-tools-qt4/PKGBUILD C:/msys64/home/xy/M/mingw-w64-webkitgtk2/PKGBUILD C:/msys64/home/xy/M/mingw-w64-farstream/PKGBUILD C:/msys64/home/xy/M/mingw-w64-gtksourceviewmm3/PKGBUILD C:/msys64/home/xy/M/mingw-w64-latexila/PKGBUILD C:/msys64/home/xy/M/mingw-w64-clutter-gst/PKGBUILD C:/msys64/home/xy/M/mingw-w64-libc++/PKGBUILD C:/msys64/home/xy/M/mingw-w64-gedit-plugins/PKGBUILD C:/msys64/home/xy/M/mingw-w64-libc++abi/PKGBUILD C:/msys64/home/xy/M/mingw-w64-libdvdread/PKGBUILD C:/msys64/home/xy/M/mingw-w64-virt-viewer/PKGBUILD C:/msys64/home/xy/M/mingw-w64-quassel/PKGBUILD C:/msys64/home/xy/M/mingw-w64-gst-rtsp-server-git/PKGBUILD ```

[lazka]

And a final one:

m2h.py build <path-to-mingw-packages-repo> <target-directory>

will see which packages need to be build, build/install them in the right order and put the result in target-directory, including logs. In case a build fails, any package in the build queue depending on it will be skipped.

Theoretically, running the above on every commit should automate all building. I don't know how package deployment works, so that's still missing.

[holgern]

I found also some github repos which try to automically build packages for archlinux. Maybe they can be adopted to msys64? https://github.com/jantman/archautorepo https://github.com/falconindy/asp https://github.com/localnet/aurci https://github.com/seblu/aurbot https://github.com/AladW/aurutils

[lazka]

I've made a small web interface for the msys2 repos: https://quodlibet.duckdns.org/msys2/

[lazka]

New page which shows the difference between the GIT repos and the pacman repo (excluding VCS packages): https://quodlibet.duckdns.org/msys2/queue

This gets updated once per day through an appveyor run.

[lazka]

@Alexpux I assume you already know, but the main mirror is down since yesterday evening (CET)

[Alexpux]

@lazka use sourceforge mirror then. I dont have access to server

[lazka]

Ok, pacman falls back anyway after the timeout :)

Is there any way I or we can help?

[Alexpux]

@lazka I'm not at IRC for a long time, so need to ask Diablo to verify what is wrong with server

[lazka]

@Alexpux see http://quodlibet.duckdns.org/msys2/irc/msys2_2018-05-17.log.html#t00:30:54-2-3-4-5-6-7-8

[Alexpux]

@lazka tell him to run server I will try to solve problem

[Diablo-D3]

@Alexpux You should come back to IRC, we miss you. I'm pretty sure you got stung by this: https://thehackernews.com/2018/02/memcached-amplification-ddos.html

[Alexpux]

@Diablo-D3 I will when get some time.

[Diablo-D3]

@Alexpux Should be now up and running. However, since I don't have access to the server, you'll have to close the memcached hole.

[Alexpux]

@Diablo-D3 I hope that issue is fixed now