[RFC] How about providing one (or more) "Stable Baseline Points"?

[softworkz]

The Problem

• Every two of few weeks, we're seeing the build getting broken
  This has consequences which are unfortunate for some and severy for other MABS users . Each time, the same things are happening, users are getting desperate, start investigating and start trying for find workarounds, which is in turn consuming time and satisfaction.
• When you are building for development...
  ...you would in most cases want to build something where everything the same as in your previous build besides the changes you made
• When you are building for testing...
  ...you typically want a build for comparison which only those changes you want to test - be it in ffmpeg, MPV or a specific depencency library
• When you are building for production...
  ...you may want a totally fresh build with all dependencies updated - every once in a while, but everywhere in-between, you want to have control and fully be aware about what has changed from one build to another

Strategy

I'm not quite sure about the strategy that MABS is using to choose and build dependencies. I can follow up to the idea where selection is based on the premise "latest must be greatest". That might be arguable but it's surely  not uncommon. But what does latest/greatest actually mean? Arch Linux for example is well known for their latest/greatest adoption and I know the maintainer for their ffmpeg builds, so I can tell that what they do not do is to pull the HEAD revisions from all dependencies at the time of build and then wait and see what happens. They are still making choices on released versions or specific tags of dependency libs.

I'm wondering how it could even be valid to arbitrarily combine libraries at their most recent HEAD commit at an arbitrary point in time? When libA of version 1.1 is compatible with version 2.2 or libB - does that mean that libA @2months-after-v1.1 is compatible with libB @2months-after-v2.2? I don't think so - it's unlikely that libA development is pulling in changes from libB on a daily basis to ensure perpetual compatibility, and even when they would do, there would still be a period of time between the publishing of libB's changes and the adaption being published for libA.

Practically, a stable and supported dependency version of one lib will in most cases be significantly older than the depending lib, for which it may take time to adapt and validate compatibility. The conclusion from this is when taking the latest state of all dependencies at the same point in time isn't a strategy that makes it more likely that they would all be compatible - instead it makes it less likely.

As an example, we can look at shaderc: It includes a DEPS file which tells exactly which revisions of its dependencies are required. MABS ignores that and just downloads the latest HEAD from all dependency repos - which is causing the build to fail just once another time (as of now).

Currently there are more than 100 dependency libraries that are pulled by the latest HEAD revision. A single commit in one of those 100 libraries can cause the MABS build to fail.
This is essentially a lottery game with nothing to win but where you often lose.

Proposal

With the introduction of the media-suite_deps.sh file, it is now possible to define a set of specific dependency versions. This allows everyone to figure out their own set of dependency versions, but this is an insane task and hardly feasible for many MABS users.

That's why it seems to make sense to include such a stable set of dependencies in the suite directly. It could be delivered as an alternate *_deps.sh file - for example: media-suite_deps_2023.1.sh

The media_autobuild_suite.bat could then provide a selection:

Dependency Version Set

1. LATEST
2. Stable 2023.1

A "stable set" should be immutable, which means that once it's out, it shouldn't be changed anymore (as far as possible) to ensure deterministic and reproducible builds. After some time, a new and additional "stable set" could be added.

Open Questions

• How can the MSYS2/MinGW packages be pinned to specific versions?
• Will everything get rebuilt properly when versions are downgraded?
  (like when selecting a stable set after doing a build with LATEST)

---

Please let me know what you think about this and share your thoughts.

Thanks

[softworkz]

PS: I'm not coming with empty hands - I got half of the dependencies pinned already and tested :smile:

[1480c1]

Personally would be fine with this, but I personally would not have enough time, so I would not be able to put forth the effort to validate tags etc. it would need to be something that is supported either by you or case by case

[softworkz]

I am willing to support this (because I have to anyway), though my goals are going for long-term stability rather than updating each time when a lib has a new version. 

In the past 2-3 years, I had achieved this by commenting out all auto-updating and keeping the same MSYS2 installation with local32/64 and build over the whole time. After all, it's about building ffmpeg and for 90-95% of the libs it doesn't matter the slightest bit whether the latest or a 2 years old version is being used.  Essentially, it burns down to a handful of libs that really needed to be updated for one or another reason - but that was decided on a case-by-case basis and done only when it was really needed (and not just because a new version existed).

Now - everybody may have different needs and decisions and that's why I'm seeing the primary objective not in curating a dependency selection that pleases as many users and use cases as possible - it's rather about providing a stable and reliable baseline - to which you can always return. And to which you can switch when the default built is broken once another time.
And which everybody can use as a starting point for one's own dependency customization.

So, what I would have in mind is:

• To establish a baseline dependency set that can be selected for building
• To setup some CI that regularly verifies that it's still valid and working
• Only make an adjustment in case something get broken
  (from experience, I know that when you don't change much, it will hold up for quite a while)
• After 1-2 years, work out a totally new baseline where everything is refreshed

That's what I would contribute. Though, I'm flexible - when there's community interest to participate and contribute for more frequent updates or something, that would be fine of course.

TL>DR:

The key word is "base line". Something that's always working and that can serve as a reference. Everybody will be able to make their own choices, but with that base line, they won't need to make 50-100 choices but only the few they care about.

[1480c1]

Sounds good to me, I'd be fine with using git tags etc as well if wanted

[softworkz]

Tags would be nicer when they'd be working, currently the deps file looks like that:

# tag=n11.0.10.2 => revision 27b5db61c9eb4b98d3f565cecf8713b4ecabd990
SOURCE_REPO_FFNVCODEC=https://git.videolan.org/git/ffmpeg/nv-codec-headers.git#revision=27b5db61c9eb4b98d3f565cecf8713b4ecabd990

# tag=1.4.2 => revision b32e5cbf9818ca23dd22aaa75522042c16ea7d17
SOURCE_REPO_FLAC=https://github.com/xiph/flac.git#revision=b32e5cbf9818ca23dd22aaa75522042c16ea7d17

SOURCE_REPO_FLITE=https://github.com/festvox/flite.git

# tag=2.14.0 => revision c45e09df1ef235d653d56aef05012f6a3cc57979
SOURCE_REPO_FONTCONFIG=https://github.com/freedesktop/fontconfig.git#revision=c45e09df1ef235d653d56aef05012f6a3cc57979

[1480c1]

Oh, I meant git tagging this repo so specific version pins are more visible

[softworkz]

Ah, alright, got you!

I have another good news. A while ago I had set up https://github.com/ffstaging/FFmpeg - it's a repo that you can use to make pull requests which are automatically sent to the ffmpeg mailing list.

This is backed by Azure DevOps jobs for which I have created an organization that got waived by MS for extended use of build agent runs. I thought this would only be limited to one or two parallel jobs (which are required for the PR-to-ML functionality). 
But I've just seen that the organization has 10 parallel jobs available - that means I should be able to use this for doing CI builds.

We'd just need to figure out how to connect these two things together.

[softworkz]

Then we could do two CI builds on each commit to this repo: One for default (with all latest deps and latest ffmpeg) and one with the stable baseline.

[GyanD]

As a 'Stable Baseline Point' recedes in time, it will get out of date. Instead, it would be useful to add automation to record one or more of a set of good runs i.e. the permutation of repo and toolchain commits & packages that resulted in a successful build.

We can then supply a argument (or a ini file) to use one of those good runs if needed.

[GyanD]

Another step is to cache existing artifacts, and then 1) delete/overwrite upon success of current build, or 2) restore upon failure and continue.

[softworkz]

As a 'Stable Baseline Point' recedes in time, it will get out of date. Instead,

This is what I'm going to curate, though. I will update this only every 12-24 months or in case of an exceptional reason which is causing it to fail.

Instead, it would be useful to add automation to record one or more of a set of good runs i.e. the permutation of repo and toolchain commits & packages that resulted in a successful build.

We can have additional dependency sets. As of the current proposal there will be:

1. All HEAD revisions (at the time of building)
2. Long-Term Stable Baseline 2023_02

It will be surely fine to have an additional set of dependencies, e.g. "Last Known Good" - if somebody would curate it.

We can then supply a argument (or a ini file) to use one of those good runs if needed.

That's what the media-suite_deps.sh file is for. I will add a media-suite_deps_lts-2304.sh and when somebody want to maintain a newer one, this will be fine.

to record one or more of a set of good runs

If somebody would contribute a script that records all the dependency versions being used during a CI build and writes them to a file, then we can add that file to the artifacts of the build output.

[softworkz]

Another step is to cache existing artifacts, and then 1) delete/overwrite upon success of current build, or 2) restore upon failure and continue.

That kind of messing around is out of the scope of CI builds. The CI builds are for ensuring that everything builds fine in a single run.

I don't even know whether there's an intention of the project owners to provide readily built binaries at all. So maybe the CI builds won't even publish the generated binaries. It will be their decision. Personally, I'm fine with either way.

[softworkz]

Another step is to cache existing artifacts, and then 1) delete/overwrite upon success of current build, or 2) restore upon failure and continue.

That kind of messing around is out of the scope of CI builds

But if you want to set up your own cloud (Azure devops) builds for testing with different dependency versions (or other variations), I can be of assistance, especially on how to set up caching to avoid rebuilding from scratch each time.

This is also for which the PR Is meant that I have submitted: Early Exit: Provide a way to cleanly exit the build at defined points

[1480c1]

I don't even know whether there's an intention of the project owners to provide readily built binaries at all. So maybe the CI builds won't even publish the generated binaries. It will be their decision. Personally, I'm fine with either way.

afaik, there wasn't really any intention one way or the other, and I guess I would be fine with doing so, as long as it's not --enable-nonfree. But at that point, it might overlap with Gyan's and BtbN's builds. Although I have a feeling it's related to the do_uninstall call we make

[GyanD]

That suggestion of cache and restore is for use during end user operation.

[softworkz]

With #2369, it will be possible to specify an environment variable in a build pipeline which determines how far the compilation should go. You set it to EE1 first and when it's successful, you can write this to a file (which will be included in the cache) and  the pipeline can trigger itself again. In the next run, the pipeline reads the file and sets the env variable to the next exit point (e.g. EE2).

The exit points are chosen in a way that when you build only 32 or 64bit (but not both), then each iteration will complete within less than 60min. This way, it should be possible to run it even on cloud build services which have a 60 minute limit.

[1480c1]

That suggestion of cache and restore is for use during end user operation.

I guess something could be done so it goes

do_vcs
do_cache files "${_check[@]}" # tars the known results into a single ball, runs do_uninstall
build || { do_uncache; exit; } # untars the tarball, exits etc
do_checkIfExist # deletes the tarball

[softworkz]

Isn't that just a kind of "continue on error"? (when you don't uninstall before building)

I'm not sure whether you get very far that way, because an individual library that doesn't build is not among the primary concerns. It's more about compatible combinations of libraries, like you're building libA and only a dozen steps later, you may find out that it doesn't work with libB.

The other point is that it's a doomed path to build again on top of an earlier build, or let's say at least "everything but clean".
As mentioned, I had done that before, but even though it worked okay for most of the time, in the end it was totally impossible to replicate that somewhere else. Too many things had happened and certain things were working there but no in a fresh build and others vice versa.

For a build that is reproducible, you need to have a clearly defined setup where the build runs from scratch and ends successfully, otherwise it's not worthwhile to take any effort into that direction - at least that's how I see it. 
(I'm talking about production builds of course - it's not something which everybody will need)

The PR with the exit points helps to work towards that goal in a way that you can build iteratively from one point to the next. In case of success, the current results are cached and you can start again with the cached results, heading to achieve the next point. 
In case of failure, nothing is cached and you can retry (with changes) starting from the last successful cache result. (in case of my DevOps setup, all successful intermediate cached results remain available, so you could also go back one or more steps.

Once you got beyond the finish line, you need to run at least once again to make sure that your intermediate changes don't screw up something at earlier points. Then you should have a really good build setup..

[1480c1]

I guess I should have added another command to make it a bit more obvious, I was meaning to combine the do_uncache with the compilation_fail function

[1480c1]

I'm assuming the idea is that even if a package is broken, the end user still has the executables still available, even if it's older, e.g. an older FFmpeg still available for daily use even though the ffmpeg build failed

[GyanD]

For a build that is reproducible, you need to have a clearly defined setup where the build runs from scratch and ends successfully, otherwise it's not worthwhile to take any effort into that direction - at least that's how I see it. (I'm talking about production builds of course - it's not something which everybody will need)

The primary purpose of this repo originally was (is ?) to build ffmpeg git master which is under active development. People wanting 'stable reproducible production builds' won't be using a suite that fetches and builds remote HEAD of every repo on every run. The end goal is utiity - having a ffmpeg binary of the latest git HEAD. Ideally, that includes the latest dependencies as well, but there are dozens of those, and a build failure of any one of those aborts the process. It's very unlikely, on average, that a user needed the latest build of that one particular library that failed. It's just a stage on the road towards the end goal. Caching and using a slightly older artifact keeps the process going. If a user did need the latest build for that library, there can be another INI which the script checks to see if build failure for that dep is a showstopper.