search

rocker-org / rocker

R configurations for Docker
https://rocker-project.org
GNU General Public License v2.0
1.45k stars 273 forks source link

Installing unstable apt packages? #541

Closed jonkeane closed 6 months ago

jonkeane commented 6 months ago

First of all, thank you for maintaining this kind of infrastructure.

I noticed this on one of our Arrow (extended) CI builds that the new 4.3.3 container was having issues installing libcurl4-openssl-dev.

It appears that this dockerfile now installs most of the the R system packages with -t unstable which wasn't there in 4.3.2 as far as I can tell. Is this intentional?

4.3.3: https://github.com/rocker-org/rocker/blob/d373d03aa4d491216905eb24b18d0f56165dd43f/r-base/4.3.3/Dockerfile#L50

4.3.2: https://github.com/rocker-org/rocker/blob/d373d03aa4d491216905eb24b18d0f56165dd43f/r-base/4.3.2/Dockerfile#L50

In our case, it means that we will need to install libcurl4-openssl-dev with apt-get install -t unstable libcurl4-openssl-dev on builds with this image in order to have matching libcurl4 versions (or, in principle we could also downgrade libcurl4, but that's more drastic + need to reinstall a bunch of other things too). I can make the changes to identify when we are in this condition in our CI — but this might be a broader frustration others run into so wanted to confirm that it was intended with this version bump.

The original error that prompted me to investigate this

Here's the failure we are seeing for the record full logs: 

#14 7.270 distribution that some required packages have not yet been created
#14 7.270 or been moved out of Incoming.
#14 7.270 The following information may help to resolve the situation:
#14 7.270 
#14 7.270 The following packages have unmet dependencies:
#14 7.385  libcurl4-openssl-dev : Depends: libcurl4 (= 8.5.0-2) but 8.6.0-3 is to be installed
#14 7.391 E: Unable to correct problems, you have held broken packages.
#14 ERROR: process "/bin/sh -c /arrow/ci/scripts/r_docker_configure.sh" did not complete successfully: exit code: 100
------
 > importing cache manifest from apache/arrow-dev:r-library-r-base-latest:
------
eddelbuettel commented 6 months ago

Is this intentional?

It is. Takes a moment to explain so please bear with me.

(And if you scroll through the commit history [total aside: is there a grep like tool that would let us do that?] or my commits of this same Dockerfile to upstream docker-library where it becomes r-base without the leading rocker/ then you see that we have to do this whenever there is skew between testing and unstable. As quick guess I'd say may 1/4 to 1/3 of the time? Most often it applies only to r-base-* though.)

In short, I wanted to get this out "the day of" as I have done for Rocker for a decade (give or take) on each R update. But running an update of the unchanged Dockerfile ran into an "apt cannot fullfil this" as the build (in unstable) relied on some libraries that had not yet moved to testing. So back in the -t unstable went.

The chief cause right now is the (fairly large) 64-bit time_t transition. You may have noticed that a bunch of libraries now come in with a new suffix 't64'. From the very container and r-base-core binary:

Depends: zip, unzip, libpaper-utils, xdg-utils, libblas3 | libblas.so.3, libbz2-1.0, libc6 (>= 2.35), libcairo2 (>= 1.6.0), libcurl4 (>= 7.28.0), libglib2.0-0t64 (>= 2.12.0), libgomp1 (>= 4.9), libicu72 (>= 72.1~rc-1~), libjpeg62-turbo (>= 1.3.1), liblapack3 | liblapack.so.3, liblzma5 (>= 5.1.1alpha+20120614), libpango-1.0-0 (>= 1.18.0), libpangocairo-1.0-0 (>= 1.14.0), libpcre2-8-0 (>= 10.22), libpng16-16t64 (>= 1.6.2), libreadline8 (>= 6.0), libtcl8.6 (>= 8.6.0), libtiff6 (>= 4.0.3), libtirpc3 (>= 1.0.2), libtk8.6 (>= 8.6.0), libx11-6, libxt6, zlib1g (>= 1:1.1.4), ucf (>= 3.0), ca-certificates

And until all this morphes over into testing we would not have r-base version 4.3.3. So it's a trade off.

In the meantime you could pin the 4.3.2 container.

That being said, in some CI setups I look after (also for work) we tend to use Ubuntu LTS instead. It also gets the same package r-base-core (via @marutter and his launchpad.net based 'ports / rebuilds' that are mirrored at CRAN; 4.3.3 should be there by tomorrow) and I happen to be a big fan of the fact that I a) get the 22k CRAN binaries via r2u and b) can also stick r-universe binaries in as @opencpu builds Linux binaries for this very combination of Ubuntu LTS and current R.

Actually, as I write this, I realize you could also try to 'patch' the Dockerfile. Run it without -t unstable and for each library package that fails add just that library with its current unstable version (as I do for r-base-*). That should build. Do you want to try that? I could take a PR too.

eddelbuettel commented 6 months ago

Ok, I got curious and tried that but could not swing it by adding 'just' libpng16-16=1.6.43-3 and libpng16-16t64 from unstable to something otherwise held to testing.

The transition may be over in a few days and I can at least rebuild rocker/r-base then. I think upstream for docker-library likes it immutable but I can ask them.

jonkeane commented 6 months ago

Aaaah got it. It looks like we misunderstood the purpose / intended stability of r-base here. For our purposes we can easily switch these to something that is more stable and that's just fine by us.

And if you scroll through the commit history [total aside: is there a grep like tool that would let us do that?] or my commits of this same Dockerfile to upstream docker-library

Do you have a link to this upstream? I tried looking for it earlier in my debugging process and didn't find it, but I could have missed it. I did scroll through the commits here, but didn't find anything that looked similar. I don't know of a tool that would make this easier (though one certainly would be welcome!)

eddelbuettel commented 6 months ago

Well it usually is a) stable (but this week the testing/unstable mix is not quite that) :-/ and b) generally guaranteed to not change. It's just a bad week.

There is also rocker/r-ver:4.3.3 (oh now I see you found it). It uses a different stack, is that the basis for the 'by date' mapping and install R itself from source (rather than from .deb binary).

The upstream Docker repo (along with 'me' among the 30k+ commits is this: https://github.com/docker-library/official-images/commits/master/?author=eddelbuettel From memory I mention when unstable is in there.

And yes, I feel your pain. Debian testing is generally good, gives us new compilers etc 'soon enough' but there can be potholes. Debian stable is (in my view) generally too old and stale. Ubuntu LTS is a fair compromise.