Linux base build fails recent ubuntu (glib, etc)

ndevenish commented 7 years ago

I'm bootstrapping on a fresh Ubuntu 17.10 VM, gcc 7.2.0. The base installation fails on glib with:

CC libglib_2_0_la-gdate.lo
gdate.c: In function 'g_date_strftime':
gdate.c:2497:7: error: format not a string literal, format string not checked [-Werror=format-nonliteral]
tmplen = strftime (tmpbuf, tmpbufsize, locale_format, &tm);
^~~~~~
cc1: some warnings being treated as errors
Makefile:1386: recipe for target 'libglib_2_0_la-gdate.lo' failed

This seems to have been recognised in Februrary 2016: https://bugzilla.gnome.org/show_bug.cgi?id=761550 and fixed shortly afterwards in 2.47.6 (we bootstrap November 2015's 2.46.2).

Does anyone know what this dependency is fulfilling, and if there are any barriers to upgrading it?

ndevenish commented 7 years ago

Adding to this ticket for now;

fontconfig also fails; glibc 2.25 included some new macro specifications which broke fontconfig. These were fixed here - for 2.12.2, released in March this year. We're still using 2.11.1 from 2014. This upgrade isn't so simple - there seems to have been an introduced build-time dependency on gperf, and we do some other patching so that would probably need to be updated.

Anthchirp commented 6 years ago

just a note that glib appears to follow the old linux odd-devel/even-stable convention, so instead of 2.47.6 we should use 2.48.x if you want to do just the minimum update.

No idea about what uses glib.

benjaminhwilliams commented 6 years ago

Since the latest Long Term Support version of Ubuntu (version 18.04) has now been released, those of us who use Ubuntu as a development environment are starting to be affected by this issue. Affected developers include myself, @jmp1985 and @dagewa.

I've created a short-term fix, which is documented here, but it would be nice to update the dependencies in bootstrap.py and libtbx/auto_build/ by way of a proper solution to the problem.

Naturally, this will require testing on all supported systems, to check that it doesn't break anything. As the oldest system we collectively still aim to support, Centos 5 is most likely to be affected. Perhaps this merits some discussion at a future build meeting?

Of course, though it would be highly desirable to fix this soon, we have the short-term workaround in place. So this can certainly wait until after the next Phenix release, when the associated cctbx_project development freeze is lifted, so there can be no danger of unforeseen knock-on effects.

benjaminhwilliams commented 6 years ago

Short summary (see here for details):

Updating GLib to fix the problem adds a couple of dependencies (libmount-dev for libmount and libpcre3-dev for PCRE), which can be turned off by configuring GLib with the flags --disable-libmount and --with-pcre=internal, respectively.
Updating GLib also means that Fontconfig needs to be updated, which adds the GNU gperf dependency, and I'm afraid I don't think this can be avoided. This, at least, would require a new dependency in the bootstrap.py apparatus.

bkpoon commented 6 years ago

I've been looking into using conda for replacing our dependencies to address these compatibility issues. There is some preliminary work in the conda_build repository (https://github.com/cctbx/conda_build). Dependencies from conda should be more portable. I don't have an Ubuntu 18.04 virtual machine, but I have built Phenix (and DIALS) on Ubuntu 16.04 with the following steps (DIALS-specific),

1) Install and setup conda (https://conda.io/miniconda.html)

bash Miniconda3-latest-Linux-x86_64.sh
. <conda installation>/etc/profile.d/conda.sh

2) Create new environment, "dials"

conda create -n dials python=2

3) Edit ~/.condarc to add channels (see conda_build repository README.md)

channels:
  - cctbx
  - defaults
  - conda-forge
  - bioconda

4) Install dependencies in new environment

conda activate dials
conda install cctbx_dependencies

5) Downgrade matplotlib (2.2.2 has some API changes)

conda install matplotlib=2.0.2

6) Install some missing dependencies

pip install procrunner
conda install tabulate

7) Run bootstrap.py in a new directory (base step is omitted)

mkdir new_dials
cd new_dials
wget "https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_build/bootstrap.py"
python bootstrap.py hot update build --builder=dials --nproc=8 --with-python=`which python`

Once everything is built, it does not look like you need to be in a conda environment to run anything. You can just source build/setpaths.sh like normal from a new terminal.

Tests will pass in dials and dials_regression. There are a few tests that fail for Phenix and it looks like scitbx.suffixtree might not be behaving correctly.

Again, this is preliminary work done a few months ago, but using conda for dependencies should help make building things more portable across different linux distributions.

ndevenish commented 6 years ago

Yes, externally maintained dependencies makes things much easier because somebody else maintains it! Much easier to rely on existing package managers than roll your own. Also allows e.g. separation of run-time and build-time dependencies (which is why the conda channel doesn't need gperf, as it seems to be a fontconfig build-time dependency)

Although the bootstrap versions explicitly work on 16.04, I notice that conda's cctbx_dependencies channel actually installs glib=2.53.6 (Aug 2017), and fontconfig=2.12.6 (Sep 2017), so above the required version changes to make this work on 18.04. As I understand it, all your systems now run off of conda, so I assume you've officially stopped maintaining and testing bootstrap's base build? Or at least matching environments?

Since we're still using bootstrap-base for our releases still, I take it there is no objection to making these updates? I am, however, unsure about the process of physically updating the tarballs on CCI - presumably it's just on a SVN our credentials should access?

bkpoon commented 6 years ago

No, we are not using conda for development or releases. We still use bootstrap.py base in our nightly builds. The conda_build repository is all preliminary work. There are still missing dependencies, additional implementation details, and lots of testing and validation before conda_build is ready for production use.

As such, any updates to the current dependencies require full testing across multiple linux distributions and versions (CentOS 5, 6, maybe 7, Ubuntu 14.04, 16.04, now 18.04, and maybe 12.04, and 32-bit CentOS 5 for legacy systems, but this may be ending soon), multiple macOS versions (10.9 - 10.13 with different versions of Xcode, more realistically build on 10.11 - 10.13, but run on 10.9 - 10.13), and Windows versions (32-bit and 64-bit for some dependencies, 32-bit support may also end soon).

ndevenish commented 5 years ago

Just had another developer run into this exact issue.

cctbx / cctbx_project

Linux base build fails recent ubuntu (glib, etc) #85