Cannot build anything

[Anthchirp]

Starting from an empty environment containing cctbx-base, installing all dials dependencies, cloning sources for dxtbx/dials/xia2 into /modules and running in /build:

$ libtbx.configure dxtbx dials xia2 --compiler=conda --enable_cxx11 --use_conda 
$ make
libtbx.scons -j "`libtbx.show_number_of_processors`"
scons: Reading SConscript files ...
libtbx.scons: using flags from initial environment: 
              CXXFLAGS =  -fvisibility-inlines-hidden -std=c++11 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem (...)/environment/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/cctbx-base_1600872004520/work=/usr/local/src/conda/cctbx-base-2020.8 -fdebug-prefix-map=(...)/environment=/usr/local/src/conda-prefix
              CFLAGS =  -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem (...)/environment/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/cctbx-base_1600872004520/work=/usr/local/src/conda/cctbx-base-2020.8 -fdebug-prefix-map=(...)/environment=/usr/local/src/conda-prefix
              CPPFLAGS =  -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem (...)/environment/include
              LDFLAGS =  -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,(...)/environment/lib -Wl,-rpath-link,(...)/environment/lib -L(...)/environment/lib
libtbx.scons: OpenMP is disabled.
BOOST_VERSION: 107200
AttributeError: 'empty' object has no attribute 'dxtbx_include':
  File "(...)/environment/build/SConstruct", line 19:
    SConscript("spotfinder/SConscript")
  File "(...)/environment/lib/python3.8/site-packages/SCons/Script/SConscript.py", line 661:
    return method(*args, **kw)
  File "(...)/environment/lib/python3.8/site-packages/SCons/Script/SConscript.py", line 598:
    return _SConscript(self.fs, *files, **subst_kw)
  File "(...)/environment/lib/python3.8/site-packages/SCons/Script/SConscript.py", line 287:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "(...)/environment/lib/python3.8/site-packages/spotfinder/SConscript", line 22:
    env_etc.dxtbx_include,
usr+sys time: 0.43 seconds
wall clock time: 1.48 seconds
make: *** [default] Error 2

[bkpoon]

For now, you should use the regular "modules" directory with everything. Also, for dxtbx use the libpath branch. Otherwise, there will probably be a linking error.

[Anthchirp]

For now, you should use the regular "modules" directory with everything.

How does that differ from the layout I described?

conda-root
  /build       <- run libtbx.configure and make here
  ...
  /lib           <- python site-packages live here
  /modules
  /modules/dxtbx    <- dxtbx lives here
  ...
  /share
  ...

I had a go a your dxtbx branch https://github.com/cctbx/dxtbx/pull/221, but I get the same error message. Also the issue here seems to be in the spotfinder SConscript rather than with dxtbx?

Edit: Do you mean put cctbx_modules into /modules as well?

[Anthchirp]

This must be in spotfinder and independent of dxtbx.

I created a new directory in modules named fake with an empty libtbx_refresh.py and a libtbx_config of

{
    "modules_required_for_use": ["iotbx"],
}

and get the same traceback on configuring just fake with a new, clean build directory.

[bkpoon]

Can you try moving the modules directory out of the environment directory? The messages saying dxtbx, dials, and xia2 are installed should go away.

Also, the modules directory needs to have cctbx_project and cbflib. It's easiest to start with the modules directory created from bootstrap.py. We're working with Herbert to get a more recent official release and some hints on modifying the Makefile so that dependencies are not downloaded and installed. Once cbflib is available, the extra cbflib_adaptbx code will be made into another output in this recipe.

Your configuration step also only needs the --use_conda flag. The compiler and associated flags will default to the ones used to build the package. The compilers default to the conda compilers.

I have built dxtbx and dials with the conda-forge packages of cctbx-base, so the above changes should work. If not, please provide the SConstruct file.

[Anthchirp]

Still no luck with my basic empty package example. Can you provide the exact commands and directory layout you were using?

[phyy-nx]

Ok, I've worked out a procedure with @bkpoon. In a new folder with a clean environment:

wget https://raw.githubusercontent.com/dials/dials/master/installer/bootstrap.py
python bootstrap.py update
cp modules/dials/.conda-envs/linux.txt .

Edit the .txt file to add:

conda-forge::cctbx-base=2020.8=py38h2841bc5_0 # pinned build until @bkpoon fixes an issue with Annlib

And remove boost, boost-cpp, and matplotlib-base, to avoid conflicts. @bkpoon notes the version numbers can come off of boost and boost-cpp instead, which might make things easier in general. Then:

conda env create -f linux.txt -n cctbx
conda activate cctbx
mkdir build; cd build
libtbx.configure dxtbx dials xia2 --use_conda
make

@bkpoon also notes --compiler=conda --enable_cxx11 is not needed in this proceedure.

[Anthchirp]

Ah. Thank you!

Okay, the specific cctbx-base version and having to put the build/ and modules/ directories above the envionment made the difference. I can now libtbx.configure and make. In the longer term it would be nice to have these directories inside the environment, but I guess that is what the prefix work is about.

Configure gives the output

Enable C++11: False

and

Compiler: default

however I see the c++11 and conda flags during the build, so scratch another two for misleading output I guess?

I configured and compiled only dxtbx, then went into modules/dxtbx and ran pytest --regression -n auto, which promptly imploded because pytest as a conda-forge curated command doesn't allow access to files in the modules directories, so this was somewhat expected.

However

$ libtbx.python -c "import dxtbx"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'dxtbx'

also failed. I guess even the libtbx dispatchers do not account for a modules directory?

Running the longer dxtbx+dials+xia2 build now.

[phyy-nx]

libtbx.python -c "import dxtbx" works for me. As does cd modules/dxtbx; libtbx.pytest -n 64. Did you source build/setpaths.sh?

[Anthchirp]

Yes, that was the missing bit

../../build/bin/libtbx.python -m pytest --regression -x -n auto

is running now. Having two sets of libtbx dispatchers is confusing.

I forgot earlier: We are already aware of the dials boost dependencies version restriction, https://github.com/dials/dials/pull/1426 🙂

[phyy-nx]

Cool.

having to put the build/ and modules/ directories above the environment

FYI, I have my conda environment somewhere completely different (whole different filesystem), so the relative location of the environment and the modules folder shouldn't matter. The build folder does need to be next to the modules folder though.

[bkpoon]

The removal of annlib needs some bookkeeping updates, so the latest build won't configure correctly. I'll fix it this weekend, which should also have the 2020.9 release.

[Anthchirp]

FYI, I have my conda environment somewhere completely different (whole different filesystem), so the relative location of the environment and the modules folder shouldn't matter. The build folder does need to be next to the modules folder though.

I've just double-checked. Starting from inside the activated environment with a fully populated modules/ directory, running mkdir build; cd build; libtbx.configure dxtbx --use_conda does not work (configure command does not enumerate dxtbx format classes, subsequent make fails)

However if I then proceed with cd ..; mv modules ..; cd ..; mkdir build; cd build; libtbx.configure dxtbx --use_conda that works fine (format classes enumerated, subsequent make runs)

[bkpoon]

With 2020.9, these are the minimal steps for building

1) conda create -n test -c conda-forge cctbx-base 2) conda activate test 3) conda install scons compilers hdf5 libtiff 4) Create a complete modules directory from bootstrap.py 5) mkdir build 6) cd build 7) libtbx.configue dxtbx dials xia2 8) make

You can specify just the C/C++ compilers in step 3. I use compilers to keep it short. There is an issue with running libtbx.configure again in the build directory. That can be fixed in 2020.10 which starts next week.

[Anthchirp]

Following your instructions starting from an empty directory with some conda environment active:

$ conda create -y -p test -c conda-forge cctbx-base scons compilers hdf5 libtiff
$ conda activate ./test
$ wget https://raw.githubusercontent.com/dials/dials/master/installer/bootstrap.py
$ python bootstrap.py update  # setting up modules/...
$ mkdir build
$ cd build
$ libtbx.configure dxtbx
$ make
libtbx.scons -j "`libtbx.show_number_of_processors`"
scons: Reading SConscript files ...
libtbx.scons: using flags from initial environment: 
              CXXFLAGS =  -fvisibility-inlines-hidden -std=c++11 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /scratch2/wra62962/files/conda/test/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/cctbx-base_1602088551966/work=/usr/local/src/conda/cctbx-base-2020.9 -fdebug-prefix-map=/scratch2/wra62962/files/conda/test=/usr/local/src/conda-prefix
              CFLAGS =  -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /scratch2/wra62962/files/conda/test/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/cctbx-base_1602088551966/work=/usr/local/src/conda/cctbx-base-2020.9 -fdebug-prefix-map=/scratch2/wra62962/files/conda/test=/usr/local/src/conda-prefix
              CPPFLAGS =  -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /scratch2/wra62962/files/conda/test/include
              LDFLAGS =  -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/scratch2/wra62962/files/conda/test/lib -Wl,-rpath-link,/scratch2/wra62962/files/conda/test/lib -L/scratch2/wra62962/files/conda/test/lib
libtbx.scons: OpenMP is disabled.
BOOST_VERSION: 107400
KeyError: 'annlib':
  File "/scratch2/wra62962/files/conda/build/SConstruct", line 14:
    SConscript("annlib_adaptbx/SConscript")
  File "/scratch2/wra62962/files/conda/test/lib/python3.8/site-packages/SCons/Script/SConscript.py", line 661:
    return method(*args, **kw)
  File "/scratch2/wra62962/files/conda/test/lib/python3.8/site-packages/SCons/Script/SConscript.py", line 598:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/scratch2/wra62962/files/conda/test/lib/python3.8/site-packages/SCons/Script/SConscript.py", line 287:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/scratch2/wra62962/files/conda/test/lib/python3.8/site-packages/annlib_adaptbx/SConscript", line 4:
    env_etc.annlib_dist = libtbx.env.dist_path("annlib")
  File "/scratch2/wra62962/files/conda/test/lib/python3.8/site-packages/libtbx/env_config.py", line 648:
    result = self.check_local_env('_dist_path', module_name, default, return_relocatable_path)
  File "/scratch2/wra62962/files/conda/test/lib/python3.8/site-packages/libtbx/env_config.py", line 553:
    return getattr(local_env, function)(*args, **kwargs)
  File "/scratch2/wra62962/files/conda/test/lib/python3.8/site-packages/libtbx/env_config.py", line 654:
    result = self.module_dist_paths[module_name]
usr+sys time: 0.37 seconds
wall clock time: 1.35 seconds
make: *** [default] Error 2

[bkpoon]

Looks like an additional check is needed for returning the location of annlib. I'll test in the 2020.10 release.

[bkpoon]

The 2020.10 release should work now. The issue with the 2020.9 release was that I was testing the modules directory from the phenix builder which includes labelit. That explicitly added the annlib dependency. The modules directory from your bootstrap.py does not have labelit and so the build step was not finding annlib correctly. I added an additional search for annlib.

There is one modification to the instructions since cbflib_adaptbx was removed from dxtbx.

1) conda create -n test -c conda-forge cctbx-base 2) conda activate test 3) conda install scons compilers hdf5 libtiff msgpack-c 4) Create a complete modules directory from bootstrap.py 5) mkdir build 6) cd build 7) libtbx.configue dxtbx dials xia2 cbflib_adaptbx 8) make

Step 3 adds the msgpack-c dependency that was missing earlier, and step 7 configures cbflib_adaptbx for building cbflib.

Also, the SConscript for dxtbx has a reference to env_etc.cbflib_common_includes. That variable should be removed if dxtbx no longer depends on cbflib_adaptbx. Otherwise, there will be an error since that variable is defined in cbflib_adaptbx/SConscript.

[Anthchirp]

This seems to work now 👍