missing __init__.py files under the <modules>/tests [long]

Hello, I am still working on the packaging of dials and cctbx.

for now, I managed to build dials and cctbx with this code and a bunch of patches.

here a snipset of the Debian make file used to build the package.

override_dh_auto_build:
    # install scons for the build
    ln -sf /usr/lib/scons modules/
    # install procrunner where it can be found by the installer
    # until we package this one file package ???
    cp -f debian/third-party/procrunner.py modules/

    dh_auto_build -- -s custom --build-args='ln -sf $(CURDIR)/modules {build_dir}/modules\
        ; cd {build_dir} && {interpreter} $(CURDIR)/bootstrap.py build\
             --builder="dials"\
             --use-conda=/usr\
             --config-flags="--build=release"\
             --config-flags="--use_environment_flags"\
             --config-flags="--enable_openmp_if_possible=1"\
             --config-flags="--enable_cxx11"'

    dh_auto_build -- -d modules/cctbx_project
    dh_auto_build -- -d modules

there is 4 steps,

I need to link the system scons into the sources of the package.
I deal with procrunner (this is for now a detail).
I build all the libraries and the python extension.
I build the Python moduels using self made setup.py.

after this I just need to install all this with

override_dh_auto_install:
    dh_auto_install -- -s custom --install-args='rm -f {build_dir}/build/bin/py.test\
        ; rm -f {build_dir}/build/bin/pytest\
        ; dh_install -p dials --sourcedir={build_dir} build/bin /usr\
        ; for i in {build_dir}/build/lib/*.so; do \
            patchelf --set-rpath /usr/lib/$(DEB_HOST_MULTIARCH)/dials $$i ;\
            if readelf -Ws $$i | grep -q PyInit ; then \
                dh_install -p python3-dials --sourcedir={build_dir} build/lib/`basename $$i` {install_dir} ; \
            else \
                dh_install -p python3-dials --sourcedir={build_dir} build/lib/`basename $$i` /usr/lib/$(DEB_HOST_MULTIARCH)/dials/ ;\
            fi; \
          done \
        ; rm -f {build_dir}/modules \
        ; rm -rf {build_dir}/build'

    dh_auto_install -- -d modules/cctbx_project
    dh_auto_install -- -d modules/

As you can see I need to remove the py.test and pytest from the binary package because it is also part of the pytest package. (another namespace collision ;).

here these is 3 steps for the installation.

install the library and the extension at the right place on the system
add an rpath to all these shared objects in order to move the c/C+= library under a private directory (this avoid providing a public API for these libraries).
install all the python code with the classical python setup.py install

everythings is for me at this point.

now I can test during the build with this simpe command

override_dh_auto_test:
    dh_auto_test -- -s custom --test-args="cd {build_dir}/build && PATH={build_dir}/build/bin:$(PATH) ./run_tests.csh"

and it works, I am quite happy with this (just remember this https://wiki.debian.org/DebianScience/cctbx remember 7 years ago. I am happy to see that a bunch of our previous problem where almost solved.

now I am facing a issue whcih is quite new for me. I want to run the integration test once the packages are installed on the system. So I started with a simple ./runt_test.py whcih end up with plenty of

libtbx.python "/modules/cctbx_project/cctbx/masks/tests/tst_flood_fill.py"
/usr/bin/python3.8: can't open file '/modules/cctbx_project/cctbx/masks/tests/tst_flood_fill.py': [Errno 2] No such file or directory

I see here at least two problems.

my modules are installer on the system, under /usr/lib/python3/dist-packages/... like all other python modules. so I need to patch something in order for provide the right path for the tests.

can you tell me what must be patch for this ? (I understand that these is something called libtbx_env which is pickled and unpickled, but that's all for me now).

I check on my disque after installtion and most of the test directory where the test are provided. do not contain an __init__.py file. so they are not installed via the setup.py install target.

Can you add these file in the source, in order to install easily the tests of cctbx/dials.

Sorry If I was long, I would like to explain what I am doing and work with you in order to provide a system integrated cctbx to the synchrotron community relying at least on Debian/Ubuntu https://wiki.debian.org/DebianPAN

Cheers

Frederic thanks

with a small script I found this

dirs=`find ../modules -name "*.py" | xargs dirname | uniq`
for d in $dirs; do
    i=$d/__init__.py
    if [ ! -f $i ]; then
        echo "not found $i";
    fi;
done;

PS, I am still working on the dials sources which embed the cctbx_project

/home/picca/Debian/dials/dials/debian/fix_init_.sh
not found ../modules/cbflib/dREL-ply-0.5/__init__.py
not found ../modules/cbflib/drel-ply/__init__.py
not found ../modules/cbflib/ply-3.2/doc/__init__.py
not found ../modules/cbflib/ply-3.2/example/BASIC/__init__.py
not found ../modules/cbflib/ply-3.2/example/GardenSnake/__init__.py
not found ../modules/cbflib/ply-3.2/example/ansic/__init__.py
not found ../modules/cbflib/ply-3.2/example/calc/__init__.py
not found ../modules/cbflib/ply-3.2/example/calcdebug/__init__.py
not found ../modules/cbflib/ply-3.2/example/classcalc/__init__.py
not found ../modules/cbflib/ply-3.2/example/closurecalc/__init__.py
not found ../modules/cbflib/ply-3.2/example/hedit/__init__.py
not found ../modules/cbflib/ply-3.2/example/newclasscalc/__init__.py
not found ../modules/cbflib/ply-3.2/example/optcalc/__init__.py
not found ../modules/cbflib/ply-3.2/example/unicalc/__init__.py
not found ../modules/cbflib/ply-3.2/example/yply/__init__.py
not found ../modules/cbflib/ply-3.2/__init__.py
not found ../modules/cbflib/ply-3.2/test/__init__.py
not found ../modules/cbflib/pycbf/__init__.py
not found ../modules/cbflib/pycbf/xmas/__init__.py
not found ../modules/cbflib/pycbf/__init__.py
not found ../modules/cbflib/src/__init__.py
not found ../modules/ccp4io_adaptbx/dev/__init__.py
not found ../modules/cctbx_project/boost_adaptbx/graph/tests/__init__.py
not found ../modules/cctbx_project/boost_adaptbx/tests/__init__.py
not found ../modules/cctbx_project/cctbx/adptbx/boost_python/__init__.py
not found ../modules/cctbx_project/cctbx/array_family/boost_python/__init__.py
not found ../modules/cctbx_project/cctbx/covariance/tests/__init__.py
not found ../modules/cctbx_project/cctbx/development/cl_template/__init__.py
not found ../modules/cctbx_project/cctbx/dmtbx/boost_python/__init__.py
not found ../modules/cctbx_project/cctbx/eltbx/tests/__init__.py
not found ../modules/cctbx_project/cctbx/geometry/tests/__init__.py
not found ../modules/cctbx_project/cctbx/maptbx/boost_python/__init__.py
not found ../modules/cctbx_project/cctbx/masks/tests/__init__.py
not found ../modules/cctbx_project/cctbx/math/boost_python/__init__.py
not found ../modules/cctbx_project/cctbx/miller/boost_python/__init__.py
not found ../modules/cctbx_project/cctbx/multipolar/regression/__init__.py
not found ../modules/cctbx_project/cctbx/sgtbx/boost_python/__init__.py
not found ../modules/cctbx_project/cctbx/symmetry_search/tests/__init__.py
not found ../modules/cctbx_project/cctbx/translation_search/boost_python/__init__.py
not found ../modules/cctbx_project/cctbx/translation_search/__init__.py
not found ../modules/cctbx_project/cctbx/uctbx/boost_python/__init__.py
not found ../modules/cctbx_project/cctbx/xray/boost_python/__init__.py
not found ../modules/cctbx_project/dox.sphinx/__init__.py
not found ../modules/cctbx_project/dox/compcomm/newsletter08/__init__.py
not found ../modules/cctbx_project/dox/compcomm/newsletter09/__init__.py
not found ../modules/cctbx_project/dox/rst/__init__.py
not found ../modules/cctbx_project/dox/siena2005/__init__.py
not found ../modules/cctbx_project/fable/test/__init__.py
not found ../modules/cctbx_project/fftw3tbx/command_line/__init__.py
not found ../modules/cctbx_project/iotbx/bioinformatics/test/__init__.py
not found ../modules/cctbx_project/iotbx/cif/tests/__init__.py
not found ../modules/cctbx_project/iotbx/examples/pdb_truncate_to_ala/__init__.py
not found ../modules/cctbx_project/iotbx/regression/secondary_structure/__init__.py
not found ../modules/cctbx_project/iotbx/xds/tests/__init__.py
not found ../modules/cctbx_project/mmtbx/building/alternate_conformations/scripts/__init__.py
not found ../modules/cctbx_project/mmtbx/examples/__init__.py
not found ../modules/cctbx_project/mmtbx/geometry/tests/__init__.py
not found ../modules/cctbx_project/mmtbx/refinement/ensemble_refinement/scripts/__init__.py
not found ../modules/cctbx_project/mmtbx/regression/fix_cablam/__init__.py
not found ../modules/cctbx_project/mmtbx/regression/model_idealization/__init__.py
not found ../modules/cctbx_project/mmtbx/regression/pdb_interpretation/__init__.py
not found ../modules/cctbx_project/mmtbx/regression/real_space_refine_chain/__init__.py
not found ../modules/cctbx_project/mmtbx/regression/tncs/__init__.py
not found ../modules/cctbx_project/scitbx/array_family/boost_python/__init__.py
not found ../modules/cctbx_project/scitbx/command_line/__init__.py
not found ../modules/cctbx_project/scitbx/dtmin/regression/__init__.py
not found ../modules/cctbx_project/scitbx/error/__init__.py
not found ../modules/cctbx_project/scitbx/fftpack/boost_python/__init__.py
not found ../modules/cctbx_project/scitbx/fftpack/timing/__init__.py
not found ../modules/cctbx_project/scitbx/glmtbx/tests/__init__.py
not found ../modules/cctbx_project/scitbx/lbfgs/dev/__init__.py
not found ../modules/cctbx_project/scitbx/lbfgsb/boost_python/__init__.py
not found ../modules/cctbx_project/scitbx/linalg/tests/__init__.py
not found ../modules/cctbx_project/scitbx/matrix/tests/__init__.py
not found ../modules/cctbx_project/scitbx/minpack/__init__.py
not found ../modules/cctbx_project/scitbx/random/tests/__init__.py
not found ../modules/cctbx_project/scitbx/sparse/tests/__init__.py
not found ../modules/cctbx_project/scitbx/suffixtree/test/__init__.py
not found ../modules/cctbx_project/scitbx/tests/__init__.py
not found ../modules/cctbx_project/simtbx/command_line/__init__.py
not found ../modules/cctbx_project/simtbx/nanoBragg/brunger_example/__init__.py
not found ../modules/cctbx_project/smtbx/ab_initio/development/__init__.py
not found ../modules/cctbx_project/smtbx/absolute_structure/tests/__init__.py
not found ../modules/cctbx_project/smtbx/command_line/tests/__init__.py
not found ../modules/cctbx_project/smtbx/masks/tests/__init__.py
not found ../modules/cctbx_project/smtbx/refinement/restraints/tests/__init__.py
not found ../modules/cctbx_project/smtbx/refinement/tests/__init__.py
not found ../modules/cctbx_project/smtbx/structure_factors/direct/tests/__init__.py
not found ../modules/cctbx_project/smtbx/tests/__init__.py
not found ../modules/cctbx_project/sphinx/__init__.py
not found ../modules/cctbx_project/spotfinder/core_toolbox/boost_python/__init__.py
not found ../modules/cctbx_project/wxtbx/regression/__init__.py
not found ../modules/cctbx_project/xfel/sacla/__init__.py
not found ../modules/dials/doc/examples/__init__.py
not found ../modules/dials/doc/examples/read_experiment_and_data/__init__.py
not found ../modules/dials/doc/examples/__init__.py
not found ../modules/dials/doc/sphinx/__init__.py
not found ../modules/dials/installer/__init__.py
not found ../modules/dials/test/algorithms/background/__init__.py
not found ../modules/dials/test/algorithms/image/connected_components/__init__.py
not found ../modules/dials/test/algorithms/image/fill_holes/__init__.py
not found ../modules/dials/test/algorithms/image/filter/__init__.py
not found ../modules/dials/test/algorithms/image/__init__.py
not found ../modules/dials/test/algorithms/image/threshold/__init__.py
not found ../modules/dials/test/algorithms/polygon/clip/__init__.py
not found ../modules/dials/test/algorithms/polygon/__init__.py
not found ../modules/dials/test/algorithms/profile_model/__init__.py
not found ../modules/dials/test/algorithms/reflection_basis/__init__.py
not found ../modules/dials/test/algorithms/shoebox/__init__.py
not found ../modules/dials/test/algorithms/spatial_indexing/__init__.py
not found ../modules/dials/test/algorithms/statistics/__init__.py
not found ../modules/dials/test/algorithms/symmetry/__init__.py
not found ../modules/dials/test/array_family/__init__.py
not found ../modules/dials/test/framework/__init__.py
not found ../modules/dxtbx/example/__init__.py
not found ../modules/dxtbx/tests/command_line/__init__.py
not found ../modules/dxtbx/tests/format/__init__.py
not found ../modules/dxtbx/tests/serialize/__init__.py
not found ../modules/tntbx/__init__.py
not found ../modules/tntbx/__init__.py
not found ../modules/xia2/Driver/Test/__init__.py
not found ../modules/xia2/Test/Handlers/__init__.py
not found ../modules/xia2/Test/Modules/Indexer/__init__.py
not found ../modules/xia2/Test/Modules/Integrater/__init__.py
not found ../modules/xia2/Test/Modules/Scaler/__init__.py
not found ../modules/xia2/Test/Modules/__init__.py
not found ../modules/xia2/Test/Schema/__init__.py
not found ../modules/xia2/Test/System/__init__.py
not found ../modules/xia2/Test/Wrappers/CCP4/__init__.py
not found ../modules/xia2/Test/Wrappers/Dials/__init__.py
not found ../modules/xia2/Test/command_line/__init__.py
not found ../modules/xia2/doc/sphinx/__init__.py

Probably all of these missing init files are "missing" on purpose. Not every directory is a python module. These paths are therefore deliberately not importable from python because they either contain documentation, C++ code, or test files. This includes python tests.

Neither libtbx testing nor pytest testing require tests to be importable.

Hello, In fact I juste compare my experience with cctbx and the wrll known numpy and scipy. they provide the test modules inside there namesapce this way it is really easy to run all the tests. numpy.tests()

It is especially important for numerical modules. the stability of numerical algo is tought.

So I imagineg that it would be nice to have this in cctbx in order to check that the cctbx installed has a sane comportement regardin the numerical stability.

This is why I pre-suppose that the test were distributed with the cctbx modules. it seems that I am wrong, and maybe the best way to cehck that the installation is ok, is to run the tests from the source but with the system installed modules.

they provide the test modules inside there namesapce this way it is really easy to run all the tests.

For better or worse, numpy and scipy have differing priorities

No problem, I just need to know how to run these test even if they are not included in the cctbx namespace.

I will investigate in order to run the test with the system cctbx instead of the one built from the sources :). The purpose is to provide these kind of test ran by https://ci.debian.net/doc/ this way the test are executed each time a dependency change :).

@picca Nice! One of the goals with the work I've been doing to switch to conda for dependencies and to build a conda package of cctbx is to generalize our build system so that cctbx can be installed into the standard $PREFIX location. It's good to know that the --use-conda flag works for you.

Some comments 1) scons does not need to be available in modules if import SCons works in your /usr/bin/python ( /usr/bin/python3 ) and /usr/lib/python<version>/site-packages/SCons exists. I added that behavior around May 2019. 2) I had the same issue with the location of test files when testing the conda package. I just ended up copying the full directories into python<version>/site-packages. You can extract a tarball (2020.03.alpha) from https://anaconda.org/cctbx-dev/cctbx/files to see how I packaged cctbx for conda. I have a script, install_build.py, in the conda_compiler branch that does that copy, along with the header files ($PREFIX/include), shared libraries ($PREFIX/lib), and Python extensions ($PREFIX/lib/python<version>/lib-dynload). There are flags to customize the locations, so let me know if you want it to do something else. 3) How do you handle libtbx_env and $LIBTBX_BUILD? In the conda_compiler branch, I do set a default for $LIBTBX_BUILD to sys.prefix. I see that @luc-j-bourhis provided you a patch earlier. Are you doing the same thing? Can you put libtbx_env in /usr? There is another script, update_libtbx_env.py, in the conda_compiler branch that copies libtbx_env to $PREFIX and updates the internal bookkeeping for the modules. There are tests and code in cctbx_project that will sometimes use this internal bookkeeping to find something. To see how I install cctbx into $PREFIX you can look at my build script for the conda package. With these changes, I can run libtbx.run_tests_parallel module=cctbx with the conda package. However, there are some tests that are compiled binaries. Those tests will not pass with the conda package because those binaries were not copied (they are not in build/bin). 4) Theupdate_libtbx_env.py script will also regenerate all the dispatchers and remove a bunch of environment variables that are normally set, but no longer necessary (e.g. PYTHONPATH, LD_LIBRARY_PATH, PATH, etc.).

Let me know if you have any questions. I'd like to make this installation process into $PREFIX generally less painful.

@piccahttps://github.com/picca Nice! One of the goals with the work I've been doing to switch to conda for dependencies and to build a conda package of cctbx is to generalize our build system so that cctbx can be installed into the standard $PREFIX > location. It's good to know that the --use-conda flag works for you.

For now I work with the cctbx embeded into the dials release distribution.

dials-v2-2-2-source.tar.xz from https://github.com/dials/dials/releases

So i am not sure that I have the latest cctbx_project. I schouse to work from there because, my first goal is to have a working dials at SOLEIL for our autoprocessing project (jhaskell based ;). Once I will have something working, I will have time to do a better job, split everythings in nice binary packages, install the public C/C++ api at the right place. So for --use-conda, I can not remember what is the purpose of this flag. I do not use conda during the build process.

Some comments

scons does not need to be available in modules if import SCons works in your /usr/bin/python ( /usr/bin/python3 ) and /usr/lib/python/site-packages/SCons exists. I added that behavior around May 2019https://github.com/cctbx /cctbx_project/commit/71f4b84d78f071e86602359fffe65e2eedc2d393>.

On Debian we are not found of scons https://wiki.debian.org/UpstreamGuide#SCons it has plenty of limitations when it comes to packaging so it seems that the maintainer of scons decided to provide only the scons applciation and not expose Scons as a python module. A few years ago I was seduced by scons also and start using it, but then I start to properly package my software in Debian and I changed my mind, I was wrong, the importt part of the build system is to be mainstream. So for C/C++ automake, cmake and now meson and for python to provide setup.py (extensions also). So it would simplify a lot our work if most upstream would follow these principe. I know it is a lot of work, so I am now pragmatic and do what I can with the project I need to work with. :)

I had the same issue with the location of test files when testing the conda package. I just ended up copying the full directories into python/site-packages. You can extract a tarball (2020.03.alpha) from https://anaconda.org/cctbx-dev/cctbx/files to see how I packaged cctbx for conda. I have a script, install_build.py, in the conda_compiler branchhttps://github.com/cctbx/cctbx_project/tree/conda_compiler/libtbx/auto_build/conda_build that does that copy, along with the header files ($PREFIX/include), shared libraries ($PREFIX/lib), and Python extensions ($PREFIX/lib/python/lib-dynload). There are flags to customize the locations, so let me know if you want it to do something else.

for testing, I will use the extracted source and run the test but for this I need to generate the run_test.py file. Do you know how to do only this from the sources ?

Since I decided to do something minimalist for now, I decided to not install the public API of cctbx, only the python modules. So the library are installed under a private directory. So I do not need to deal with C/C++ public API. I do not know the satbility of these C/C++ library and it is a lot of work to maintain C/C++ library in that case on Debian whcih is quite strict with this. so for now I install things by and via the debian/rules file and it was not so bad.

Idfealy, the C/C++ library should be build via cmake in your case (multi platform support) and the python part + extension should be build from setup.py.

How do you handle libtbx_env and $LIBTBX_BUILD? In the conda_compiler branch, I do set a default for $LIBTBX_BUILD to sys.prefix. I see that @luc-j-bourhishttps://github.com/luc-j-bourhis provided you a patch earlier. Are you doing the same thing? Can you put libtbx_env in /usr? There is another script, update_libtbx_env.py, in the conda_compiler branch that copies libtbx_env to $PREFIX and updates the internal bookkeeping for the modules. There are tests and code in cctbx_project that will sometimes use this internal bookkeeping to find something. To see how I install cctbx into $PREFIX you can look at my build scripthttps://github.com/bkpoon/staged-recipes/blob/cctbx/recipes/cctbx/build.sh for the conda package. With these changes, I can run libtbx.run_tests_parallel module=cctbx with the conda package. However, there are some tests that are compiled binaries. Those tests will not pass with the conda package because those binaries were not copied (they are not in build/bin).

I have a real issu with the libtbx_env part. first I do not understant it completly ;), second it seems to me that it hardcode some information given by the user during the build. It is important to have something relocatable under //... for the packaging. My workaround for now is to move the generated libtbx_env under /etc on user machine. It seems to me that this is a configuration part of the libtbx module. AFter it could be somewhere else.

I understood that I need to regenerate a libtbx_env for the package once installed under /prefix so the path are correct. So I need a script whcih allows me to create the right libtbx_env file.

but I have another probleme :)

If I need to split my packaging effort and package cctbx apart of dials, I will end up with two different libtbx_env files. (list of modules are different in thaht case)

I do not know now how to deal with this, but I have the impression that I should create this libtbx_enf file during the installation. so each package should provide their libtbx_env somewhere and during the installation, I should merge them to be uptodate with all software managed via libtbx.

this is lot's of work, so I decided for now to concentrate myself on dials whcih contain cctbx.

All this should be avoid with cmake + setuptools :p

Theupdate_libtbx_env.py script will also regenerate all the dispatchers and remove a bunch of environment variables that are normally set, but no longer necessary (e.g. PYTHONPATH, LD_LIBRARY_PATH, PATH, etc.).

I need indeed a way to regenerate all the command_line file which endup under /bin

I will look at it :).

Let me know if you have any questions. I'd like to make this installation process into $PREFIX generally less painful.

cmake + setuptools :p

they did it because no one tell them that it was impossible ;)

cheers and thank a lot for your explainations.

For now I work with the cctbx embeded into the dials release distribution. dials-v2-2-2-source.tar.xz from https://github.com/dials/dials/releases So i am not sure that I have the latest cctbx_project.

The bundled version of cctbx_project corresponds to https://github.com/dials/cctbx/tree/dials-2.2, which was branched on the 15th of March and has seen some maintenance since. From a Debian point of view that branch is a sensible choice because it will have long-term support. Well, long-term in the sense of "until the end of the year", which although a lot shorter than a Debian cycle is a lot longer than our usual releases.

scons vs cmake

You do have a point here, and @ndevenish has looked into using cmake for dials builds. It is possible but also very involved as scons sits at the heart of the cctbx build machinery. So it is not something we can offer or support at this point. The same goes for setup.py based package installation. We have transmogrified some ancillary cctbx-packages to normal python packages already, eg. https://pypi.org/project/fast-dp/, and more are in the pipeline. Again this is a lot of work to get right within a cctbx environment. Moving towards established practices is very much on our todo list, and we are making steady process - but it will take time.

I have a real issu with the libtbx_env part. first I do not understant it completely ;)

For reference: you can use libtbx.show_env $path_to_libtbx_env to display what is stored within the file. There is some global build configuration and a package and package-dependency registry. In a normal python project the equivalent information would mostly sit in setup.py and be maintained by pip. It is technically possible to remove the need for libtbx_env entirely and use standard python conventions, but that is in an even more distant future.

@picca The --use-conda flag would add some standard locations from $PREFIX for headers and libraries. Essentially, I'm adding -I${PREFIX}/include and -L${PREFIX}/lib to the base building environment. You can think of it as a --prefix flag. Originally, it was meant for adding support for conda (hence --use-conda), but there is no specific requirement that the directory provided is a conda environment. It can be any directory, like /usr that follows the bin, lib, include structure. On Windows, it is conda-specific since that directory structure is replicated in a specific location.

For libtbx_env, the update_libtbx_env.py script does the updates, but it assumes the location of libtbx_env is in $PREFIX. I can make that script more general and add a flag that lets you choose a location. I can do that next week. The final location of libtbx_env may change for the conda package so these changes will help me as well (maybe ${PREFIX}/share/cctbx?). However, are you setting $LIBTBX_BUILD anywhere? If not, you'll have to patch cctbx_project/libtbx/env_config.py to find libtbx_env in a specific location. You can take a look at my changes in the conda_compiler branch https://github.com/cctbx/cctbx_project/blob/cfa432a66e5220d2d6af2170869728a4f91e05cf/libtbx/env_config.py#L2782-L2788 You would change build_dir to your final location of libtbx_env. For now, you can use the libtbx_env from the DIALS installation. I will be adding functionality to add and remove modules from libtbx_env, but that's not ready yet.

As for running tests, you can take a look at this section from cctbx_project/cctbx/run_tests.py, https://github.com/cctbx/cctbx_project/blob/cc035acb293881349887d7cd7b5fd15056de3511/cctbx/run_tests.py#L164-L171 build_dir would be $PREFIX and dist_dir can be your full source location. I have not personally tested changing these variables, but this should be the area that affects where the tests are found.

As for build systems, I don't think it's likely that we'll be changing the underlying build system for cctbx since we've already been building cross-platform for many years. Supporting portable installations into standard locations like $PREFIX is a good idea, but it does not require fundamentally changing the build system, and the work I'm doing for building conda packages will generally do that. And once cctbx is in standard locations, pure Python packages that use cctbx can be built with normal setuptools and conda approaches.

Hello, sorry for this long silence. I was updating the Debian cbflib package and add the pycbf module.

Thansk a lot for all these info, in order to select a path for the system libtbx_env, I need to know if this file is bit identical on the architectures architecture (amd64, arm64, etc...). We build for a bunch of architecture and since we manage co installation of libraries on the same disk, we have different path if the files are arch dependent or not.

I try to un-bundle as much as possible third-party libraries and use the system ones. On a long run, it would be great if all these bundles library couls be under a thir-party directory. This way un-bundling is a simple rm -rf thir-party, I can fill a bug for this is you thinq that this is interesting. I found a couple of issues about bundled libraries that we can discuss in this bug.

here is my todo list for dial/cctbx for now

https://salsa.debian.org/science-team/dials/-/blob/master/debian/TODO

The file is a python pickle file, so format-wise I believe it should be cross-platform. Obviously if you need to change contents it won't be.

Further bits for your todo list:

dials_data is not a runtime requirement, you only need it if you want to run regression tests
same with pytest
entry points are registered in per-module libtbx_refresh.py files via libtbx.pkg_utils.define_entry_points

Ok so let's start with something simple. We can can set the search path for libtbx_env I would put it under /var/lib/libtbx/

If you install another module which use libtbx this files should be updated.

Idealy the runtime dependencies should be in the setup.py in order to auto generates the dependecies during the package build process. I will do it for now manually.

for the netry points I need to patch libtbx_refresh.py in order to simplify all the entry_points ?

Hello, I try to understand the content of libtbx_env

So I juste unpickle it's content and print its dict

/tmp$ python3 test.py 
{'python_version_major_minor': (3, 8), 'build_path': absolute_path("/home/picca/Debian/dials/dials/.pybuild/cpython3_3.8_dials/build/build"), '_dispatcher_precall_commands': None, 'bin_path': relocatable_path(anchor="/home/picca/Debian/dials/dials/.pybuild/cpython3_3.8_dials/build/build", relocatable="bin"), 'exe_path': relocatable_path(anchor="/home/picca/Debian/dials/dials/.pybuild/cpython3_3.8_dials/build/build", relocatable="exe"), 'lib_path': relocatable_path(anchor="/home/picca/Debian/dials/dials/.pybuild/cpython3_3.8_dials/build/build", relocatable="lib"), 'include_path': relocatable_path(anchor="/home/picca/Debian/dials/dials/.pybuild/cpython3_3.8_dials/build/build", relocatable="include"), 'python_exe': relocatable_path(anchor="/home/picca/Debian/dials/dials/.pybuild/cpython3_3.8_dials/build/build", relocatable="/usr/bin/python3.8"), 'command_version_suffix': None, 'explicitly_requested_modules': {'scitbx', 'wxtbx', 'mmtbx', 'dials', 'iotbx', 'iota', 'prime', 'libtbx', 'dxtbx', 'cbflib', 'xia2', 'smtbx', 'gltbx', 'cctbx'}, 'build_options': <libtbx.env_config.build_options object at 0x7f2bf136e970>, 'repository_paths': [relocatable_path(anchor="/home/picca/Debian/dials/dials/.pybuild/cpython3_3.8_dials/build/build", relocatable="../../../../modules"), relocatable_path(anchor="/home/picca/Debian/dials/dials/.pybuild/cpython3_3.8_dials/build/build", relocatable="../../../../modules/cctbx_project")], 'command_line_redirections': {}, 'module_list': [<libtbx.env_config.module object at 0x7f2bf136ea60>, <libtbx.env_config.module object at 0x7f2bf136eaf0>, <libtbx.env_config.module object at 0x7f2bf136eb80>, <libtbx.env_config.module object at 0x7f2bf136ec10>, <libtbx.env_config.module object at 0x7f2bf136ed00>, <libtbx.env_config.module object at 0x7f2bf136edc0>, <libtbx.env_config.module object at 0x7f2bf136ee50>, <libtbx.env_config.module object at 0x7f2bf136eee0>, <libtbx.env_config.module object at 0x7f2bf136ef70>, <libtbx.env_config.module object at 0x7f2bf130b0a0>, <libtbx.env_config.module object at 0x7f2bf130b130>, <libtbx.env_config.module object at 0x7f2bf130b1c0>, <libtbx.env_config.module object at 0x7f2bf130b2b0>, <libtbx.env_config.module object at 0x7f2bf130b340>, <libtbx.env_config.module object at 0x7f2bf130b3d0>, <libtbx.env_config.module object at 0x7f2bf130b490>, <libtbx.env_config.module object at 0x7f2bf130b4f0>, <libtbx.env_config.module object at 0x7f2bf130b580>, <libtbx.env_config.module object at 0x7f2bf130b670>, <libtbx.env_config.module object at 0x7f2bf130b700>, <libtbx.env_config.module object at 0x7f2bf130b790>, <libtbx.env_config.module object at 0x7f2bf130b820>, <libtbx.env_config.module object at 0x7f2bf130b8b0>, <libtbx.env_config.module object at 0x7f2bf130b940>, <libtbx.env_config.module object at 0x7f2bf130b9d0>, <libtbx.env_config.module object at 0x7f2bf130ba60>, <libtbx.env_config.module object at 0x7f2bf130baf0>, <libtbx.env_config.module object at 0x7f2bf130bb80>], 'module_dict': {'libtbx': <libtbx.env_config.module object at 0x7f2bf136ea60>, 'cctbx': <libtbx.env_config.module object at 0x7f2bf136eee0>, 'chiltbx': <libtbx.env_config.module object at 0x7f2bf136eaf0>, 'scitbx': <libtbx.env_config.module object at 0x7f2bf136ee50>, 'boost': <libtbx.env_config.module object at 0x7f2bf136ec10>, 'tbxx': <libtbx.env_config.module object at 0x7f2bf136eb80>, 'omptbx': <libtbx.env_config.module object at 0x7f2bf136ed00>, 'fable': <libtbx.env_config.module object at 0x7f2bf136edc0>, 'cbflib': <libtbx.env_config.module object at 0x7f2bf136ef70>, 'dxtbx': <libtbx.env_config.module object at 0x7f2bf130b340>, 'iotbx': <libtbx.env_config.module object at 0x7f2bf130b2b0>, 'ucif': <libtbx.env_config.module object at 0x7f2bf130b0a0>, 'smtbx': <libtbx.env_config.module object at 0x7f2bf130b130>, 'ccp4io': <libtbx.env_config.module object at 0x7f2bf130b1c0>, 'mmtbx': <libtbx.env_config.module object at 0x7f2bf130b3d0>, 'gltbx': <libtbx.env_config.module object at 0x7f2bf130b490>, 'wxtbx': <libtbx.env_config.module object at 0x7f2bf130b4f0>, 'dials': <libtbx.env_config.module object at 0x7f2bf130ba60>, 'rstbx': <libtbx.env_config.module object at 0x7f2bf130b700>, 'annlib': <libtbx.env_config.module object at 0x7f2bf130b580>, 'spotfinder': <libtbx.env_config.module object at 0x7f2bf130b670>, 'crys3d': <libtbx.env_config.module object at 0x7f2bf130b790>, 'xfel': <libtbx.env_config.module object at 0x7f2bf130b9d0>, 'cma_es': <libtbx.env_config.module object at 0x7f2bf130b820>, 'simtbx': <libtbx.env_config.module object at 0x7f2bf130b8b0>, 'prime': <libtbx.env_config.module object at 0x7f2bf130b940>, 'xia2': <libtbx.env_config.module object at 0x7f2bf130baf0>, 'iota': <libtbx.env_config.module object at 0x7f2bf130bb80>}, 'module_dist_paths': {'libtbx': relocatable_path(anchor="/home/picca/Debian/dials/dials/.pybuild/cpython3_3.8_dials/build/build", relocatable="../../../../modules/cctbx_project/libtbx"), 'cctbx': relocatable_path(anchor="/home/picca/Debian/dials/dials/.pybuild/cpython3_3.8_dials/build/build", relocatable="../../../../modules/cctbx_project/cctbx"),

From what I see, there is a ansolute build_path

and relocatable_path (which seems relativ) exemple for the bin_path 'bin_path': elocatable_path(anchor="/home/picca/Debian/dials/dials/.pybuild/cpython3_3.8_dials/build/build", relocatable="bin")

at the end all my bin will by installed under /usr/bin. so how must I tune libtbx_env in order to deal with the install path and the build path. It seems to me that we should have a build_path and an as_installed_path.

what is th epurpos of this

'repository_paths': [relocatable_path(anchor="/home/picca/Debian/dials/dials/.pybuild/cpython3_3.8_dials/build/build", relocatable="../../../../modules"), relocatable_path(anchor="/home/picca/Debian/dials/dials/.pybuild/cpython3_3.8_dials/build/build", relocatable="../../../../modules/cctbx_project")]

there is also the 'module_dist_paths', whcih contain a disctionnary with for each module a relocatable_path with contain an anchor.

It is not clear to me how to setup a libtbx_env which knows were the modules are installed on the system.

I am wondering also if distributing libtbx whcih is a build system is necessary. But maybe I am wrong and libtbx is required inorder to have all the cctbx_project modules up and running on the system.

thanks for helping, clarifying all this, and sorry if this is too much noise.

Frederic

Hello, I try to understand the command line part

I took the dials command as an example

I have all these command once installed

picca@2a02-8420-6c55-6500-d012-4688-0bee-a0c6:~/Debian/dials/dials$ dials.
dials.align_crystal                  dials.export_best                    dials.model_background               dials.sequence_to_stills
dials.analyse_background             dials.export_bitmaps                 dials.modify_geometry                dials.shadow_plot
dials.anvil_correction               dials.filter_reflections             dials.plot_Fo_vs_Fc                  dials.show
dials.apply_mask                     dials.find_hot_pixels                dials.plot_reflections               dials.show_build_path
dials.assign_experiment_identifiers  dials.find_shared_models             dials.plot_scan_varying_model        dials.show_dist_paths
dials.augment_spots                  dials.find_spots                     dials.plugins                        dials.slice_sequence
dials.background                     dials.find_spots_client              dials.predict                        dials.split_experiments
dials.check_indexing_symmetry        dials.find_spots_server              dials.python                         dials.spot_counts_per_image
dials.cluster_unit_cell              dials.frame_orientations             dials.reciprocal_lattice_viewer      dials.spot_resolution_shells
dials.combine_experiments            dials.generate_distortion_maps       dials.refine                         dials.stereographic_projection
dials.compare_orientation_matrices   dials.generate_mask                  dials.refine_bravais_settings        dials.stills_process
dials.complete_full_sphere           dials.geometry_viewer                dials.refine_error_model             dials.stills_process_mpi
dials.compute_delta_cchalf           dials.goniometer_calibration         dials.reflection_viewer              dials.symmetry
dials.convert_to_cbf                 dials.image_viewer                   dials.reindex                        dials.two_theta_offset
dials.cosym                          dials.import                         dials.report                         dials.two_theta_refine
dials.create_profile_model           dials.import_xds                     dials.resolutionizer                 dials.unit_cell_histogram
dials.damage_analysis                dials.index                          dials.rl_png                         dials.version
dials.detect_blanks                  dials.integrate                      dials.rs_mapper                      
dials.estimate_gain                  dials.merge                          dials.scale                          
dials.export                         dials.merge_cbf                      dials.search_beam_position

but if I look at the libtbx_refresh.py, I find only this

libtbx.pkg_utils.define_entry_points(
    {
        "dxtbx.profile_model": [
            "gaussian_rs = dials.extensions.gaussian_rs_profile_model_ext:GaussianRSProfileModelExt"
        ],
        "dxtbx.scaling_model_ext": [
            "physical = dials.algorithms.scaling.model.model:PhysicalScalingModel",
            "KB = dials.algorithms.scaling.model.model:KBScalingModel",
            "array = dials.algorithms.scaling.model.model:ArrayScalingModel",
        ],
        "dials.index.basis_vector_search": [
            "fft1d = dials.algorithms.indexing.basis_vector_search:FFT1D",
            "fft3d = dials.algorithms.indexing.basis_vector_search:FFT3D",
            "real_space_grid_search = dials.algorithms.indexing.basis_vector_search:RealSpaceGridSearch",
        ],
        "dials.index.lattice_search": [
            "low_res_spot_match = dials.algorithms.indexing.lattice_search:LowResSpotMatch"
        ],
        "dials.integration.background": [
            "Auto = dials.extensions.auto_background_ext:AutoBackgroundExt",
            "glm = dials.extensions.glm_background_ext:GLMBackgroundExt",
            "gmodel = dials.extensions.gmodel_background_ext:GModelBackgroundExt",
            "simple = dials.extensions.simple_background_ext:SimpleBackgroundExt",
            "null = dials.extensions.null_background_ext:NullBackgroundExt",
            "median = dials.extensions.median_background_ext:MedianBackgroundExt",
        ],
        "dials.integration.centroid": [
            "simple = dials.extensions.simple_centroid_ext:SimpleCentroidExt"
        ],
        "dials.spotfinder.threshold": [
            "dispersion = dials.extensions.dispersion_spotfinder_threshold_ext:DispersionSpotFinderThresholdExt",
            "dispersion_extended = dials.extensions.dispersion_extended_spotfinder_threshold_ext:DispersionExtendedSpotFinderThresholdExt",
        ],
    }
)

I do not understand the relation between the bin scripts and the content of this file.

Can you help me understand this ?

The most common use of python entry points is to declare command line scripts, but that is only one specific use with the special entry point name console_scripts. The libtbx architecture predates python packaging history and the use of entry points. In pure libtbx-packages command line scripts are therefore identified by them being a python file in a $module/command_line directory.

We have started using entry points to register dials extensions and dxtbx format classes and stuff like this, but we are not using them at this point in time to declare command line scripts. There is a reason for that: if you look at one of those command line scripts in an installation they look wildly different compared to a python-generated console_scripts dispatcher. We set up a lot more environment variables.

Thanks and what is the function incharge of the production of the wrappers ? I would like to patch in order to simplify their content.

cctbx / cctbx_project

missing init.py files under the <modules>/tests [long] #460

cctbx / cctbx_project

missing __init__.py files under the <modules>/tests [long] #460

missing init.py files under the <modules>/tests [long] #460