Explore requirements for removing dxtbx from cctbx_project to top_level i.e. https://github.com/cctbx/dxtbx

[graeme-winter]

Following conversation this afternoon (morning Pacific time) need to work out practicalities for moving dxtbx to it's own repo but ensuring no circular dependencies etc (though strictly this is orthogonal)

At minimum

• move up one level
• update bootstrap to check out on install of cctbx
• add tool for existing developers to auto-update to reflect this new position

A bit of looking around suggests using git submodules for this so cctbx_project can keep references to specific commits which get checked out when you get cctbx_project, but can be replaced for active development.

Motivation:

• if we are in a position where “conda install cctbx” becomes a reality, the area which we most depend on in DIALS is dxtbx - if we can update this on a different tempo to the wider cctbx this would be useful
• dxtbx is also used outside of DIALS, in fast_dp, and will want to be updated every so often but fast_dp is only loosely coupled to cctbx so can run with very old versions
• for CCP4, it could be realistic to include newer dxtbx versions without updating the full dials install e.g. to get support for new beamline installations
• it would also reduce dependencies for cctbx - already there is a “light” cctbx which does not include e.g. HDF5 - which may simplify conda developments

Discussed with @bkpoon @nksauter @dagewa Paul Adams, Gwyndaf Evans

[graeme-winter]

@nksauter identifies a clear cost of this namely:

"Many of our projects--including simtbx and xfel---are heavily dependent on dxtbx. This seems like a needless administrative change that will simply increase the number of steps a developer needs to perform work. It will increase the difficulty of code bisection, to move the code base back and forward in time to trouble shoot issues, which is already difficult enough."

[graeme-winter]

Info: https://git-scm.com/book/en/v2/Git-Tools-Submodules

"It often happens that while working on one project, you need to use another project from within it. Perhaps it’s a library that a third party developed or that you’re developing separately and using in multiple parent projects. A common issue arises in these scenarios: you want to be able to treat the two projects as separate yet still be able to use one from within the other."

This sounds like the challenge we are encountering here.