phyy-nx
commented
6 years ago Reproduce at slac: cd /reg/d/psdm/mfx/mfxls3416/scratch/brewster/runsnake bsub -n 1 -q psanaq -o process.out libtbx.python -m cProfile -o xtc_out.prof $sources/cctbx_project/xfel/command_line/xtc_process.py input.experiment=mfxls3416 input.run_num=152 params_1.phil max_events=100
This will generate xtc_out.prof which can be viewed with runsnake or a similar profiler.
During a recent experiment, a very dense dataset was collected, including many lattices per frame. Usually, during XFEL processing the import, find spots, index, refine and integrate steps take roughly equal amounts of time. Here, however, when we used multiple lattice indexing and indexing.method_list=fft1d real_space_grid_search fft3d, nearly 90% of the processing time was spent in the indexing step, and most of that time is spent in the function find_candidate_orientation_matrices in dials/algorithms/indexing/indexer.py. Specifically, I ran xtc_process.py single process on a small run with 310 events. The program ran for 6100s, of which 5500s was spent in indexing, of which 3000s was spent in find_candidate_orientation_matrices, or about half.
The profile from cProf is available on request.