search

helenginn / cppxfel

cppxfel source - I finished my doctorate. This program is FOSSILISED! Assume no more support!!
3 stars 2 forks source link

Make post-refinement scalable #11

Open biochem-fan opened 8 years ago

biochem-fan commented 8 years ago

At SACLA, we have only one fat-memory node (1TB RAM) and a time limit of 24 hr for a job. Thus, the memory and time requirement of post-refinement limits the size of the dataset we can process.

I have a suggestion to make post-refinement more scalable on multiple nodes.

  1. refine images in parallel on multiple worker nodes using a common reference
  2. write the refined parameters to the (shared) disk
  3. the master node waits for all worker nodes
  4. the master node merges all images using refined partialities and creates a new reference
  5. distribute the new merged reference to worker nodes and repeat from 1.

This can be ideally implemented using MPI, but more easily by shell scripts using GNU parallel with minimum modifications to cppxfel.

helenginn commented 8 years ago

I think this is divided into two problems: the memory aspect and the multiple worker nodes aspect.

The memory aspect of this has now been fixed (I hope, pending @biochem-fan checking this) by adding a LOW_MEMORY_MODE ON (plus OLD_MERGE OFF) functionality. Ambiguity breaker still needs to be made low-memory.