XFEL GUI updates

[phyy-nx]

Series of updates from most recent beamtime:

• Switch extant usage of mpi4py to libtbx.mpi4py
• Refresh cached database connection after 5 minutes
• add pydrive2 to psana_environment
• XFEL GUI: only move on to next task if the previous one is in the DONE state
• XFEL GUI: more tweaks for energy tab
• Prevent duplicate gdrive uploads
• Improve mpi safety
• only disable mpi if the gui actually runs
• XFEL GUI: energy tab, label eBeam using average
• XFEL GUI: open image viewer when clicking on a dot

Note 1f4c25c9b89b92aba5873d3f5d37ebaf5c749328 switches over to using libtbx.mpi4py everywhere in cctbx, including simtbx.

[dermen]

Interesting edge case on Summit compute cluster

$ jsrun -n 1 python test_mpi4py.py 
0 1

$ python test_mpi4py.py 
Error: OMPI_COMM_WORLD_RANK is not set in environment like it should be.
Error checking ibm license.

$ cat test_mpi4py.py 
from libtbx.mpi4py import MPI
COMM = MPI.COMM_WORLD
print(COMM.rank, COMM.size)

I would hope that from libtbx.python import MPI could catch the failure, but seems not ?

[phyy-nx]

I would hope that from libtbx.python import MPI could catch the failure, but seems not ?

libtbx.mpi4py is designed to emulate MPI on systems without it, such that at least the program can run single process. It is not designed to catch or fix a broken MPI.

That said, we have found ourselves recently, on more than one occasion, in a situation such that on the login nodes for a given computing cluster, MPI is broken, but on the compute nodes, it works fine. In programs such as the XFEL GUI, this meant that if any import triggered by the XFEL GUI brings in mpi4py, the GUI would break on these login nodes. This PR includes 3 steps to mitigate this problem, without addressing the underlying reason that MPI was broken on that cluster:

• b3408746545daa9f81c886700a3110f3c564606c switches an import in xfel.merge's phil scope to avoid importing mpi4py. This allows the XFEL GUI to validate phil without importing MPI
• 1f4c25c9b89b92aba5873d3f5d37ebaf5c749328 changes all imports of mpi4py in cctbx_project to libtbx.mpi4py
• b3408746545daa9f81c886700a3110f3c564606c adds a guard that will put the XFEL GUI into MPI emulation mode if libtbx.mpi4py ever happens to be imported

With these changes, your example could be changed to this, similar to the new guard in the XFEL GUI, and it would run without a warning on a login node or without jsrun:

import libtbx
libtbx.mpi_import_guard.disable_mpi = True
from libtbx.mpi4py import MPI
COMM = MPI.COMM_WORLD
print(COMM.rank, COMM.size)

But of course, MPI would be disabled.