Deployment of the workflow?

[yxqd]

We would like to figure out what is the best strategy to deploy the software developed by Rick&Rick in Acumen project. The first step is probably that we learn how to run your reconstruction software (a demo would really help), and learn the requirements on the computing resources, and also timing and limitations of the current code.

On our side we probably can try to visualize the data that R&R have reconstructed and try to have a better understanding of the quality of those data. We probably will have questions such as how to fine-tune the weighting parameter on the regularization term, or how to determine it from the noise level of the raw data.

Specific questions regarding current workflow

• Can we feed sinograms directly?
• Data transformation: cif->HPC data (what is HPC data?)
• Do the programs need recompiling everytime? - No (Rick A)
• Can we make a software (python) package out of it?
• How to make the job submission part easy? Burst possible?
• What is the cpu time for reconstruction?

[yxqd]

• HPC data is MPI data file format (and is currently the only format supported to run on super computer but there is plan to support also HDF5)
• Rick B has a python program that does TV on a sinogram: https://github.com/ornlneutronimaging/bregman_tomo

[yxqd]

Rick A demo on Titan

• Apply for OLCF user account. Ask for Project number. (check with Vickie for this)
• $ qsub -I -lnodes=1 -lwalltime=02:00:00 -X -A {project}
• now inside a node, running interactively
  • /lustre/atlas?/{project}
  • proj-shared: 50GB. mostly for project software
  • scratch
    • {user}
  • world-shared
  • tape: 500TB
  • module load ...
  • set env vars (they are the inputs and configuration)
  • compile the program
  • run
  • 900 angles X 2048 X 2048, filtered-back projection: ~<5min
  • one slice TV reconstruction: ~5min
• to get data out: sftp

[yxqd]

Rick B:

• A list of pre-determined angle scan would make the Radon transformation matrices fixed. Then they can be stored, and do not need to be computed on the fly. Just load these into the memory.
• Will only need a few seconds for a TV of one slice

Question:

• Will the memory access time be a problem for GPU?

[yxqd]

Action items

• Get an OLCF account if does not have one
• Try the full Fortran-based workflow on titan
• Try the python scripts (recon from sinogram to slice only) by Rick B

[JeanBilheux]

Getting the data in and out of Titan is simply accomplished by using sftp. Once the data are coming back from super computer cluster, the file format (binary) are converted back to images using currently a matlab code.