search

suchyta1 / Valarauko

Super-parallel DES coadd implementation of Balrog
1 stars 0 forks source link

Using Shifter at NERSC #8

Closed suchyta1 closed 8 years ago

suchyta1 commented 8 years ago

I've been going back and forth with Lisa Gerhardt at NERSC, and it should be possible now, I think, to do everything here in the Shifter environment, which runs Docker containers on the compute nodes. This moves metadata operations to the compute nodes, which drastically decreases python import times, so there's some chance for performance improvement here. (I don't know if it would help with the sextractor time, I'll have to try.)

I've had the docker image mostly set up for awhile. So hopefully there will be a branch for this coming soon. On top of the performance improvements, this completely removes installation from the equation which is awesome. (But unfortunately not all supercomputing center have an HPC containerization setup -- it's a burgeoning field.)

If you think like me, this supercomputing + containerization is just super awesomely cool so I hope to have time for this!

suchyta1 commented 8 years ago

Shifter compatibility is included in master. Closing issue.