search

ijiraq / gemini_processing

scripts and stuff to process gemini data witin the CADC.
1 stars 1 forks source link

NIFS Arcade containers #9

Closed Nat1405 closed 3 years ago

Nat1405 commented 4 years ago

This adds draft arcade containers for both an interactive and automatic version of nifty for #4 . The interactive version launches into a bash shell and the automatic version (when provided a program ID) automatically begins reducing that program ID. Some remaining issues to solve:

ijiraq commented 4 years ago

The person running NIFTY interactively will need a cavern storage allocation. They will manage their own persistence. Generally the container will have the cavern storage mounted to it.

For batch processing the results should be persisted to the vault VOSpace. This requires the person running in batch to have a VOSpace allocation and then to use the Python vos tools to persist the data into VOSpace.

ijiraq commented 4 years ago 
* Where should data products be persisted to? So far they're stored in a local directory "/scratch" in the container
Nat1405 commented 4 years ago

After chatting with Brian the next to-do is to push the image to Dockerhub and then test it. Later, maybe we could set up some sort of persistence in /cavern.

ijiraq commented 4 years ago

Could one of the Dockerfiles (interactive or automatic) be built by bringing in the other Dockerfile as a root? I'm wondering if that would help prevent the two execution environments from separating. I guess the 'interactive' Dockerfile would use the 'automatic' one as its root. (I think that you mean 'arcade' rather than 'interactive' BTW).

Nat1405 commented 4 years ago

Hi JJ, for sure. Maybe once I see what it takes to get the container running on batch I'll set them up in that way (if two are even needed, it might be possible to have the same container work for both).

Nat1405 commented 4 years ago

Now that things are running on batch it looks like one container is good for both things. cc96e69 removes the redundant containers. This is working on Arcade and ready to be reviewed (even by me) for style and production deployment options (currently, this is pulling from both my personal docker hub and development copy of Nifty).

Nat1405 commented 4 years ago

Note that the container is now quite large (~6GB), at some point I can go back and try to reduce its size if need be.