Update directories in Spark scripts to match Kestrel changes

NREL / HPC

A collection of various resources, examples, and executables for the general NREL HPC user community's benefit. Use the following website for accessing documentation.

https://nrel.github.io/HPC/

Other

103 stars 64 forks source link

Update directories in Spark scripts to match Kestrel changes #621

Closed daniel-thom closed 2 months ago

daniel-thom commented 2 months ago

The Spark scripts were broken by two recent changes to Kestrel:

/tmp/scratch is no longer writable.
Apptainer images were moved to a new location.

I noticed that several other documentation pages and scripts are affected by the change to /tmp/scratch. Is there any chance that $LOCAL_SCRATCH (only on Kestrel compute nodes with local storage) can be changed to $TMPDIR (instead of /tmp/scratch)? It seems problematic to have that environment variable point to a read-only location.

yandthj commented 2 months ago

Thanks for the updates - working on updating other docs for these changes. I will follow up with the Kestrel team about the $LOCAL_SCRATCH variable and let you know.