search

NVIDIA / pyxis

Container plugin for Slurm Workload Manager
Apache License 2.0
281 stars 31 forks source link

Export all Environment Variables #124

Closed sean-smith closed 1 year ago

sean-smith commented 1 year ago

Is there a way to export all environment variables without explicitly setting them in --container-env=?

Basically I want to do this:

#!/bin/bash
#SBATCH -N 1

# environment variables
export NCCL_DEBUG=info

srun -l --container-image ... bash script.py
flx42 commented 1 year ago

It's the case by default, --container-env is used to override environment variables that are already set in the container image, as the environment variables in the container have precedence over the environment variables from the job.

$ srun --container-image=nvcr.io#nvidia/pytorch:23.09-py3 env | grep NCCL
pyxis: importing docker image: nvcr.io#nvidia/pytorch:23.09-py3
pyxis: imported docker image: nvcr.io#nvidia/pytorch:23.09-py3
NCCL_IB_PCI_RELAXED_ORDERING=0
NCCL_VERSION=2.18.5

$ NCCL_DEBUG=info srun --container-image=nvcr.io#nvidia/pytorch:23.09-py3 env | grep NCCL
pyxis: importing docker image: nvcr.io#nvidia/pytorch:23.09-py3
pyxis: imported docker image: nvcr.io#nvidia/pytorch:23.09-py3
NCCL_DEBUG=info
NCCL_IB_PCI_RELAXED_ORDERING=0
NCCL_VERSION=2.18.5