argument type error in pixsim

[lastephey]

Hi DESI developers,

I provided a pixsim multiprocessing example to one of our vendors some time ago that we are now revisiting. I'm having trouble running the example

pixsim --night 20171102 --expid 1 --cameras b0

This results in the traceback:

(/global/common/software/das/stephey/desi-multi) stephey@perlmutter:login28:/pscratch/sd/s/stephey/desi-multiprocessing> pixsim --night 20171102 --expid 1 --cameras b0
Traceback (most recent call last):
  File "/pscratch/sd/s/stephey/desisim/bin/pixsim", line 14, in <module>
    args = pixsim.parse()
  File "/pscratch/sd/s/stephey/desisim/py/desisim/scripts/pixsim.py", line 110, in parse
    expand_args(args)
  File "/pscratch/sd/s/stephey/desisim/py/desisim/scripts/pixsim.py", line 36, in expand_args
    args.simspec = io.findfile('simspec', args.night, args.expid)
  File "/pscratch/sd/s/stephey/desisim/py/desisim/io.py", line 62, in findfile
    outdir = simdir(night, expid)
  File "/pscratch/sd/s/stephey/desisim/py/desisim/io.py", line 1083, in simdir
    dirname = os.path.join(
  File "/global/common/software/das/stephey/desi-multi/lib/python3.8/posixpath.py", line 90, in join
    genericpath._check_arg_types('join', a, *p)
  File "/global/common/software/das/stephey/desi-multi/lib/python3.8/genericpath.py", line 152, in _check_arg_types
    raise TypeError(f'{funcname}() argument must be str, bytes, or '
TypeError: join() argument must be str, bytes, or os.PathLike object, not 'NoneType'

Initially I wondered if this was due to a software installation problem, but I just rebuilt my test environment with the workaround Stephen kindly pointed me to, so I think it may be something else.

Here are all the steps I used to build and configure my environment on Perlmutter to reproduce this error.

module load PrgEnv-gnu
module load python
conda create -n desi-multi python=3.8 -y
conda activate desi-multi
cd $SCRATCH

git clone https://github.com/desihub/desisim
git clone https://github.com/desihub/specter
git clone https://github.com/desihub/desimodel
git clone https://github.com/desihub/desiutil
git clone https://github.com/desihub/desispec
git clone https://github.com/desihub/desitarget

export PYTHONPATH=$SCRATCH/desisim/py:$PYTHONPATH
export PATH=$SCRATCH/desisim/bin:$PATH

export PYTHONPATH=$SCRATCH/specter/py:$PYTHONPATH
export PATH=$SCRATCH/specter/bin:$PATH

export PYTHONPATH=$SCRATCH/desimodel/py:$PYTHONPATH
export PATH=$SCRATCH/desimodel/bin:$PATH

export PYTHONPATH=$SCRATCH/desiutil/py:$PYTHONPATH
export PATH=$SCRATCH/desiutil/bin:$PATH

export PYTHONPATH=$SCRATCH/desispec/py:$PYTHONPATH
export PATH=$SCRATCH/desispec/bin:$PATH

export PYTHONPATH=$SCRATCH/desitarget/py:$PYTHONPATH
export PATH=$SCRATCH/desitarget/bin:$PATH

export DESI_SPECTRO_SIM=$SCRATCH

It looks to me like specsim is handled a bit differently, so for that package I did the following

git clone https://github.com/desihub/specsim
cd specsim
python setup.py build
python setup.py install

Then I installed desispec's dependencies using the fitsio compatibility workaround

cd $SCRATCH/desispec
python -m pip install -r requirements.txt
python -m pip install -U astropy==5.0
python -m pip cache remove fitsio
python -m pip install --no-deps --force-reinstall --ignore-installed fitsio==1.1.6

cd to my directory with my data files. Finally:

pixsim --night 20171102 --expid 1 --cameras b0

which gives the traceback I posted at the top.

Am I doing something wrong in building my environment? Has pixsim or one of its dependencies changed? I'd appreciate any help troubleshooting this.

Thank you very much, Laurie

[sbailey]

Hi Laurie. It is failing on desisim/io.py line 1083:

        dirname = os.path.join(
            os.getenv('DESI_SPECTRO_SIM'), os.getenv('PIXPROD'),
            str(night), '{:08d}'.format(expid)
            )

I suspect that you don't have PIXSIM defined, since os.getenv('PIXPROD') will return None which then chokes os.path.join. It can be anything; it just defines a subdir under DESI_SPECTRO_SIM for where to put outputs of this particular run.

The code could be clearer about how it logs/crashes if required environment variables aren't defined, but I suspect that setting PIXSIM will fix your run.

[lastephey]

Yes, I was missing PIXPROD and a few other settings. I appreciate your quick suggestion. Thanks again and sorry for the noise.