Closed moustakas closed 1 year ago
I don't know if it's related to this ticket, but the 2.0.0
module is borked.
Again, on perlmutter:
source /global/cfs/cdirs/desi/software/desi_environment.sh 23.1
module load fastspecfit/2.0.0
fastspec -h
-bash: /global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/fastspecfit/2.0.0/bin/fastspec: python: bad interpreter: No such file or directory
Meanwhile, main
is fine:
module unload fastspecfit/2.0.0
module load fastspecfit/main
fastspec -h
usage: fastspec [-h] -o OUTFILE [--mp MP] [-n NTARGETS] [--firsttarget FIRSTTARGET] [--targetids TARGETIDS] [--solve-vdisp] [--no-broadlinefit]
[--nophoto] [--percamera-models] [--templates TEMPLATES] [--redrockfile-prefix REDROCKFILE_PREFIX] [--specfile-prefix SPECFILE_PREFIX]
[--qnfile-prefix QNFILE_PREFIX] [--mapdir MAPDIR] [--dr9dir DR9DIR] [--verbose]
[redrockfiles ...]
positional arguments:
redrockfiles Full path to input redrock file(s). (default: None)
[snip]
I tested that on my laptop I can pip-install fastspecfit/2.0.0
just fine (in a throw-away conda environment):
python -m pip install git+https://github.com/desihub/fastspecfit.git@2.0.0#egg=fastspecfit
I can also do (again, on my laptop):
cd /path/to/fastspecfit
python setup.py install
running install
/opt/homebrew/Caskroom/miniforge/base/envs/test-fastspecfit/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
/opt/homebrew/Caskroom/miniforge/base/envs/test-fastspecfit/lib/python3.10/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running bdist_egg
running egg_info
writing py/FastSpecFit.egg-info/PKG-INFO
writing dependency_links to py/FastSpecFit.egg-info/dependency_links.txt
writing top-level names to py/FastSpecFit.egg-info/top_level.txt
reading manifest file 'py/FastSpecFit.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'build'
no previously-included directories found matching 'sandbox'
adding license file 'LICENSE.rst'
writing manifest file 'py/FastSpecFit.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-11.0-arm64/egg
running install_lib
running build_py
copying py/fastspecfit/_version.py -> build/lib/fastspecfit
[snip]
Installing mpi-fastspecfit.sh script to /opt/homebrew/Caskroom/miniforge/base/envs/test-fastspecfit/bin
Installing jupyter-kernel.sh script to /opt/homebrew/Caskroom/miniforge/base/envs/test-fastspecfit/bin
Installing build-fsps-templates script to /opt/homebrew/Caskroom/miniforge/base/envs/test-fastspecfit/bin
Installed /opt/homebrew/Caskroom/miniforge/base/envs/test-fastspecfit/lib/python3.10/site-packages/FastSpecFit-2.0.0-py3.10.egg
Processing dependencies for FastSpecFit==2.0.0
Finished processing dependencies for FastSpecFit==2.0.0
@sbailey @weaverba137 do either of you have any suggestions for how to get 2.0.0
installed? I'm hoping to run a large fuji test sample as soon as perlmutter comes back from maintenance tomorrow.
I'm taking a look at this. I have one initial finding that might or might not be related. Although it appears that desimodules/23.1 (which is not checked in to GitHub, but separate issue) targets desiconda/20230111-2.1.0, the "current" symlink still points to 20220119-2.0.1. @sbailey, is this intentional?
@weaver thanks for the help. 23.1 is the environment for Iron, but I was waiting on committing and tagging to make sure we didn't have any last minute patches before committing to the new desiconda as the new current. If that link is causing problems for the install somehow, it would be fine to update it.
For some reason "python setup.py install" is replacing the fastspec/bin scripts' "#!/usr/bin/env python" shebang with "#!python" instead of "#!/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/conda/bin/python" like the other desi-installed scripts get. I don't see the underlying cause of that though.
I thought it might be something like that. I can dig further.
Are there any previous desiInstall
ed versions of fastspecfit that I can look at, whether on cori or perlmutter? Specifically tagged versions, not main
.
This used to work on cori. I haven't checked but @sbailey mentioned last night that it wasn't working either.
source /global/cfs/cdirs/desi/software/desi_environment.sh 22.5 module load fastspecfit/v1.0.1
[cori10 desimodules] module load fastspecfit/v1.0.1
[cori10 desimodules] fastspec -h
-bash: /global/common/software/desi/cori/desiconda/20211217-2.0.0/code/fastspecfit/v1.0.1/bin/fastspec: python: bad interpreter: No such file or directory
[cori10 desimodules] head -1 $(which fastspec)
#!python
I don't know why/how this previously seemed to work.
I think I have a clue: In the setup.py file:
setup_keywords['name'] = 'FastSpecFit'
and later:
setup_keywords['version'] = get_version(setup_keywords['name'])
However, py/FastSpecFit
does not exist, therefore it cannot determine the version string. I would fix that first, then see if desiInstall still installs scripts with the wrong #!
path.
I fixed the error in my setup.py and re-tagged 2.0.0
(I know tags are supposed to be immutable, but I don't want a small error like this blocking version 2.0.0
and I'm the only one using this release at the moment) but I can't overwrite the existing version.
As the desisproc
user on perlmutter, I get
desiInstall fastspecfit 2.0.0 -F
INFO:install.py:413:get_code:2023-01-26T04:06:12: Detected old working directory, /global/u2/d/desiproc/build/fastspecfit-2.0.0. Deleting...
Traceback (most recent call last):
File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desiutil/3.2.6/bin/desiInstall", line 6, in <module>
exit(main())
File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desiutil/3.2.6/lib/python3.10/site-packages/desiutil/install.py", line 1076, in main
status = di.run()
File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desiutil/3.2.6/lib/python3.10/site-packages/desiutil/install.py", line 1050, in run
self.set_install_dir()
File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desiutil/3.2.6/lib/python3.10/site-packages/desiutil/install.py", line 615, in set_install_dir
shutil.rmtree(self.install_dir)
File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/conda/lib/python3.10/shutil.py", line 724, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/conda/lib/python3.10/shutil.py", line 657, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/conda/lib/python3.10/shutil.py", line 680, in _rmtree_safe_fd
onerror(os.unlink, fullname, sys.exc_info())
File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/conda/lib/python3.10/shutil.py", line 678, in _rmtree_safe_fd
os.unlink(entry.name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'fastspecfit-html'
OK, if you're using desiproc
, then that explains it. We only ever install DESI software with the desi
collaboration account. Can you try it with that account?
Installing the latest tag on perlmutter via
results in the following warnings:
Task: figure out what all this means and how to make it go away.