WARNING: Built wheel for FastSpecFit is invalid

[moustakas]

Installing the latest tag on perlmutter via

source /global/cfs/cdirs/desi/software/desi_environment.sh 23.1
cd $HOME/build
desiInstall fastspecfit 2.0.0

results in the following warnings:

WARNING:install.py:845:install:2023-01-24T16:12:37: Pip emitted messages on STDERR; these can probably be ignored:
  WARNING: Built wheel for FastSpecFit is invalid: Metadata 1.2 mandates PEP 440 version, but 'unknown' is not
  DEPRECATION: FastSpecFit was installed using the legacy 'setup.py install' method, because a wheel could not 
be built for it. pip 23.1 will enforce this behaviour change. A possible replacement is to fix the wheel build 
issue reported above. Discussion can be found at https://github.com/pypa/pip/issues/8368

Task: figure out what all this means and how to make it go away.

[moustakas]

I don't know if it's related to this ticket, but the 2.0.0 module is borked.

Again, on perlmutter:

source /global/cfs/cdirs/desi/software/desi_environment.sh 23.1
module load fastspecfit/2.0.0
fastspec -h
-bash: /global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/fastspecfit/2.0.0/bin/fastspec: python: bad interpreter: No such file or directory

Meanwhile, main is fine:

module unload fastspecfit/2.0.0
module load fastspecfit/main
fastspec -h
usage: fastspec [-h] -o OUTFILE [--mp MP] [-n NTARGETS] [--firsttarget FIRSTTARGET] [--targetids TARGETIDS] [--solve-vdisp] [--no-broadlinefit]
                [--nophoto] [--percamera-models] [--templates TEMPLATES] [--redrockfile-prefix REDROCKFILE_PREFIX] [--specfile-prefix SPECFILE_PREFIX]
                [--qnfile-prefix QNFILE_PREFIX] [--mapdir MAPDIR] [--dr9dir DR9DIR] [--verbose]
                [redrockfiles ...]

positional arguments:
  redrockfiles          Full path to input redrock file(s). (default: None)
[snip]

I tested that on my laptop I can pip-install fastspecfit/2.0.0 just fine (in a throw-away conda environment):

python -m pip install git+https://github.com/desihub/fastspecfit.git@2.0.0#egg=fastspecfit

I can also do (again, on my laptop):

cd /path/to/fastspecfit
python setup.py install
running install
/opt/homebrew/Caskroom/miniforge/base/envs/test-fastspecfit/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/opt/homebrew/Caskroom/miniforge/base/envs/test-fastspecfit/lib/python3.10/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
running egg_info
writing py/FastSpecFit.egg-info/PKG-INFO
writing dependency_links to py/FastSpecFit.egg-info/dependency_links.txt
writing top-level names to py/FastSpecFit.egg-info/top_level.txt
reading manifest file 'py/FastSpecFit.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'build'
no previously-included directories found matching 'sandbox'
adding license file 'LICENSE.rst'
writing manifest file 'py/FastSpecFit.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-11.0-arm64/egg
running install_lib
running build_py
copying py/fastspecfit/_version.py -> build/lib/fastspecfit
[snip]
Installing mpi-fastspecfit.sh script to /opt/homebrew/Caskroom/miniforge/base/envs/test-fastspecfit/bin
Installing jupyter-kernel.sh script to /opt/homebrew/Caskroom/miniforge/base/envs/test-fastspecfit/bin
Installing build-fsps-templates script to /opt/homebrew/Caskroom/miniforge/base/envs/test-fastspecfit/bin

Installed /opt/homebrew/Caskroom/miniforge/base/envs/test-fastspecfit/lib/python3.10/site-packages/FastSpecFit-2.0.0-py3.10.egg
Processing dependencies for FastSpecFit==2.0.0
Finished processing dependencies for FastSpecFit==2.0.0

@sbailey @weaverba137 do either of you have any suggestions for how to get 2.0.0 installed? I'm hoping to run a large fuji test sample as soon as perlmutter comes back from maintenance tomorrow.

[weaverba137]

I'm taking a look at this. I have one initial finding that might or might not be related. Although it appears that desimodules/23.1 (which is not checked in to GitHub, but separate issue) targets desiconda/20230111-2.1.0, the "current" symlink still points to 20220119-2.0.1. @sbailey, is this intentional?

[sbailey]

@weaver thanks for the help. 23.1 is the environment for Iron, but I was waiting on committing and tagging to make sure we didn't have any last minute patches before committing to the new desiconda as the new current. If that link is causing problems for the install somehow, it would be fine to update it.

For some reason "python setup.py install" is replacing the fastspec/bin scripts' "#!/usr/bin/env python" shebang with "#!python" instead of "#!/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/conda/bin/python" like the other desi-installed scripts get. I don't see the underlying cause of that though.

[weaverba137]

I thought it might be something like that. I can dig further.

[weaverba137]

Are there any previous desiInstalled versions of fastspecfit that I can look at, whether on cori or perlmutter? Specifically tagged versions, not main.

[moustakas]

This used to work on cori. I haven't checked but @sbailey mentioned last night that it wasn't working either.

source /global/cfs/cdirs/desi/software/desi_environment.sh 22.5 module load fastspecfit/v1.0.1

[sbailey]

[cori10 desimodules] module load fastspecfit/v1.0.1
[cori10 desimodules] fastspec -h
-bash: /global/common/software/desi/cori/desiconda/20211217-2.0.0/code/fastspecfit/v1.0.1/bin/fastspec: python: bad interpreter: No such file or directory
[cori10 desimodules] head -1 $(which fastspec)
#!python

I don't know why/how this previously seemed to work.

[weaverba137]

I think I have a clue: In the setup.py file:

setup_keywords['name'] = 'FastSpecFit'

and later:

setup_keywords['version'] = get_version(setup_keywords['name'])

However, py/FastSpecFit does not exist, therefore it cannot determine the version string. I would fix that first, then see if desiInstall still installs scripts with the wrong #! path.

[moustakas]

I fixed the error in my setup.py and re-tagged 2.0.0 (I know tags are supposed to be immutable, but I don't want a small error like this blocking version 2.0.0 and I'm the only one using this release at the moment) but I can't overwrite the existing version.

As the desisproc user on perlmutter, I get

desiInstall fastspecfit 2.0.0 -F
INFO:install.py:413:get_code:2023-01-26T04:06:12: Detected old working directory, /global/u2/d/desiproc/build/fastspecfit-2.0.0. Deleting...
Traceback (most recent call last):
  File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desiutil/3.2.6/bin/desiInstall", line 6, in <module>
    exit(main())
  File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desiutil/3.2.6/lib/python3.10/site-packages/desiutil/install.py", line 1076, in main
    status = di.run()
  File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desiutil/3.2.6/lib/python3.10/site-packages/desiutil/install.py", line 1050, in run
    self.set_install_dir()
  File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desiutil/3.2.6/lib/python3.10/site-packages/desiutil/install.py", line 615, in set_install_dir
    shutil.rmtree(self.install_dir)
  File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/conda/lib/python3.10/shutil.py", line 724, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/conda/lib/python3.10/shutil.py", line 657, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/conda/lib/python3.10/shutil.py", line 680, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/conda/lib/python3.10/shutil.py", line 678, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'fastspecfit-html'

[weaverba137]

OK, if you're using desiproc, then that explains it. We only ever install DESI software with the desi collaboration account. Can you try it with that account?