search

sdss / lvmdrp

Local Volume Mapper (LVM) Data Reduction Pipeline
BSD 3-Clause "New" or "Revised" License
2 stars 0 forks source link

gaiaxpy #36

Closed kslong closed 6 months ago

kslong commented 10 months ago

I updated the lvmdrp today, and now regenerating the metadata is failing:

/Users/long/anaconda3/envs/lvmdrp/bin/drp:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  __import__('pkg_resources').require('lvmdrp==0.1.1.dev0')
[ERROR]: Traceback (most recent call last):
  File "/Users/long/anaconda3/envs/lvmdrp/bin/drp", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/Users/long/SDSS/lvmdrp/bin/drp", line 10, in <module>
    from lvmdrp.functions.run_drp import run_drp, get_config_options, reduce_file
  File "/Users/long/SDSS/lvmdrp/python/lvmdrp/functions/__init__.py", line 3, in <module>
    from . import (
  File "/Users/long/SDSS/lvmdrp/python/lvmdrp/functions/cubeMethod.py", line 17, in <module>
    from lvmdrp.external import ancillary_func
  File "/Users/long/SDSS/lvmdrp/python/lvmdrp/external/__init__.py", line 3, in <module>
    from . import ancillary_func, astrolib
  File "/Users/long/SDSS/lvmdrp/python/lvmdrp/external/ancillary_func.py", line 5, in <module>
    from gaiaxpy import calibrate
ModuleNotFoundError: No module named 'gaiaxpy'

I see how to pip install this, but did you mean to add it to the repository instead.

kslong commented 10 months ago

Well I tried pip install on a test environment, and that does not work so there must be more to it.

This is the commit that is failing : 36e248d85af9d3a3f335e2e6e1a0789f100bb9e2

ajmejia commented 10 months ago

Please show me the log of the failed reduction.

kslong commented 10 months ago

I am not sure what you mean, as I put the log in the dialogue of my initial message, but here it is again:


(lvmdrp) rainbow:lco long$ drp metadata regenerate -m 60216
/Users/long/anaconda3/envs/lvmdrp/bin/drp:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  __import__('pkg_resources').require('lvmdrp==0.1.1.dev0')
[ERROR]: Traceback (most recent call last):
  File "/Users/long/anaconda3/envs/lvmdrp/bin/drp", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/Users/long/SDSS/lvmdrp/bin/drp", line 10, in <module>
    from lvmdrp.functions.run_drp import run_drp, get_config_options, reduce_file
  File "/Users/long/SDSS/lvmdrp/python/lvmdrp/functions/__init__.py", line 3, in <module>
    from . import (
  File "/Users/long/SDSS/lvmdrp/python/lvmdrp/functions/cubeMethod.py", line 17, in <module>
    from lvmdrp.external import ancillary_func
  File "/Users/long/SDSS/lvmdrp/python/lvmdrp/external/__init__.py", line 3, in <module>
    from . import ancillary_func, astrolib
  File "/Users/long/SDSS/lvmdrp/python/lvmdrp/external/ancillary_func.py", line 5, in <module>
    from gaiaxpy import calibrate
ModuleNotFoundError: No module named 'gaiaxpy'```
kslong commented 10 months ago

OK, I figured it out at least partially. I should have installed GaiaXPy, which I have now done on a test branch.

This now gives:

(foo) rainbow:lco long$ drp metadata regenerate -m 60216
/Users/long/anaconda3/envs/foo/bin/drp:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  __import__('pkg_resources').require('lvmdrp==0.1.1.dev0')
[INFO]: loading metadata store at /Users/long/Projects/lvm_data/sas/sdsswork/lvm/spectro/redux/0.1.1.dev0/1111/60216/raw_metadata.hdf5
[INFO]: removing metadata store at /Users/long/Projects/lvm_data/sas/sdsswork/lvm/spectro/redux/0.1.1.dev0/1111/60216/raw_metadata.hdf5
[INFO]: Creating new metadata store.
[INFO]: going to extract metadata from 873 frames
extracting metadata:   3%|###2                                                                                                                    | 24/873 [00:03<01:55,  7.32frame/s][ERROR]: Cannot read FITS header: Empty or corrupt FITS file
extracting metadata:   5%|#####7                                                                                                                  | 42/873 [00:05<02:03,  6.70frame/s][ERROR]: Cannot read FITS header: Empty or corrupt FITS file
extracting metadata:  31%|#####################################                                                                                  | 272/873 [00:36<01:19,  7.52frame/s][ERROR]: Cannot read FITS header: Empty or corrupt FITS file
extracting metadata:  37%|###########################################4                                                                           | 319/873 [00:43<01:17,  7.19frame/s][ERROR]: Cannot read FITS header: Empty or corrupt FITS file
extracting metadata:  44%|####################################################4                                                                  | 385/873 [00:51<01:19,  6.17frame/s][ERROR]: Cannot read FITS header: Empty or corrupt FITS file
extracting metadata:  66%|##############################################################################7                                        | 578/873 [01:18<00:34,  8.56frame/s][ERROR]: Cannot read FITS header: Empty or corrupt FITS file
extracting metadata:  83%|##################################################################################################4                    | 722/873 [01:38<00:20,  7.47frame/s][ERROR]: Cannot read FITS header: Empty or corrupt FITS file
extracting metadata:  87%|#######################################################################################################3               | 758/873 [01:43<00:16,  7.10frame/s][ERROR]: Cannot read FITS header: Empty or corrupt FITS file
extracting metadata:  93%|##############################################################################################################8        | 813/873 [01:51<00:07,  8.05frame/s][ERROR]: Cannot read FITS header: Empty or corrupt FITS file
extracting metadata: 100%|#######################################################################################################################| 873/873 [01:59<00:00,  7.31frame/s]
[INFO]: creating metadata store at /Users/long/Projects/lvm_data/sas/sdsswork/lvm/spectro/redux/0.1.1.dev0/1111/60216/raw_metadata.hdf5
[INFO]: creating metadata store for tileid = 1111 and mjd = 60216 with 864 new rows
[INFO]: writing raw metadata store to disk
[INFO]: loading metadata store at /Users/long/Projects/lvm_data/sas/sdsswork/lvm/spectro/redux/0.1.1.dev0/1111/60216/raw_metadata.hdf5
[INFO]: updating metadata store for tileid = 1111 and mjd = 60216 with 0 new rows
[INFO]: final number of rows 864
[INFO]: writing raw metadata store to disk

Is that what you expected. If so, then the problem is that you need to add this module to the repository.

kslong commented 10 months ago

In looking, at the above, please make sure to scroll to the right in the box, as there are some Errors that say thinks like.

[ERROR]: Cannot read FITS header: Empty or corrupt FITS file

This is one of the things I am wondering about. At the beginning the routine says it si going to extract meta data from 873 files, at the end, it says 864 .

kslong commented 6 months ago

Although no one has responded to this message. this is no longer happening