Closed svw26 closed 3 years ago
Hi Sarah,
I haven't put a new release on PyPI in ages (sorry), so the version there shouldn't have a Xe arc.
Do you have this problem when using master? Perhaps there is some confusion from clashing python installations (?), making it look for the arc models in the data directory from a different installation?
Cheers Matt
OK, let me investigate with a return to master.
Re: where the arc models are located, each conda environment has its own directory, and the contents are the same for both (see below). However, it may be that I've overwritten the installation in the first environment by calling 'pip install' (I can't quite remember now...) x
Sarahs-MacBook-Pro-2:envs sarahwhite$ cd /Users/sarahwhite/anaconda/envs/rsspipeline_conda/lib/python3.5/site-packages/RSSMOSPipeline/data/modelArcSpectra/
Sarahs-MacBook-Pro-2:modelArcSpectra sarahwhite$ ls
PG0900_Ar_2x2.txt PG1300_Ne_1x2.txt RefModel_PG1300_Ar_2x2.pickle
PG0900_Ar_2x2_1.txt PG1800_Ne_2x2.txt RefModel_PG1300_Ar_2x2_1.pickle
PG0900_Ne_1x2.txt README RefModel_PG1300_Cu Ar_2x2.pickle
PG0900_Ne_2x2.txt RefModel_PG0900_Ar_2x2.pickle RefModel_PG1300_Ne_1x2.pickle
PG0900_Ne_2x2_1.txt RefModel_PG0900_Ar_2x2_1.pickle RefModel_PG1800_Ne_2x2.pickle
PG0900_Xe_2x2.txt RefModel_PG0900_Ne_1x2.pickle __pycache__
PG1300_Ar_1x2.txt RefModel_PG0900_Ne_2x2.pickle convertPickles.py
PG1300_Ar_2x2.txt RefModel_PG0900_Ne_2x2_1.pickle fixPickles.py
PG1300_Ar_2x2_1.txt RefModel_PG0900_Xe_2x2.pickle fixRefModel.py
PG1300_Cu Ar_2x2.txt RefModel_PG1300_Ar_1x2.pickle
Sarahs-MacBook-Pro-2:modelArcSpectra sarahwhite$ cd /Users/sarahwhite/anaconda/envs/rssmospipeline_conda/lib/python3.5/site-packages/RSSMOSPipeline/data/modelArcSpectra/
Sarahs-MacBook-Pro-2:modelArcSpectra sarahwhite$ ls
PG0900_Ar_2x2.txt PG1300_Ne_1x2.txt RefModel_PG1300_Ar_2x2.pickle
PG0900_Ar_2x2_1.txt PG1800_Ne_2x2.txt RefModel_PG1300_Ar_2x2_1.pickle
PG0900_Ne_1x2.txt README RefModel_PG1300_Cu Ar_2x2.pickle
PG0900_Ne_2x2.txt RefModel_PG0900_Ar_2x2.pickle RefModel_PG1300_Ne_1x2.pickle
PG0900_Ne_2x2_1.txt RefModel_PG0900_Ar_2x2_1.pickle RefModel_PG1800_Ne_2x2.pickle
PG0900_Xe_2x2.txt RefModel_PG0900_Ne_1x2.pickle __pycache__
PG1300_Ar_1x2.txt RefModel_PG0900_Ne_2x2.pickle convertPickles.py
PG1300_Ar_2x2.txt RefModel_PG0900_Ne_2x2_1.pickle fixPickles.py
PG1300_Ar_2x2_1.txt RefModel_PG0900_Xe_2x2.pickle fixRefModel.py
PG1300_Cu Ar_2x2.txt RefModel_PG1300_Ar_1x2.pickle
[Both directories have PG0900_Xe_2x2.txt and RefModel_PG0900_Xe_2x2.pickle...]
Hmm, so I've created a new installation (within its own environment), based on master, by doing the following:
conda create --name rssmospipeline_master python=3.5
conda activate rssmospipeline_master
conda install numpy scipy astropy matplotlib IPython
pip install git+https://github.com/mattyowl/RSSMOSPipeline
pip install cython
conda uninstall parso
conda install ipython=6.5.0 jedi=0.12.1 parso=0.7
The contents of the corresponding modelArcSpectra directory are as follows:
Sarahs-MacBook-Pro-2:modelArcSpectra sarahwhite$ pwd
/Users/sarahwhite/anaconda/envs/rssmospipeline_master/lib/python3.5/site-packages/RSSMOSPipeline/data/modelArcSpectra
Sarahs-MacBook-Pro-2:modelArcSpectra sarahwhite$ ls
PG0900_Ar_2x2.txt PG1300_Ar_2x2_1.txt RefModel_PG0900_Ne_2x2.pickle RefModel_PG1300_Xe_1x2.pickle
PG0900_Ar_2x2_1.txt PG1300_Cu Ar_2x2.txt RefModel_PG0900_Ne_2x2_1.pickle RefModel_PG1800_Ne_2x2.pickle
PG0900_Ne_1x2.txt PG1300_Ne_1x2.txt RefModel_PG0900_Xe_1x2.pickle __pycache__
PG0900_Ne_2x2.txt PG1300_Xe_1x2.txt RefModel_PG0900_Xe_2x2.pickle convertPickles.py
PG0900_Ne_2x2_1.txt PG1800_Ne_2x2.txt RefModel_PG1300_Ar_1x2.pickle fixPickles.py
PG0900_Xe_1x2.txt README RefModel_PG1300_Ar_2x2.pickle fixRefModel.py
PG0900_Xe_2x2.txt RefModel_PG0900_Ar_2x2.pickle RefModel_PG1300_Ar_2x2_1.pickle
PG1300_Ar_1x2.txt RefModel_PG0900_Ar_2x2_1.pickle RefModel_PG1300_Cu Ar_2x2.pickle
PG1300_Ar_2x2.txt RefModel_PG0900_Ne_1x2.pickle RefModel_PG1300_Ne_1x2.pickle
When I run the pipeline over a target from Semester 2021-1 (G4Jy 1268) I get:
(rssmospipeline_master) Sarahs-MacBook-Pro-2:2020-1-MLT-008.20210516_G4Jy1268 sarahwhite$ rss_mos_reducer product reduced all
2021-06-02 14:44:49,156 - RSSMOSPipeline - INFO - started: 2021-06-02T14:44:49.156096
2021-06-02 14:44:49,156 - RSSMOSPipeline - INFO - parameters: Namespace(excludeMasks='', extensionsList='all', iterativeMethod=False, longslitThreshold=2.0, maskName='all', rawDir='product', reducedDir='reduced', skipDone=False, subFrac=0.8, threshold=0.1)
2021-06-02 14:44:49,156 - RSSMOSPipeline - INFO - Reading image headers (cache file location: product/imageInfo.pkl)
2021-06-02 14:44:49,341 - RSSMOSPipeline - WARNING - Skipping mask 'G4Jy 1268_PL0200N001' as no wavelength calibration reference model currently exists.
2021-06-02 14:44:49,341 - RSSMOSPipeline - INFO - Use rss_mos_create_arc_model to make a reference model and then re-run (after deleting product/imageInfo.pkl).
2021-06-02 14:44:49,341 - RSSMOSPipeline - INFO - arc files list: ['product/mbxgpP202105160040.fits']
Using the same conda environment, if I go back to the target that worked when I emailed you on 24th March (G4Jy 552), I get:
(rssmospipeline_master) Sarahs-MacBook-Pro-2:2020-1-MLT-008.20210218_G4Jy_552 sarahwhite$ rss_mos_reducer product reduced_master all
2021-06-02 15:02:19,087 - RSSMOSPipeline - INFO - started: 2021-06-02T15:02:19.087165
2021-06-02 15:02:19,087 - RSSMOSPipeline - INFO - parameters: Namespace(excludeMasks='', extensionsList='all', iterativeMethod=False, longslitThreshold=2.0, maskName='all', rawDir='product', reducedDir='reduced_master', skipDone=False, subFrac=0.8, threshold=0.1)
2021-06-02 15:02:19,087 - RSSMOSPipeline - INFO - Reading image headers (cache file location: product/imageInfo.pkl)
2021-06-02 15:02:19,089 - RSSMOSPipeline - INFO - Mask: G4Jy 552_PL0200N001
2021-06-02 15:02:19,113 - RSSMOSPipeline - INFO - Making masterFlats (it is a good idea to check alignment with object spectra at the cutting stage, and remove any flats which aren't aligned)
2021-06-02 15:02:19,113 - RSSMOSPipeline - INFO - making reduced_master/G4Jy 552_PL0200N001/masterFlat_0.fits (['product/mbxgpP202102180055.fits', 'product/mbxgpP202102180056.fits', 'product/mbxgpP202102180057.fits', 'product/mbxgpP202102180058.fits', 'product/mbxgpP202102180059.fits'])
2021-06-02 15:02:23,514 - RSSMOSPipeline - INFO - It is a good idea to check from the corresponding DS9 .reg file that object spectra are actually centred in the pseudo-slits
2021-06-02 15:02:23,515 - RSSMOSPipeline - INFO - cutting product/mbxgpP202102180051.fits (and arcs, flats) using product/mbxgpP202102180051.fits for slit definition ...
2021-06-02 15:02:23,722 - RSSMOSPipeline - INFO - cutting product/mbxgpP202102180052.fits (and arcs, flats) using product/mbxgpP202102180052.fits for slit definition ...
2021-06-02 15:02:23,933 - RSSMOSPipeline - INFO - cutting product/mbxgpP202102180053.fits (and arcs, flats) using product/mbxgpP202102180053.fits for slit definition ...
2021-06-02 15:02:24,147 - RSSMOSPipeline - INFO - Applying flat field
2021-06-02 15:02:24,568 - RSSMOSPipeline - INFO - Finding 2d wavelength solution
2021-06-02 15:02:24,568 - RSSMOSPipeline - INFO - arc = reduced_master/G4Jy 552_PL0200N001/cmbxgpP202102180053mbxgpP202102180054.fits
2021-06-02 15:02:24,575 - RSSMOSPipeline - INFO - extension = SLIT1
2021-06-02 15:02:24,936 - RSSMOSPipeline - INFO - using refModel = /Users/sarahwhite/anaconda/envs/rssmospipeline_master/lib/python3.5/site-packages/RSSMOSPipeline/data/modelArcSpectra/RefModel_PG0900_Xe_2x2.pickle
...
2021-06-02 15:06:04,929 - RSSMOSPipeline - INFO - SLIT4
2021-06-02 15:06:04,960 - RSSMOSPipeline - INFO - reduced_master/G4Jy 552_PL0200N001/rwcmbxgpP202102180051.fits
2021-06-02 15:06:12,228 - RSSMOSPipeline - INFO - reduced_master/G4Jy 552_PL0200N001/rwcmbxgpP202102180052.fits
2021-06-02 15:06:19,340 - RSSMOSPipeline - INFO - reduced_master/G4Jy 552_PL0200N001/rwcmbxgpP202102180053.fits
2021-06-02 15:06:26,142 - RSSMOSPipeline - INFO - individual frame - sky wavelength calib check: medianOffset = 0.157 Angstroms, numLines = 5
2021-06-02 15:06:26,155 - RSSMOSPipeline - INFO - individual frame - sky wavelength calib check: medianOffset = -0.434 Angstroms, numLines = 5
2021-06-02 15:06:26,170 - RSSMOSPipeline - INFO - individual frame - sky wavelength calib check: medianOffset = 0.547 Angstroms, numLines = 5
2021-06-02 15:06:26,252 - RSSMOSPipeline - INFO - extractAndStack - sky wavelength calib check: medianOffset = 0.532 Angstroms, numLines = 5
2021-06-02 15:06:32,811 - RSSMOSPipeline - INFO - stackAndExtract - sky wavelength calib check: medianOffset = 0.532 Angstroms, numLines = 5
I thought that was weird, so I've gone back to the folder for G4Jy 1268 and have run the command rss_mos_reducer product reduced_master all
(in case the imageInfo.pkl file needed to be in place for some reason). This gives me the same "no wavelength calibration reference model currently exists" message as before :(
Shall I email you a copy of the data for G4Jy 1268? x
Sorry for the delay - I think this is a dumb bug (in which case sorry for wasting your time) that I'll sort out just now.
Please can you give this a go with the updated version in master?
(No worries -- I'm slower at responding!)
Within my 'rssmospipeline_master' conda environment I ran pip install git+https://github.com/mattyowl/RSSMOSPipeline
, which ended with:
Successfully built RSSMOSPipeline
Installing collected packages: RSSMOSPipeline
Found existing installation: RSSMOSPipeline 0.1.0+11.gf8215b1
Uninstalling RSSMOSPipeline-0.1.0+11.gf8215b1:
Successfully uninstalled RSSMOSPipeline-0.1.0+11.gf8215b1
Successfully installed RSSMOSPipeline-0.1.0+12.ge887f8d
You are using pip version 10.0.1, however version 20.3.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Unfortunately, when I run the pipeline, it now takes me into IPython:
(rssmospipeline_master) Sarahs-MacBook-Pro-2:2020-1-MLT-008.20210516_G4Jy1268 sarahwhite$ rss_mos_reducer product reduced_master all
2021-06-15 10:03:10,112 - RSSMOSPipeline - INFO - started: 2021-06-15T10:03:10.112426
2021-06-15 10:03:10,112 - RSSMOSPipeline - INFO - parameters: Namespace(excludeMasks='', extensionsList='all', iterativeMethod=False, longslitThreshold=2.0, maskName='all', rawDir='product', reducedDir='reduced_master', skipDone=False, subFrac=0.8, threshold=0.1)
2021-06-15 10:03:10,112 - RSSMOSPipeline - INFO - Reading image headers (cache file location: product/imageInfo.pkl)
Python 3.5.6 |Anaconda, Inc.| (default, Aug 26 2018, 16:30:03)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.5.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]:
Sorry - I've removed the stray IPython. However, that coming up means it's probably still triggering the same problem for you. Hmm. Could you maybe put the data somewhere for me to check to see if I can get it to run? Sorry this is being such a pain.
Okiedoke, will email x
Yay, I have:
Hello, me again...
I have checked what models are available for my new conda environment:
The grating for this target is PG0900, and the lamp is Xe, so I was expecting RefModel_PG0900_Xe_2x2.pickle to be used (if the slit-length binning is to be ignored).
So something appears to have changed between the version of master I'd used on 24th March and the version through PyPI? (I'm also not sure why this run manages to get further along before complaining about no reference model.)
Perhaps the log messages could include the location of the modelArcSpectra folder that is being checked? I'm not sure if something is not linking up anymore.
Thanks, Sarah x