Using --force-reinstall but new .pickle file still not picked up(?)

[svw26]

Hiya again again :)

Having got the main test to run through fine, I then followed each of the instructions given below:

rss_mos_create_arc_model product/mbxgpP202010130062.fits
coordsFileName = PG0900_Xe_2x4.txt
modelFileName = RefModel_PG0900_Xe_2x4.pickle
Coordinates file not found - you need to make one...
1. Use the plot window to record approximate pixel coords (x-axis) for spectral lines identified in a calibrated arc plot
   (see http://pysalt.salt.ac.za/lineatlas/lineatlas.html and look under the heading 'Arc Lamp Plots').
2. Save a text file (name it PG0900_Xe_2x4.txt) in the current directory, which has columns: Wavelengh (Angstroms, float), pixel coord (integer) on x-axis.
3. Close the plot window and re-run this script to generate the model.
4. Copy the model .pickle and .txt file to the RSSMOSPipeline/data/modelArcSpectra/ directory and re-run the setup.py script

For me (using a virtual conda environment), this meant moving the .pickle and .txt files to /Users/sarahwhite/anaconda/envs/rsspipeline_conda/lib/python3.5/site-packages/RSSMOSPipeline/data/modelArcSpectra . Because I installed RSSMOSPipeline via

pip install RSSMOSPipeline

I thought that an equivalent way of re-running the setup.py script would be to do

pip install RSSMOSPipeline --force-reinstall

Unfortunately this gives me the following:

Requirement already satisfied: RSSMOSPipeline in /Users/sarahwhite/anaconda/envs/rsspipeline_conda/lib/python3.5/site-packages
Requirement already satisfied: numpy>=1.10 in /Users/sarahwhite/anaconda/envs/rsspipeline_conda/lib/python3.5/site-packages (from RSSMOSPipeline)
Requirement already satisfied: scipy>=1.0 in /Users/sarahwhite/anaconda/envs/rsspipeline_conda/lib/python3.5/site-packages (from RSSMOSPipeline)
Requirement already satisfied: matplotlib>=2.0 in /Users/sarahwhite/anaconda/envs/rsspipeline_conda/lib/python3.5/site-packages (from RSSMOSPipeline)
Requirement already satisfied: astropy>=3.2 in /Users/sarahwhite/anaconda/envs/rsspipeline_conda/lib/python3.5/site-packages (from RSSMOSPipeline)
Requirement already satisfied: kiwisolver>=1.0.1 in /Users/sarahwhite/anaconda/envs/rsspipeline_conda/lib/python3.5/site-packages (from matplotlib>=2.0->RSSMOSPipeline)
Requirement already satisfied: cycler>=0.10 in /Users/sarahwhite/anaconda/envs/rsspipeline_conda/lib/python3.5/site-packages (from matplotlib>=2.0->RSSMOSPipeline)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /Users/sarahwhite/anaconda/envs/rsspipeline_conda/lib/python3.5/site-packages (from matplotlib>=2.0->RSSMOSPipeline)
Requirement already satisfied: python-dateutil>=2.1 in /Users/sarahwhite/anaconda/envs/rsspipeline_conda/lib/python3.5/site-packages (from matplotlib>=2.0->RSSMOSPipeline)
Requirement already satisfied: setuptools in /Users/sarahwhite/anaconda/envs/rsspipeline_conda/lib/python3.5/site-packages (from kiwisolver>=1.0.1->matplotlib>=2.0->RSSMOSPipeline)
Requirement already satisfied: six in /Users/sarahwhite/anaconda/envs/rsspipeline_conda/lib/python3.5/site-packages (from cycler>=0.10->matplotlib>=2.0->RSSMOSPipeline)
You are using pip version 9.0.1, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Then, when I re-run rss_mos_reducer product reduced all I get:

2020-10-16 09:16:30,392 - RSSMOSPipeline - INFO - started: 2020-10-16T09:16:30.392280
2020-10-16 09:16:30,392 - RSSMOSPipeline - INFO - parameters: Namespace(excludeMasks='', extensionsList='all', iterativeMethod=False, longslitThreshold=2.0, maskName='all', rawDir='product', reducedDir='reduced', skipDone=False, subFrac=0.8, threshold=0.1)
2020-10-16 09:16:30,392 - RSSMOSPipeline - INFO - Reading image headers (cache file location: product/imageInfo.pkl)
2020-10-16 09:16:30,395 - RSSMOSPipeline - WARNING - Skipping mask 'G4Jy 1657_PL0200N001' as no wavelength calibration reference model currently exists.
2020-10-16 09:16:30,395 - RSSMOSPipeline - INFO -     Use rss_mos_create_arc_model to make a reference model and then re-run.
2020-10-16 09:16:30,395 - RSSMOSPipeline - INFO -     arc files list: ['product/mbxgpP202010130062.fits']

which seems to confirm that it is not picking up the new .pickle and .txt files.

Now that I've written this out(!), I've had the idea to restart the container... so let me try that. Other suggestions also welcome.

Thanks, Sarah x

[svw26]

OK, the correct command to use is

pip install RSSMOSPipeline --upgrade --force-reinstall

which ends with

Successfully installed RSSMOSPipeline-0.1.0 astropy-3.2.3 cycler-0.10.0 kiwisolver-1.1.0 matplotlib-3.0.3 numpy-1.18.5 pyparsing-2.4.7 python-dateutil-2.8.1 scipy-1.4.1 setuptools-50.3.1 six-1.15.0
You are using pip version 9.0.1, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

I ran this after 11:14 this morning, and judging from the timestamps below, the new .pickle file still wasn't picked up:

Sarahs-MacBook-Pro:modelArcSpectra sarahwhite$ ls -ltr
total 1000
-rw-r--r--  1 sarahwhite  staff  13831 Oct 15 23:06 RefModel_PG0900_Xe_2x4.pickle
-rw-r--r--  1 sarahwhite  staff    207 Oct 16 11:17 PG0900_Ar_2x2.txt
-rw-r--r--  1 sarahwhite  staff    180 Oct 16 11:17 PG0900_Ar_2x2_1.txt
-rw-r--r--  1 sarahwhite  staff    539 Oct 16 11:17 PG0900_Ne_1x2.txt
-rw-r--r--  1 sarahwhite  staff    386 Oct 16 11:17 PG0900_Ne_2x2.txt
-rw-r--r--  1 sarahwhite  staff    515 Oct 16 11:17 PG0900_Ne_2x2_1.txt
-rw-r--r--  1 sarahwhite  staff    431 Oct 16 11:17 PG0900_Xe_2x2.txt
-rw-r--r--  1 sarahwhite  staff    359 Oct 16 11:17 PG1300_Ar_1x2.txt
-rw-r--r--  1 sarahwhite  staff    366 Oct 16 11:17 PG1300_Ar_2x2.txt
-rw-r--r--  1 sarahwhite  staff    290 Oct 16 11:17 PG1300_Ar_2x2_1.txt
-rw-r--r--  1 sarahwhite  staff    431 Oct 16 11:17 PG1300_Cu Ar_2x2.txt
-rw-r--r--  1 sarahwhite  staff    440 Oct 16 11:17 PG1300_Ne_1x2.txt
-rw-r--r--  1 sarahwhite  staff    794 Oct 16 11:17 PG1800_Ne_2x2.txt
-rw-r--r--  1 sarahwhite  staff    381 Oct 16 11:17 README
-rw-r--r--  1 sarahwhite  staff  32847 Oct 16 11:17 RefModel_PG0900_Ar_2x2.pickle
-rw-r--r--  1 sarahwhite  staff  33604 Oct 16 11:17 RefModel_PG0900_Ar_2x2_1.pickle
-rw-r--r--  1 sarahwhite  staff  27444 Oct 16 11:17 RefModel_PG0900_Ne_1x2.pickle
-rw-r--r--  1 sarahwhite  staff  34187 Oct 16 11:17 RefModel_PG0900_Ne_2x2.pickle
-rw-r--r--  1 sarahwhite  staff  35018 Oct 16 11:17 RefModel_PG0900_Ne_2x2_1.pickle
-rw-r--r--  1 sarahwhite  staff  33034 Oct 16 11:17 RefModel_PG0900_Xe_2x2.pickle
-rw-r--r--  1 sarahwhite  staff  26987 Oct 16 11:17 RefModel_PG1300_Ar_1x2.pickle
-rw-r--r--  1 sarahwhite  staff  34697 Oct 16 11:17 RefModel_PG1300_Ar_2x2.pickle
-rw-r--r--  1 sarahwhite  staff  33350 Oct 16 11:17 RefModel_PG1300_Ar_2x2_1.pickle
-rw-r--r--  1 sarahwhite  staff  35891 Oct 16 11:17 RefModel_PG1300_Cu Ar_2x2.pickle
-rw-r--r--  1 sarahwhite  staff  27115 Oct 16 11:17 RefModel_PG1300_Ne_1x2.pickle
-rw-r--r--  1 sarahwhite  staff  34415 Oct 16 11:17 RefModel_PG1800_Ne_2x2.pickle
-rw-r--r--  1 sarahwhite  staff    706 Oct 16 11:17 convertPickles.py
-rw-r--r--  1 sarahwhite  staff    540 Oct 16 11:17 fixPickles.py
-rw-r--r--  1 sarahwhite  staff   9048 Oct 16 11:17 fixRefModel.py
drwxr-xr-x  5 sarahwhite  staff    160 Oct 16 11:18 __pycache__
-rw-r--r--  1 sarahwhite  staff    155 Oct 16 11:34 PG0900_Xe_2x4.txt

(I removed a newline from PG0900_Xe_2x4.txt, hence the updated timestamp for that new file.) x

[mattyowl]

If you have copied the .txt and .pickle file to a directory where the code can find it and it's still not picked up, try deleting the product/imageInfo.pkl file and re-running the pipeline. It's a bit clunky and only checks for matching arc models the first time it scans the FITS headers in the product/ directory (it works like this so you can keep dumping data into product/ as it comes in, without forcing a re-build of imageInfo.pkl each time - it just scans images that aren't listed in imageInfo.pkl). The latest version in 'master' prints a warning message about deleting imageInfo.pkl if you add a new arc model.

[svw26]

Haha, yes, I couldn't find a imageInfo.pkl file in the raw directory but the product directory instead. No worries, but maybe I didn't get the warning message because the latest version in 'master' hasn't been released via PyPI yet? (Since I went the pip install route.)

Anyway, deleting that file and re-running makes lots of things happen, and I get many files and folders appearing in the 'reduced' directory, including 'diagnostics' where 5 'slits' have been detected per exposure. Awesome! :tada: And do I have permissions to commit my RefModel_PG0900_Xe_2x4.pickle and PG0900_Xe_2x4.txt files to a branch of this repository, or would you prefer I emailed them? x

[svw26]

(This was my first pass, so totally cool if you'd prefer me to do more quality-assessment first!) x