Installation of latest version does not run because of poppy compatibility problem

[benw1]

Although I still do not see installation documentation, I followed the installation instructions at the end of Issue #2 .

conda create -n forSTIPS python=2.7.12 astropy=1.3.2 numpy=1.12.1 scipy=0.18.1 photutils=0.3.2 pysynphot=0.9.8.5 webbpsf=0.6 source activate forSTIPS echo $WEBBPSF_PATH export PYSYN_CDBS="/local/tmp/Work/bin/grp/hst/cdbs" pip install "esutil==0.6.0" pip install "montage-wrapper==0.9.9" pip install "jwst_backgrounds==1.1.1" pip install "pandeia.engine==1.0" export pandeia_refdata="/local/tmp/Work/bin/pandeia_data-1.0" cd /local/tmp/Work git clone https://github.com/spacetelescope/STScI-STIPS-UI.git export stips_data="/local/tmp/Work/STScI-STIPS-UI/sim_input/stips_data" git clone https://github.com/spacetelescope/STScI-STIPS.git cd STScI-STIPS python setup.py install

I am using straight copies of pandeia_data-1.0 and cdbs from earlier versions, but the stips data is from the clone of https://github.com/spacetelescope/STScI-STIPS-UI.git The install completes without error, but If I attempt to generate an observation, there is a compatability error with poppy. The current master and bugfix branches both result in the following error when attempting to run:

from stips.observation_module import ObservationModule obs = {'instrument': 'WFI', 'filters': ['H158'], 'detectors': 1, 'distortion': False, 'oversample': 10, 'pupil_mask': '', 'background': 'avg', 'observations_id': 1, 'exptime': 10000, 'offsets': [{'offset_id': 1, 'offset_centre': False, 'offset_ra': 0.5, 'offset_dec': 0.0, 'offset_pa': 27.0}]} scene_general = {'ra': 254.56583104, 'dec': 24.10779887, 'pa': 0.0, 'seed': 1234} obm = ObservationModule(obs, scene_general=scene_general) obm.nextObservation() 2018-09-24 15:43:40,042 INFO: Initializing Observation 0 of 1 [in /local/tmp/miniconda2/envs/forSTIPS/lib/python2.7/site-packages/stips-1.0.0rc6-py2.7.egg/stips/observation_module/observation_module.py:300] 2018-09-24 15:43:40,042 INFO: Observation Filter is H158 [in /local/tmp/miniconda2/envs/forSTIPS/lib/python2.7/site-packages/stips-1.0.0rc6-py2.7.egg/stips/observation_module/observation_module.py:300] 2018-09-24 15:43:40,042 INFO: Observation (RA,DEC) = (254.565970,24.107799) with PA=27.000000 [in /local/tmp/miniconda2/envs/forSTIPS/lib/python2.7/site-packages/stips-1.0.0rc6-py2.7.egg/stips/observation_module/observation_module.py:300] 2018-09-24 15:43:40,043 INFO: Resetting [in /local/tmp/miniconda2/envs/forSTIPS/lib/python2.7/site-packages/stips-1.0.0rc6-py2.7.egg/stips/instruments/instrument.py:926] 2018-09-24 15:43:43,110 INFO: PSF choosing between 215, 40960, and 204 [in /local/tmp/miniconda2/envs/forSTIPS/lib/python2.7/site-packages/stips-1.0.0rc6-py2.7.egg/stips/instruments/instrument.py:926] 2018-09-24 15:43:43,112 INFO: No source spectrum supplied, therefore defaulting to 5700 K blackbody [in /local/tmp/miniconda2/envs/forSTIPS/lib/python2.7/site-packages/poppy/instrument.py:825] 2018-09-24 15:43:43,112 INFO: Computing wavelength weights using synthetic photometry for H158... [in /local/tmp/miniconda2/envs/forSTIPS/lib/python2.7/site-packages/poppy/instrument.py:836] Traceback (most recent call last): File "", line 1, in File "/local/tmp/miniconda2/envs/forSTIPS/lib/python2.7/site-packages/stips-1.0.0rc6-py2.7.egg/stips/observation_module/observation_module.py", line 176, in nextObservation self.instrument.reset(ra, dec, pa, filter, self.obs_count) File "/local/tmp/miniconda2/envs/forSTIPS/lib/python2.7/site-packages/stips-1.0.0rc6-py2.7.egg/stips/instruments/instrument.py", line 127, in reset self.resetPSF() File "/local/tmp/miniconda2/envs/forSTIPS/lib/python2.7/site-packages/stips-1.0.0rc6-py2.7.egg/stips/instruments/wfi.py", line 63, in resetPSF psf = ins.calcPSF(oversample=self.oversample,fov_pixels=min(max_safe_size, max_ins_size, max_conv_size)) File "/local/tmp/miniconda2/envs/forSTIPS/lib/python2.7/site-packages/poppy/instrument.py", line 251, in calc_psf monochromatic=local_options['monochromatic']) File "/local/tmp/miniconda2/envs/forSTIPS/lib/python2.7/site-packages/poppy/instrument.py", line 837, in _get_weights band = self._get_synphot_bandpass(self.filter) File "/local/tmp/miniconda2/envs/forSTIPS/lib/python2.7/site-packages/poppy/instrument.py", line 752, in _get_synphot_bandpass bpname = self._synphot_bandpasses[filtername] AttributeError: 'WFI' object has no attribute '_synphot_bandpasses'

[benw1]

I have attempted to use an older version of the STScI-STIPS and STScI-STIPS-UI directories from a machine at UW with a working older version of STIPS, but the same poppy error occurs, which suggests something has been updated in poppy that makes it different. Perhaps there is a way to specify the poppy version that I'm missing...

[benw1]

Looks like we need poppy version 0.6.1. Because the version is not specified in the current suggested environment creation line, it installs version 0.7.0. I am attempting specifying 0.6.1 now to see if that fixes things.

[york-stsci]

I think that you might need to just create your conda environment, use pip to uninstall webbpsf and poppy, and then install poppy 0.6.1 and webbpsf 0.6.0 before running STIPS.

I'm looking into what might be causing the incompatibility, and hopefully will have a fix soon.

[benw1]

This works for getting things running on a head node. However, now I'm seeing that if I run on a cluster compute node, it fails trying to connect to the web. (Cluster compute nodes do not generally have internet connections). Is there a way to tell it not to use the internet? Why does it need the internet to run?

[york-stsci]

I suspect it's that it's using the JWST background tool to get a background spectrum for the specified co-ordinates.

[benw1]

Yes, that appears to be the case. I need to find a way to get jwst-backgrounds to use a local file cache. I put an issue into their GitHub page.

On Fri, Sep 28, 2018, 7:35 PM Brian York notifications@github.com wrote:

I suspect it's that it's using the JWST background tool to get a background spectrum for the specified co-ordinates.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/spacetelescope/STScI-STIPS/issues/18#issuecomment-425608945, or mute the thread https://github.com/notifications/unsubscribe-auth/AIdK8hlOyMNwwer9Qa_7rlMFdFQPAs_Yks5uftwfgaJpZM4W3lD4 .

[york-stsci]

For the moment, I've added the option to, in the observation dictionary, set the background string to 'custom'. If you do that, then the background count rate function will look for a key named 'custom_background' in the dictionary, and set the background value to that value (and never import jet or try to access the internet). In addition, whilst in the process of doing that, I found out that I was dividing the per-pixel background count rate by (oversample*oversample) both when I was calculating the background rate and when I was adding the background to the image. I've removed one of those, which might well also fix your issue with background count rate being dependent on oversample. All of this is in the same branch as the rotation fix for the moment.

[benw1]

Thanks Brian. It would be more appropriate to put the files jwst_backgrounds needs into a local spot, but at least we can try this as a workaround for now.

[benw1]

Perhaps I'm doing something wrong. I get an error on the keyword:

File "/gscratch/astro/benw1/miniconda3/envs/forSTIPS/lib/python2.7/site-packages/stips-1.0.0rc6-py2.7.egg/stips/observation_module/observation_module.py", line 131, in initParams 'Background Value: %s' % self.instrument.BGTEXT[self.background], KeyError: 'custom' (forSTIPS) [benw1@mox1 STScI-STIPS]$ git status

On branch square_pixel_fix

nothing to commit, working directory clean (forSTIPS) [benw1@mox1 STScI-STIPS]$ (forSTIPS) [benw1@mox1 STScI-STIPS]$ git pull Already up-to-date. (forSTIPS) [benw1@mox1 STScI-STIPS]$ Thoughts on how to get the correct code?

[benw1]

Here's the dictionary I'm using:

obs = {'instrument': 'WFI', 'filters': ['H158'], 'detectors': 1, 'distortion': False, 'oversample': 10, 'pupil_mask': '', 'background': 'custom', 'observations_id': 1, 'exptime': 10000, 'custom_background': 4000.0, 'offsets': [{'offset_id': 1, 'offset_centre': False, 'offset_ra': 0.0, 'offset_dec': 0.0, 'offset_pa': 0.0}]}

[york-stsci]

You weren't doing anything wrong. I forgot to add the revised instrument files to the commit.

[york-stsci]

It should be fixed now.

[benw1]

This is working so far, but I need to now find a way to put the correct value in for the custom background.

[york-stsci]

In your abs dictionary above, add the key "custom_background", with a value of whatever you want the background count rate to be (in counts/s/pixel).

[benw1]

Thanks! How do I determine what the correct value for my filter and exposure time to enter in order to properly simulate WFIRST data?

[york-stsci]

Ah, I'm not actually sure how to do that in general. If what you want is to have the same value as the JBT would give you, you can run the following ahead of time and put the values into your dictionary (sorry, I didn't immediately understand what you were asking, I had though you were asking how to tell STIPS what the filter and exposure time were, and I hadn't thought that i'd changed that).

import numpy as np
from jwst_backgrounds import jet
import pysynphot as ps
bg = jwst_background(ra, dec, photplam)
wave_array = bg.bkg_data['wave_array']
combined_bg_array = bg.bkg_data['total_bg']
flux_array = np.mean(combined_bg_array, axis=0)
flux_array_pixels = 1.e9 * flux_array * 2.3504e-11 * pixel_scale**2
sp = ps.ArraySpectrum(wave_array, flux_array_pixels, waveunits='micron', fluxunits='mjy')
sp.convert('angstroms')
sp.convert('photlam')
obs = ps.Observation(sp, bandpass, binset=sp.wave)
background_count_rate = obs.countrate()

where ra and dec are values you choose, the PHOTPLAM values are in the stips/instruments/wfi.py on line 142, pixel_scale is the WFI pixel scale (assumed by STIPS to be 0.11 arcsec/pixel), and bandpass is the filter bandpass (which you can get from STIPS using the Instrument class' bandpass property, or via pandeia).

[york-stsci]

Just as a note, I'm currently compiling a local cache of countrate values returned from jbt (basically I'm finding as many as possible of the unique RA and DEC values that it's measured, and I'll put together a count rate from each value). I'm also putting a function into stips.instrument.Instrument that checks whether or not STIPS can find an internet connection. For now, if it can't, it logs a warning and sets the background to 0.0 (note that it only does this check if the background value supplied isn't 'none' or 'custom'), but when there's an available cache it'll check that cache and get a count rate from it.

[benw1]

Thanks! It is possible to download the full background cache to your local machine. Instructions for downloading the background cache can be found at http://archive.stsci.edu/archive_news/2017/08-Aug/index.html#article1 So we can get a full set locally, but I am not sure how to use it...

[york-stsci]

I looked at that. Unfortunately, it doesn't seem as if there's any way to tell jwst_backgrounds to use that local path. As such, I'm currently working on downloading the cache, putting it in a local directory, and subclassing the jbt background class to let me feed it a local path as the cache URL. I'll test whether or not that works and, if it does, push the changes for you to look at (currently they depend on your downloading the cache data and putting it into your stips_data directory, with the cached files in stips_data/background/remote_cache and the jwst_backgrounds module built-in refdata directory copied into stips_data/background/jbt_refdata). If you want to set that up yourself, then it'll be ready for when I've finished my local testing. Hopefully jwst_backgrounds will add in the ability to override cache_url in some way at some point without this (rather hacked-together) solution.

[york-stsci]

The most recently pushed version (still in the square_pixel_fix branch) now allows you to use a local cache of the JBT background data. In order to use it, you must:

• Download everything from the page listed above, and put all of the numbered files (0000, 0001, .., 1966) into a directory named 'remote_cache'
• Put a file named VERSION into remote_cache. This file is not included in the downloads for some reason, but can be found at https://archive.stsci.edu/missions/jwst/simulations/straylight/sl_cache/VERSION
• Put 'remote_cache' into a directory named 'background'. Put 'background' into your stips_data directory at the root level
• In stips_data/background, put a directory named 'jbt_refdata'. In that directory, put a copy of the 'refdata' directory from the jwst_backgrounds python package.

[benw1]

Thanks! Do I then set the background back to 'avg'?

[york-stsci]

You can then set the background to any of the known values ('avg'/average, 'med'/median, 'max', or 'min'), and get hopefully exactly the same results that you would have gotten from the internet-based background calculation.

[benw1]

Working! Is there a way to turn down the screen output? It's very verbose at the moment.

[benw1]

Got it. Switched verbose to False in stips/astro_image/astro_image.py

[benw1]

With background set to 'avg' I'm getting zero background level. The noise is reasonable, but the mean is roughly 0. When I used the version that looked up the backgrounds using an internet connection, the mean was in the thousands, so something must be off...

[york-stsci]

What's the log output?

[benw1]

There's a lot of logging, but I see this:

2018-10-05 16:42:55,984 ERROR: Retrieving local cache produced error [Errno 2] No such file or directory: '/usr/lusers/benw1/data/STIPS/STScI-STIPS-UI/sim_input/stips_data/background/remote_cache/VERSION' [in /gscratch/astro/benw1/miniconda3/envs/forSTIPS/lib/python2.7/site-packages/stips-1.0.0rc6-py2.7.egg/stips/instruments/instrument.py:953]

I see my mistake. I named the directory remote-cache instead of remote_cache.

In the final step for setting up using the remote_cache, do you mean there should be a directory named 'refdata' inside of 'jbt_refdata' or that the contents of refdata should be placed into jbt_refdata?

[york-stsci]

The contents of refdata should be placed in jbt_refdata (well, copied in to).

[benw1]

Looking good now: slurm-353185.out:2018-10-08 13:20:28,951 INFO: Using local cache of JBT background data [in /gscratch/astro/benw1/miniconda3/envs/forSTIPS/lib/python2.7/site-packages/stips-1.0.0rc6-py2.7.egg/stips/instruments/instrument.py:953] slurm-353185.out:2018-10-08 13:20:29,130 INFO: Returning background 0.550724486357 for 'avg' [in /gscratch/astro/benw1/miniconda3/envs/forSTIPS/lib/python2.7/site-packages/stips-1.0.0rc6-py2.7.egg/stips/instruments/instrument.py:953]

[benw1]

The new backgrounds are looking reasonable, but the WCS of the output images have 1 degree pixels... CDELT1 = 1.0 / [deg] Coordinate increment at reference point CDELT2 = 1.0 / [deg] Coordinate increment at reference point So something is off with the WCS header keywords. Previous good looking images had CDELT1 = -3.0555555555556E-05 / [deg] Coordinate increment at reference point CDELT2 = 3.0555555555556E-05 / [deg] Coordinate increment at reference point

[york-stsci]

Sorry about not replying to this sooner, I've been on vacation for the past week. Do the new images have a CD matrix? If so, the presence of the CD matrix (I believe) overrides the CDELT keywords (in which case I should probably remove them).

[benw1]

They do not. They only have the CDELT keywords. I would actually prefer if they did have a CD matrix.

[york-stsci]

Okay. Please provide sample inputs and attach an output file.

[benw1]

Here you go

Mixed_fake_zero_Z.tbl.gz

[benw1]

The output file is too large to attach

[benw1]

You can get it here: http://faculty.washington.edu/benw1/sim_1_0.fits.gz

[york-stsci]

Thank you. I think I see what's going on, and it has to do with astropy automatically converting from a CD matrix to a PC matrix when writing out co-ordinates to a header, but not considering the scale properly (i.e. using CDELT to define the pixel scale, but not letting CDELT values be set after the CD matrix has been set). Take a look at the version I just pushed and see if it produces something more accurate (although it will still have CDELT values set to 1.0 because apparently astropy will always do that when turning a CD matrix into a PC matrix).

[benw1]

I updated to this version, but the headers still have no CD matrix: benw1@bacon(1369): imhead -full *fits | grep CD CDELT1 = 1.0 / [deg] Coordinate increment at reference point
CDELT2 = 1.0 / [deg] Coordinate increment at reference point
benw1@bacon(1369):

[york-stsci]

I'm sorry, I wasn't clear. When Astropy writes out a FITS header, it never includes the CD matrix, but instead includes the PC matrix. My question isn't whether there's a CD matrix now, but whether the PC matrix has changed so that, by applying the PC matrix to the CDELT values, the proper scale is preserved.

[benw1]

The scale has not changed from the previous version.

[york-stsci]

Going back to the origin of this particular issue thread, my local testing of the most recent version in the square_pixel_fix branch was done with poppy 0.7.0 and webbpsf 0.7.0 (and the webbpsf data files designed for use with version 0.7.0). At some point, please test this version of STIPS to see if the problem still exists once you have updated the webbpsf-data directory to the new release with version 0.7.0.

[york-stsci]

Another check on whether you're still getting crashes with webbpsf/poppy 0.7 (as long as you're using the 0.7 data files)?

[benw1]

To check this, I did a pip install "webbpsf==0.7.0" pip install "poppy==0.7.0"

After this update, the current STIPS version runs without error.

[york-stsci]

Okay. I'm closing this issue. If something shows up in the future, please re-open it.