Point sources in oversampled images are MUCH more centrally-concentrated than the model PSF

benw1 commented 3 years ago

For example, if we compare an oversample=10 image to the output model PSF file:

The F062 model PSF shows an average central brightness of about 0.32, and average of 0.08 at +/- 1 pixel in X or Y, for a ratio of ~4:1 peak to adjacent pixels. The simulated image with oversample=10 nearest chip center shows a central pixel brightness of 161000 counts above background, while the four adjacent pixels have an average brightness of 12000 counts above background, for a ratio of ~13:1. This is vastly more centrally concentrated than the model.
The F184 model PSF shows a central brightness of about 0.18, and average of 0.07 at +/- 1 pixel in X or Y, for a ratio of ~2.5:1 peak to adjacent pixels. The simulated image with oversample=10 nearest chip center shows a central pixel brightness of 22000 counts above background, while the four adjacent pixels have an average brightness of 3000 counts above background, for a ratio of ~7:1. The extreme central concentration seen in the oversample=10 examples shown here are seen in all six filters.

Thus, there appears to be a problem in the routines that are adding the PSF to the pixel grid when oversampling.

york-stsci commented 3 years ago

There is currently active PSF development happening on a branch of my fork of STIPS. Note that this development is happening under the 2.x.y version line because it will be making substantial changes to the way that STIPS works in general. If you would like to see how things are going there, or test that work in development, please look at the psf_dev branch at https://github.com/york-stsci/STScI-STIPS-1/tree/psf_dev.

benw1 commented 3 years ago

Glad to know you're working on this. Thanks! I'll try that version and see if it looks better.

benw1 commented 3 years ago

When I run python setup.py install, at the end I get Finished processing dependencies for stips==1.0.8 Is there a way to be sure I've got the correct version?

benw1 commented 3 years ago

I'm also getting an error on import

import stips WARNING: LOCAL JWST PRD VERSION PRDOPSSOC-034 DOESN'T MATCH THE CURRENT ONLINE VERSION PRDOPSSOC-036 Please consider updating pysiaf, e.g. pip install --upgrade pysiaf or conda update pysiaf Traceback (most recent call last): File "", line 1, in File "/nobackupp11/bwilli24/STScI-STIPS-1/stips/init.py", line 10, in from .observation_module import ObservationModule File "/nobackupp11/bwilli24/STScI-STIPS-1/stips/observation_module/init.py", line 3, in from .observation_module import ObservationModule File "/nobackupp11/bwilli24/STScI-STIPS-1/stips/observation_module/observation_module.py", line 7, in from ..astro_image import AstroImage File "/nobackupp11/bwilli24/STScI-STIPS-1/stips/astro_image/init.py", line 4, in from .astro_image import AstroImage File "/nobackupp11/bwilli24/STScI-STIPS-1/stips/astro_image/astro_image.py", line 30, in from ..utilities import apply_ipc File "/nobackupp11/bwilli24/STScI-STIPS-1/stips/utilities/init.py", line 4, in from .utilities import StipsEnvironment File "/nobackupp11/bwilli24/STScI-STIPS-1/stips/utilities/utilities.py", line 651 elif use_cwd and (conf := GetConfigFile("cwd")) is not None: ^ SyntaxError: invalid syntax

york-stsci commented 3 years ago

That's my mistake. I forgot to update the required python version to 3.8. I should have an updated version of STIPS ready later this afternoon, but meanwhile if you install it on python 3.8 or higher it will work fine. I've also fixed that the version was being reported as 1.0.8 in setup.py and 2.0.0 in version.py, which is helpful to no one.

benw1 commented 3 years ago

Thanks! When I attempt to update to 3.8, I get the following: The following packages will be REMOVED:

esutil-0.6.4-py37hb3f55d8_0 healpy-1.13.0-py37hf476568_1 imagecodecs-lite-2019.12.3-py37h03ebfcd_1 jwxml-0.3.0-py37_1 photutils-0.7.2-py37h516909a_1 poppy-0.9.0-py37_0 pyfftw-0.12.0-py37hc3ef4f8_1 pysiaf-0.11.0-py37_0 pysynphot-0.9.14-py37_0 python_abi-3.7-1_cp37m stsci_rtd_theme-0.0.1-py37_0 webbpsf-0.9.0-py37_1

How are you able to get stips to work without these packages? Or do you need to install them by hand after the upgrade?

benw1 commented 3 years ago

The dependencies need to be improved greatly for install. Here's the list I needed to add by hand: 550 pip install photutils 552 pip install scipy 554 pip install webbpsf 558 pip install yaml 561 pip install pyyaml 563 pip install jwst_backgrounds 565 pip install esutil 567 pip install pandeia 570 pip install pandeia.engine

benw1 commented 3 years ago

Also, after a python setup.py install I only can import stips if I'm in the directory where I ran the install, so something about the install script seems to not be working right: Finished processing dependencies for stips==1.0.8 Now if I launch python and import stips it works: (stips2) -bash-4.3$ python Python 3.8.11 (default, Aug 3 2021, 15:09:35) [GCC 7.5.0] :: Anaconda, Inc. on linux Type "help", "copyright", "credits" or "license" for more information.

import stips WARNING: LOCAL JWST PRD VERSION PRDOPSSOC-034 DOESN'T MATCH THE CURRENT ONLINE VERSION PRDOPSSOC-036 Please consider updating pysiaf, e.g. pip install --upgrade pysiaf or conda update pysiaf

Now if I change directories and do the same thing, it doesn't work: (stips2) -bash-4.3$ cd .. (stips2) -bash-4.3$ python Python 3.8.11 (default, Aug 3 2021, 15:09:35) [GCC 7.5.0] :: Anaconda, Inc. on linux Type "help", "copyright", "credits" or "license" for more information. import stips Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'stips'

benw1 commented 3 years ago

Looks like it is not properly installing for several reasons:

(stips2) -bash-4.3$ python setup.py install running install running bdist_egg running egg_info writing stips.egg-info/PKG-INFO writing dependency_links to stips.egg-info/dependency_links.txt writing requirements to stips.egg-info/requires.txt writing top-level names to stips.egg-info/top_level.txt reading manifest file 'stips.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no previously-included files found matching '.pyc' warning: no previously-included files found matching '.pyo' warning: no files found matching 'README.rst' warning: no files found matching 'stips/tests/coveragerc' warning: no files found matching '.pyx' under directory 'stips' warning: no files found matching '.c' under directory 'stips' warning: no files found matching '.pxd' under directory 'stips' warning: no files found matching '' under directory 'cextern' warning: no files found matching '' under directory 'scripts' no previously-included directories found matching 'build' no previously-included directories found matching 'docs/_build' no previously-included directories found matching 'docs/api' warning: no previously-included files matching '.o' found anywhere in distribution writing manifest file 'stips.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib warning: install_lib: 'build/lib' does not exist -- no Python modules to install

york-stsci commented 3 years ago

Okay, it looks like there are two issues here. The first is that I didn't have setup.py actually installing STIPS itself, just its dependencies. I've fixed that. The second issue looks like you're not installing the psf_dev version. After you clone the repository, and before you run setup.py, did you remember to do "git checkout psf_dev" in the cloned directory?

benw1 commented 3 years ago

Thanks! Looks like I was on the right branch: (stips2) -bash-4.3$ git checkout psf_dev Already on 'psf_dev' Your branch is up to date with 'origin/psf_dev'. (stips2) -bash-4.3$ But I was behind: (stips2) -bash-4.3$ git pull remote: Enumerating objects: 29, done. remote: Counting objects: 100% (29/29), done. remote: Compressing objects: 100% (9/9), done. remote: Total 29 (delta 16), reused 29 (delta 16), pack-reused 0 Unpacking objects: 100% (29/29), 4.28 KiB | 20.00 KiB/s, done. From https://github.com/york-stsci/STScI-STIPS-1 de98fae..d4b004a psf_dev -> origin/psf_dev Updating de98fae..d4b004a Fast-forward .github/workflows/ci_workflows.yml | 10 +--------- .gitignore | 3 +++ environment.yml | 2 +- environment_dev.yml | 2 +- setup.cfg | 28 +++++++++++++++++++++------- setup.py | 18 +----------------- stips/astro_image/astro_image.py | 9 ++++++++- stips/instruments/instrument.py | 5 ++++- 8 files changed, 40 insertions(+), 37 deletions(-) (stips2) -bash-4.3$

After updating to the latest version of the branch, it seems to be installing properly!

benw1 commented 3 years ago

Also, after a python setup.py install I only can import stips if I'm in the directory where I ran the install, so something about the install script seems to not be working right: Finished processing dependencies for stips==1.0.8 Now if I launch python and import stips it works: (stips2) -bash-4.3$ python Python 3.8.11 (default, Aug 3 2021, 15:09:35) [GCC 7.5.0] :: Anaconda, Inc. on linux Type "help", "copyright", "credits" or "license" for more information.

import stips WARNING: LOCAL JWST PRD VERSION PRDOPSSOC-034 DOESN'T MATCH THE CURRENT ONLINE VERSION PRDOPSSOC-036 Please consider updating pysiaf, e.g. pip install --upgrade pysiaf or conda update pysiaf

Now if I change directories and do the same thing, it doesn't work: (stips2) -bash-4.3$ cd .. (stips2) -bash-4.3$ python Python 3.8.11 (default, Aug 3 2021, 15:09:35) [GCC 7.5.0] :: Anaconda, Inc. on linux Type "help", "copyright", "credits" or "license" for more information. import stips Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'stips'

benw1 commented 3 years ago

git pull 12:36 remote: Enumerating objects: 29, done. remote: Counting objects: 100% (29/29), done. remote: Compressing objects: 100% (9/9), done. remote: Total 29 (delta 16), reused 29 (delta 16), pack-reused 0 Unpacking objects: 100% (29/29), 4.28 KiB | 20.00 KiB/s, done. From https://github.com/york-stsci/STScI-STIPS-1 de98fae..d4b004a psf_dev -> origin/psf_dev Updating de98fae..d4b004a Fast-forward .github/workflows/ci_workflows.yml | 10 +--------- .gitignore | 3 +++ environment.yml | 2 +- environment_dev.yml | 2 +- setup.cfg | 28 +++++++++++++++++++++------- setup.py | 18 +----------------- stips/astro_image/astro_image.py | 9 ++++++++- stips/instruments/instrument.py | 5 ++++- 8 files changed, 40 insertions(+), 37 deletions(-)

Now it installs!

benw1 commented 3 years ago

However, it is not yet working, because it looks like I need a newer version of pandeia and webbpsf data than linked to the the installation instructions. Can you please point me to the version I need?

If I use the versions of the data files that work with the released version of STIPS, then I get: pandeia.engine.custom_exceptions.EngineInputError: Missing JSON file: /nobackup/bwilli24/STScI-STIPS/stips_data/pandeia_data-1.5.2rc2_roman/roman/wfi/config.json

york-stsci commented 3 years ago

I will need to look it up. Meanwhile, try, in python:

from stips.utilities import DownloadReferenceData DownloadReferenceData()

benw1 commented 3 years ago

Slick! Thanks for the help. That worked for getting beyond the stips_data error, but now I'm getting something a bit more cryptic about missing a keyword:

File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/stips-2.0.0-py3.8.egg/stips/observation_module/observation_module.py", line 228, in nextObservation self.instrument.reset(ra, dec, pa, filter, self.obs_count) File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/stips-2.0.0-py3.8.egg/stips/instruments/instrument.py", line 178, in reset self.resetDetectors(psf=psf) File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/stips-2.0.0-py3.8.egg/stips/instruments/instrument.py", line 203, in resetDetectors detector = AstroImage(parent=self, ra=ra, dec=dec, pa=pa, psf=psf, File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/stips-2.0.0-py3.8.egg/stips/astro_image/astro_image.py", line 79, in init self.zeropoint = self.parent.zeropoint File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/stips-2.0.0-py3.8.egg/stips/instruments/instrument.py", line 1114, in zeropoint return self.zeropoint_unit.value File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/stips-2.0.0-py3.8.egg/stips/instruments/instrument.py", line 1131, in zeropoint_unit bp = self.bandpass File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/stips-2.0.0-py3.8.egg/stips/instruments/instrument.py", line 1054, in bandpass i = self.pandeia_instrument File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/stips-2.0.0-py3.8.egg/stips/instruments/instrument.py", line 1109, in pandeia_instrument self._instrument = InstrumentFactory(config=conf) File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/pandeia/engine/instrument_factory.py", line 50, in InstrumentFactory cls = inst_map[instrument](mode=mode, config=config, webapp=webapp, kwargs) File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/pandeia/engine/roman.py", line 32, in init Instrument.init(self, telescope=telescope, mode=mode, config=config, kwargs) File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/pandeia/engine/instrument.py", line 124, in init self.exposure_spec = self.get_exposure_pars() File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/pandeia/engine/instrument.py", line 792, in get_exposure_pars ndrop2 = self.readout_pattern_config[readout_pattern]["ndrop2"] KeyError: 'ndrop2'

benw1 commented 3 years ago

My guess is that this may be because it is using pandeia 1.6, but the data from DownloadReferenceData() are pandeia 1.5.2 pandeia_refdata=/nobackup/bwilli24/STScI-STIPS-1/stips_data/pandeia_data-1.5.2rc2_roman

benw1 commented 3 years ago

Found a promising candidate at: https://stsci.app.box.com/v/pandeia-refdata-v1p6p2-roman

benw1 commented 3 years ago

With those data, I now get a new, very similar, error: File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/stips-2.0.0-py3.8.egg/stips/instruments/instrument.py", line 1109, in pandeia_instrument self._instrument = InstrumentFactory(config=conf) File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/pandeia/engine/instrument_factory.py", line 50, in InstrumentFactory cls = inst_map[instrument](mode=mode, config=config, webapp=webapp, kwargs) File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/pandeia/engine/roman.py", line 32, in init Instrument.init(self, telescope=telescope, mode=mode, config=config, kwargs) File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/pandeia/engine/instrument.py", line 124, in init self.exposure_spec = self.get_exposure_pars() File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/pandeia/engine/instrument.py", line 792, in get_exposure_pars nskip = self.readout_pattern_config[readout_pattern]["nskip"] KeyError: 'nskip'

benw1 commented 3 years ago

Ah... It looks like I had pandeia 1.6.2 installed, but STIPS only works with pandeia 1.6.
Now I have pandeia.engine==1.6 installed and the data from https://stsci.app.box.com/v/pandeia-refdata-v1p6-roman and it no error yet!

benw1 commented 3 years ago

Now I'm getting a STIPS error, related to sampling the PSF perhaps...:

2021-09-16 18:20:25,190 WARNING poppy CAUTION: Just interpolating rather than integrating filter profile, over 40 steps 2021-09-16 18:27:25,910 INFO: Calculating PSF at (4092.0,4092.0) 2021-09-16 18:27:25,910 INFO stips Calculating PSF at (4092.0,4092.0) 2021-09-16 18:27:25,910 WARNING poppy Pysynphot unavailable (or invalid source supplied)! Assuming flat # of counts versus wavelength. 2021-09-16 18:27:25,915 WARNING poppy CAUTION: Just interpolating rather than integrating filter profile, over 40 steps 2021-09-16 18:34:23,159 INFO: SCA01: (RA, DEC, PA) := (149.4, 69.2, 0.0), detected as (149.4, 69.2, 0.0) 2021-09-16 18:34:23,159 INFO stips SCA01: (RA, DEC, PA) := (149.4, 69.2, 0.0), detected as (149.4, 69.2, 0.0) 2021-09-16 18:34:23,161 INFO: Detector SCA01 created 2021-09-16 18:34:23,161 INFO stips Detector SCA01 created 2021-09-16 18:34:23,162 INFO: Reset Instrument 2021-09-16 18:34:23,162 INFO stips Reset Instrument 2021-09-16 18:34:23,162 INFO: Running catalogue /nobackupp11/bwilli24/Wpipes2/testgrid92/data/grid.cat/proc_grid_1psf/Mixed_grid_F062.tbl 2021-09-16 18:34:23,162 INFO stips Running catalogue /nobackupp11/bwilli24/Wpipes2/testgrid92/data/grid.cat/proc_grid_1psf/Mixed_grid_F062.tbl 2021-09-16 18:34:23,162 INFO: Adding catalogue /nobackupp11/bwilli24/Wpipes2/testgrid92/data/grid.cat/proc_grid_1psf/Mixed_grid_F062.tbl 2021-09-16 18:34:23,162 INFO stips Adding catalogue /nobackupp11/bwilli24/Wpipes2/testgrid92/data/grid.cat/proc_grid_1psf/Mixed_grid_F062.tbl 2021-09-16 18:34:23,270 INFO: Finished converting catalogue to internal format 2021-09-16 18:34:23,270 INFO stips Finished converting catalogue to internal format 2021-09-16 18:34:23,270 INFO: Adding catalogue to detector SCA01 2021-09-16 18:34:23,270 INFO stips Adding catalogue to detector SCA01 2021-09-16 18:34:23,271 INFO: Adding catalogue Mixed_grid_F062.tbl to AstroImage SCA01 2021-09-16 18:34:23,271 INFO stips Adding catalogue Mixed_grid_F062.tbl to AstroImage SCA01 2021-09-16 18:34:24,785 INFO: Determining pixel co-ordinates 2021-09-16 18:34:24,785 INFO stips Determining pixel co-ordinates 2021-09-16 18:34:24,816 INFO: Keeping 0 items 2021-09-16 18:34:24,816 INFO stips Keeping 0 items 2021-09-16 18:34:25,191 INFO: Determining pixel co-ordinates 2021-09-16 18:34:25,191 INFO stips Determining pixel co-ordinates 2021-09-16 18:34:25,223 INFO: Keeping 5872 items 2021-09-16 18:34:25,223 INFO stips Keeping 5872 items 2021-09-16 18:34:25,297 INFO: Writing 5872 stars 2021-09-16 18:34:25,297 INFO stips Writing 5872 stars 2021-09-16 18:34:25,297 INFO: Adding 5872 point sources to AstroImage SCA01 2021-09-16 18:34:25,297 INFO stips Adding 5872 point sources to AstroImage SCA01 Traceback (most recent call last): File "build/run_stips.py", line 83, in run_stips(this_event_id, this_dp_id, float(ra_dither), float(dec_dither)) File "build/run_stips.py", line 55, in run_stips source_count_catalogues = obm.addCatalogue(str(filename)) File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/stips-2.0.0-py3.8.egg/stips/observation_module/observation_module.py", line 254, in addCatalogue cats = self.instrument.addCatalogue(catalogue, self.id, *args, kwargs) File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/stips-2.0.0-py3.8.egg/stips/instruments/instrument.py", line 286, in addCatalogue cats.append(detector.addCatalogue(cat, dist=self.distortion, *args, *kwargs)) File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/stips-2.0.0-py3.8.egg/stips/astro_image/astro_image.py", line 595, in addCatalogue out_chunk = self.addTable(current_chunk, dist, args, kwargs) File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/stips-2.0.0-py3.8.egg/stips/astro_image/astro_image.py", line 532, in addTable self.addPoints(ids[stars_idx], xs[stars_idx], ys[stars_idx], File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/stips-2.0.0-py3.8.egg/stips/astro_image/astro_image.py", line 653, in addPoints img,ix,iy = self.makePoint(*values, psf_params, self.overrides, self.logger) File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/stips-2.0.0-py3.8.egg/stips/astro_image/astro_image.py", line 612, in makePoint psf_img = getReducedPsf(psf_params, x, y) File "/nobackup/bwilli24/miniconda3/envs/stips2/lib/python3.8/site-packages/stips-2.0.0-py3.8.egg/stips/utilities/utilities.py", line 953, in getReducedPsf psf_img = psf_img.reshape(psf_y//os, os, psf_x//os, os).sum(axis=(1, 3))[1:-1,1:-1] ValueError: cannot reshape array of size 72900 into shape (67,4,67,4)

york-stsci commented 3 years ago

Okay, I think there are a number of things going on here.

First, DownloadReferenceData() does a few things:

It sets up environment variables if they haven't been yet (for all the reference data directories)
It looks at the existing reference data directories to see if they exist
If they don't exist, it downloads the appropriate data

However, if the directory already exists, it leaves it alone. I'm making a fix so that it will, if possible, look for the version of the data directory, and make sure that it matches the installed data version.

With respect to the other error you ran into, can you please let me know what input parameters you used to get that error?

benw1 commented 3 years ago

Makes sense. It also only seems to work with pandeia 1.6, not 1.6.2, which is the default that gets installed by pip, so one must install pandeia.engine==1.6. Here is the code I used to set the parameters:

scene_general = {'ra': 149.4, 'dec': 69.2, 'pa': 0, 'seed': 123} obs = {'instrument': 'WFI', 'filters': F062, 'detectors': 1, 'distortion': False, 'pupil_mask': '', 'background': 'avg', 'observations_id': 35, 'exptime': 3600, 'offsets': [{'offset_id': 999, 'offset_centre': False, 'offset_ra': 0.0, 'offset_dec': 0.0, 'offset_pa': 0.0}]} obm = ObservationModule(obs, scene_general=scene_general, psf_grid_size=5, oversample=6) obm.nextObservation() source_count_catalogues = obm.addCatalogue('/nobackupp11/bwilli24/Wpipes2/testgrid92/data/grid.cat/proc_grid_1psf/Mixed_grid_F062.tbl') psf_file = obm.addError() fits_file, mosaic_file, params = obm.finalize(mosaic=False)

york-stsci commented 3 years ago

Note that the setup.cfg file specifies that pandeia must be version 1.6, as later versions are not compatible with Roman. I'm not sure how you ended up with a higher version installed if you did setup.py on a clean environment.

benw1 commented 3 years ago

Ah. I was trying all kinds of things to get past that error. I re-installed pandeia several times when I was doing all that, so it may have had the correct version initially, but just the wrong data file. Either way, it does work (other than my current psf error) when both the version and the data are 1.6.

benw1 commented 3 years ago

Interestingly, even though I specified oversample=6, it seems to have produced a psf at oversample 4: psf_v2.0.0dev_roman_wfi_sca01_f062_over4_grid5_epsf.fits

york-stsci commented 3 years ago

As far as I can tell, this is a result of the "oversample" keyword being changed to "psf_oversample", because oversampling now applies only to PSF creation, but some parts of the code accepting either 'oversample' or 'psf_oversample'. As a result, when the PSF was created, its oversample was set to the default (4.0), because it didn't recognize the keyword. However, when evaluated, the evaluation function saw 'oversample=6' and tried to bin the generated PSF from an oversample of 6 to 1, which naturally didn't work because the provided PSF wasn't at that binning to begin with.

I think that, if you do a "git pull", you will get revised code that fixes the issue. My tests are still ongoing, but have worked so far. Let me know.

benw1 commented 3 years ago

Thanks! OK, it is now making an oversample=6 psf file: 2021-09-17 12:09:03,380 INFO: PSF File is /nobackup/bwilli24/STScI-STIPS1/stips_data/psf_cache/psf_v2.0.0dev_roman_wfi_sca01_f062_over6_grid5_epsf.fits

However, when it imports I'm getting strange warnings:

import stips WARNING: LOCAL JWST PRD VERSION PRDOPSSOC-034 DOESN'T MATCH THE CURRENT ONLINE VERSION PRDOPSSOC-039 Please consider updating pysiaf, e.g. pip install --upgrade pysiaf or conda update pysiaf WARNING: stips: data directory not found. WARNING: synphot: data directory not found. WARNING: webbpsf: data directory not found. WARNING: pandeia: data directory not found.

These are odd because I didn't get them before and those data directories are defined in my environment:

stips_data=/nobackup/bwilli24/STScI-STIPS-1/stips_data PYSYN_CDBS=/nobackup/bwilli24/STScI-STIPS-1/stips_data/grp/hst/cdbs WEBBPSF_PATH=/nobackup/bwilli24/STScI-STIPS-1/stips_data/webbpsf-data pandeia_refdata=/nobackup/bwilli24/STScI-STIPS-1/stips_data/pandeia_data-1.6_roman

In any case, it is running, so we'll see if it generates proper images soon.

york-stsci commented 3 years ago

So, those reports are, in fact, incorrect, and are caused by a bug in the SetupDataPaths() function that was sometimes reporting directories as not being found even if they existed (this only happened if the environment variable was present, but the configuration file entry for the environment variable had a different name. In any case, there's now a fixed version, but the only effect should be the incorrect warnings being printed out.

benw1 commented 3 years ago

STIPS completes now; however, the image has no background, and the PSF is again far too sharp. Effectively all of the flux of the point sources is contained in a single pixel. Thus, this issue is still present on the psf_dev branch.

I'm guessing the background problem is another missing data issue of some sort, but not sure because the log doesn't mention where it is looking for the background info: 2021-09-17 15:09:01,273 INFO: Adding Error 2021-09-17 15:09:01,273 INFO stips Adding Error 2021-09-17 15:09:01,273 INFO: Adding residual error 2021-09-17 15:09:01,273 INFO stips Adding residual error 2021-09-17 15:09:01,470 INFO: SCA01: (RA, DEC, PA) := (0.0, 0.0, 0.0), detected as (0.0, 0.0, 0.0) 2021-09-17 15:09:01,470 INFO stips SCA01: (RA, DEC, PA) := (0.0, 0.0, 0.0), detected as (0.0, 0.0, 0.0) 2021-09-17 15:09:02,338 INFO: SCA01: (RA, DEC, PA) := (0.0, 0.0, 0.0), detected as (0.0, 0.0, 0.0) 2021-09-17 15:09:02,338 INFO stips SCA01: (RA, DEC, PA) := (0.0, 0.0, 0.0), detected as (0.0, 0.0, 0.0) 2021-09-17 15:09:02,338 INFO: Created AstroImage SCA01 and imported data from FITS file err_flat_wfi.fits 2021-09-17 15:09:02,338 INFO stips Created AstroImage SCA01 and imported data from FITS file err_flat_wfi.fits 2021-09-17 15:09:02,571 INFO: SCA01: (RA, DEC, PA) := (0.0, 0.0, 0.0), detected as (0.0, 0.0, 0.0) 2021-09-17 15:09:02,571 INFO stips SCA01: (RA, DEC, PA) := (0.0, 0.0, 0.0), detected as (0.0, 0.0, 0.0) 2021-09-17 15:09:03,569 INFO: SCA01: (RA, DEC, PA) := (0.0, 0.0, 0.0), detected as (0.0, 0.0, 0.0) 2021-09-17 15:09:03,569 INFO stips SCA01: (RA, DEC, PA) := (0.0, 0.0, 0.0), detected as (0.0, 0.0, 0.0) 2021-09-17 15:09:03,569 INFO: Created AstroImage SCA01 and imported data from FITS file err_rdrk_wfi.fits 2021-09-17 15:09:03,569 INFO stips Created AstroImage SCA01 and imported data from FITS file err_rdrk_wfi.fits 2021-09-17 15:09:03,585 INFO: Adding error to detector SCA01 2021-09-17 15:09:03,585 INFO stips Adding error to detector SCA01 2021-09-17 15:09:03,585 INFO: Adding background 2021-09-17 15:09:03,585 INFO stips Adding background 2021-09-17 15:09:03,585 INFO: Returning background 0.0. 2021-09-17 15:09:03,585 INFO stips Returning background 0.0. 2021-09-17 15:09:03,585 INFO: Background is 0.0 counts/s/pixel 2021-09-17 15:09:03,585 INFO stips Background is 0.0 counts/s/pixel 2021-09-17 15:09:03,586 INFO: Returning background 0.0. 2021-09-17 15:09:03,586 INFO stips Returning background 0.0. 2021-09-17 15:09:03,586 INFO: Added background of 0.0 counts/s/pixel 2021-09-17 15:09:03,586 INFO stips Added background of 0.0 counts/s/pixel 2021-09-17 15:09:03,610 INFO: Inserting correct exposure time 2021-09-17 15:09:03,610 INFO stips Inserting correct exposure time

york-stsci commented 3 years ago

The problem may be that the PSF image default fov is 45 detector pixels, but point source flux is currently normalized as if this FOV includes the entire enclosed energy. I have a development version which, I think, fixes this, and I'll let you know when that's available.

benw1 commented 3 years ago

I'm also noticing that with this version, the noise is identical in all simulated images, no matter what the SEED value is set to. In these tests, no background is being added, which would probably hide this problem.

benw1 commented 3 years ago

Also, this version is creating a new PSF file even when the appropriate PSF is already in the psf_cache/ directory.

york-stsci commented 3 years ago

A few things. First, what do you mean by noise? Because, of the noise types that get added by STIPS (if enabled) are:

Poisson Noise, which should vary depending on seed and flux in a given pixel
Readnoise, which should vary depending on seed and flux in a given pixel
Flatfield Residual, which is a fixed multiplier
Dark residual, which is a fixed multiplier
Cosmic Ray residual, which should vary depending on seed and flux in a given pixel
IPC, which should vary only based on flux in a given pixel

Second, by "appropriate" PSF, do you mean a PSF that has been generated with the psf_dev version of STIPS? Because if not, then there isn't actually an appropriate PSF already in the directory. If so, that's a bug and should be fixed.

benw1 commented 3 years ago

By noise, I mean the values of pixels in the resulting image that are away from the added point sources. That is probably the sum of all of the various components of added noise. Since the pattern is identical across all simulations, that may mean all of the individual components are also identical for all simulations.

By appropriate, I mean a PSF file that was just generated a few hours prior by the psf_dev version.

york-stsci commented 3 years ago

The pattern shouldn't be identical between simulations with different seeds. Please turn off dark and flatfield residuals and check.

Okay. I'll take a look at PSF generation.

benw1 commented 3 years ago

Here's an example log: 2021-09-20 09:49:45,438 INFO: Adding poisson noise 2021-09-20 09:49:45,438 INFO stips Adding poisson noise 2021-09-20 09:49:47,118 INFO: Adding Poisson Noise with mean -0.0004914749234851096 and standard deviation 7.569877257064649 2021-09-20 09:49:47,118 INFO stips Adding Poisson Noise with mean -0.0004914749234851096 and standard deviation 7.569877257064649 2021-09-20 09:49:47,162 INFO: Adding readnoise 2021-09-20 09:49:47,162 INFO stips Adding readnoise 2021-09-20 09:49:47,162 INFO: Adding Readnoise with mean per-pixel value 12.0 2021-09-20 09:49:47,162 INFO stips Adding Readnoise with mean per-pixel value 12.0 2021-09-20 09:49:48,513 INFO: Adding Read noise with mean 0.001802144449690447 and standard deviation 12.00455406208562 2021-09-20 09:49:48,513 INFO stips Adding Read noise with mean 0.001802144449690447 and standard deviation 12.00455406208562 2021-09-20 09:49:48,564 INFO: Adding flatfield residual 2021-09-20 09:49:48,564 INFO stips Adding flatfield residual 2021-09-20 09:49:48,680 INFO: Adding Flatfield residual with mean 1.0098040542150601 and standard deviation 0.0006764572806503836 2021-09-20 09:49:48,680 INFO stips Adding Flatfield residual with mean 1.0098040542150601 and standard deviation 0.0006764572806503836 2021-09-20 09:49:48,726 INFO: Adding dark residual 2021-09-20 09:49:48,726 INFO stips Adding dark residual 2021-09-20 09:49:48,843 INFO: Adding Dark residual with mean 2.496650471204527 and standard deviation 13.207552234371837 2021-09-20 09:49:48,843 INFO stips Adding Dark residual with mean 2.496650471204527 and standard deviation 13.207552234371837 2021-09-20 09:49:48,884 INFO: Adding cosmic ray residual 2021-09-20 09:49:48,884 INFO stips Adding cosmic ray residual 2021-09-20 09:49:59,526 INFO: Adding Cosmic Ray residual with mean 0.9995958252914828 and standard deviation 6.548997697943301 2021-09-20 09:49:59,526 INFO stips Adding Cosmic Ray residual with mean 0.9995958252914828 and standard deviation 6.548997697943301 2021-09-20 09:49:59,568 INFO: Finished adding error

benw1 commented 3 years ago

OK. Please instruct on how to turn off dark and flat field residuals, and I'll run that.

york-stsci commented 3 years ago

In your configuration file, set the residual_flat and residual_dark values to false.

york-stsci commented 3 years ago

So that log looks like STIPS is running two detectors at a time, which means they were created as part of the same observation, which in turn means that the detector's seed would have been identical, so np.random.RandomState(seed=self.seed) would have resulted in identical results. To get different seeds, you need to use different ObservationModule objects.

benw1 commented 3 years ago

I do not have a configure file. Can I set those parameters in the python call directly? If so, is it in the ObservationModule() call?
All of my simulations (including this one) have been single detector. I don't know how to make more than one detector, but that sounds like a great feature!

I have made many simulations with different seeds, oversamples and filters, but all have identical noise.

benw1 commented 3 years ago

Perhaps something is updated about how the seed works? Currently, I set it in the scene_general, like this: scene_general = {'ra': float(ra), 'dec': float(dec), 'pa': pa, 'seed': seed} which then gets passed to the ObservationModule() obm = ObservationModule(obs, scene_general=scene_general, psf_grid_size=5, oversample=1)

york-stsci commented 3 years ago

Yes, you can set things as keyword arguments when creating the ObservationModule. In particular,

residual_ are boolean arguments, where here would be "flat" and "dark"
seed must be set in the configuration file or as a direct keyword argument to ObservationModule, and I don't know whether "seed" is currently accepted, but "random_seed" is now and will continue to be accepted.

benw1 commented 3 years ago

Thanks! Using random_seed in the ObservationModule fixes the problem. However, it is a bit concerning that if I don't set that explicitly, every simulation gets identical noise.

york-stsci commented 3 years ago

That's intended by the design of STIPS. One of the design requirements was that, if you give it the same data with the same seed, it will produce the same observation. As such, the configuration file specifies a default random seed that will be used unless overridden.

benw1 commented 3 years ago

Thanks! Next I'm working on having it give me a non-zero background. Likely some background files need to be linked to be used by the new installation.

york-stsci commented 3 years ago

Again, this is done with the configuration file, or by direct keyword argument. 'observation_default_background' can be set to a floating-point value (in counts/s/pixel), or can be set to 'none' or 'avg', which gives values of either 0 or {'F062': 1.401E+00, 'F087': 1.401E+00, 'F106': 1.401E+00, 'F129': 7.000E-01, 'F158': 7.521E-01, 'F184': 8.500E-01, 'F146': 7.000E-01, 'F149': 7.000E-01}, again in counts/s/pixel.

benw1 commented 3 years ago

Thanks! The keyword used to be just 'background' : 'avg'. That keyword (set in the observation dictionary) actually still works, but I needed to have a background/ subdirectory in my stips_data/ directory, with all of the proper background files.

benw1 commented 3 years ago

Actually, scratch that. I was looking at the wrong file. I need that subdirectory and I need to set observation_default_background='avg' in the ObservationModule. However, I do have it working, so now the only issues are the recalculating of PSF files already in the psf_cache/, and the far too sharp point sources in the oversample>1 images.

benw1 commented 3 years ago

Also, did you happen to change the file structure of the output fits file? I have code that automatically grabs the 0th extension, which worked before but now seems to end up with an empty image. Thanks again for all of this help!

spacetelescope / STScI-STIPS

Point sources in oversampled images are MUCH more centrally-concentrated than the model PSF #142