Add centroid file capability.

cwwalter commented 6 years ago

This PR adds centroid capability and is driven by imSim code. There aren't that many changes but the part that opens, closes and writes to files is here in sims_GalSimInterface. See https://github.com/LSSTDESC/imSim/pull/112. More information is copied from that PR below including current limitations.

I am working on adding more pixel level truth to the simulation but thought it would be useful to first add basic 'centroid file' capability.

My work on adding information based on what each object looks like on the sensor is currently adding a lot of CPU run time and I need more work to optimize it. So, this initial PR uses only truth information about the objects in the centroid file. This makes it more difficult to use this to do imSim/PhoSim validation comparisons (because the PhoSim centroid values are average X and Y values on the sensor) but it is better for comparing catalog truth with the sensor since the true values are not affected by being near the edge of the sensor etc.

This code will make centroid files corresponding to each fits file in the output directory. For example:

wasabi:fits % ls
centroid_197356_R22_S01_r.txt   lsst_e_197356_R22_S01_r.fits
centroid_197356_R22_S10_r.txt   lsst_e_197356_R22_S10_r.fits
centroid_197356_R22_S11_r.txt   lsst_e_197356_R22_S11_r.fits
centroid_197356_R22_S12_r.txt   lsst_e_197356_R22_S12_r.fits
centroid_197356_R22_S21_r.txt   lsst_e_197356_R22_S21_r.fits

With a small piece of code this can be converted into a ds9 region file for overlay. This pandas code example puts a circle at each true object position with the radius scaled by the log of the number of photons for one of the centroid files:

import math
import numpy as np
import pandas as pd

centroids = pd.read_csv('fits/centroid_197356_R22_S11_r.txt', delim_whitespace=True, comment="#")

with open('region_file.reg','w') as outfile:
    outfile.write('physical\n')
    centroids.query('Photons>0').to_string(outfile, columns=['xpos','ypos','Photons'], 
                                           header=False, index=False,
                                           formatters=['circle {:10.3f}'.format,
                                                       '{:10.3f}'.format,
                                                       lambda x: '{:10.3f}'.format(math.log(x))])

Loading this regions file after loading the fits file results in:

To my eye, there is a small shift between the circle centers and the objects. I'm not sure if this is a plotting issue, a misunderstanding on my part of the variables I used, or some small bug in the code.

The imSim code only sets the options to turn this on and sets the file prefix in the config file. The code to open, close and write to the files is in sims_GalSimInterface. I will open a separate PR there.

Currently, this code only is in the non-sensor version of the GalSimInterpreter class. I would like some feedback from @jchiang87, @rmjarvis and @danielsf before I add much more to make this approach seems reasonable to people.

I started by making this look like a PhoSim centroid file since I thought this might make things easier on the people running the pipeline but in principle we could also add more information into the file.

danielsf commented 6 years ago

Without looking at the code, I would like to say this:

I just finished a big PR that required comparing CatSim-predicted pixel position with actually realized PhoSim pixel position. The fact that PhoSim centroid files were segregated by chip and only reported positions in the local chip coordinate system (as opposed to some global, full-focal plane coordinate system) was quite annoying. I would vote for producing one giant centroid file for the whole InstanceCatalog that contained:

object id number of photons chipName (possibly mangled to look like 'R22S11' instead of 'R:2,2 S:1,1') x pixel (on chip) y pixel (on chip) x focal plane position (in mm; relative to center of focal plane) y focal plane position (in mm; relative to center of focal plane)

The "focal plane position" coordinates can be calculated with this method

https://github.com/lsst/sims_coordUtils/blob/u/danielsf/fix/focal/plane/python/lsst/sims/coordUtils/LsstCameraUtils.py#L44

which will hopefully get merged in the next few days.

cwwalter commented 6 years ago

Requested changes on this PR are complete.

lsst-sims / legacy_sims_GalSimInterface

Add centroid file capability. #51