search

LSSTDESC / ImageProcessingPipelines

Alert Production and Data Release image processing pipelines using the LSST Stack
BSD 3-Clause "New" or "Revised" License
3 stars 2 forks source link

Create masks for imSim for DM processing and find hot pixels etc in PhoSim if they are being produced. #67

Open cwwalter opened 6 years ago

cwwalter commented 6 years ago

For the DM processing we need to supply mask files for hot pixels and columns etc.

In imSim we plan to forgo simulating them and just apply actual masks taken from the camera team sensor testing. For PhoSim, I believe the current plan is to turn that feature on (I am not positive) in which case code should also be supplied to find the hot pixels and columns and produce the mask files.

heather999 commented 5 years ago

Is this still an active issue @cwwalter? Or is it ok to close?

cwwalter commented 5 years ago

Hi @heather999 there is a longer conversation in another (GH?) thread we should find. We also had a meeting with DM where we talked about how to mark pixels that would be interpreted over (also for display purposes) and there was a proposed extension to the data object using some sort of heavy foot print.

I think this is still an active issue. It is necessary for PhoSim (in that these defects are actually in the images) but we can decide, given other issues with 1.2p, if we want to bother with them.

For imSim, we could choose to ignore it (thus assuming perfect sensors) but I still think it would be better to do this if we can. We should at least discuss before closing.

RobertLuptonTheGood commented 5 years ago

I continue to think that "It is necessary" is a gross exaggeration, but we agreed to implement it for you. We have been looking at this (https://jira.lsstcorp.org/browse/DM-17471), and it's a bit more of a pain than it might seem as we have to interpolate, apply BF corrections (a convolution), revert the interpolation, finish the ISR processing, save the interpolated pixels for you, and then finally interpolate a second time.

cwwalter commented 5 years ago

Hi Robert,

The necessary part I was talking about here (for PhoSim) isn't the ability to look at the pixels before interpolation (although it is true that we also want that). I am talking about giving you the masks for the locations of the hot pixels so the interpolation can be done. Does that make a difference to you in how you interpret what I wrote?

RobertLuptonTheGood commented 5 years ago

Offline, Chris and I sorted out what's needed. If you give us a Mask with a bit set for all bad objects we can convert this to the format we use to describe defects. This is unrelated to displays, heavy footprints, etc. I'd give you the code, except that I'm not quite sure how we're doing the book-keeping for all the cameras in obs_lsst and it may take a minor fiddle

johannct commented 4 years ago

@cwwalter can we close this issue?

cwwalter commented 4 years ago

So, this was never finished because the defect work being done by the project wasn't ready by the time we started processing 2.1. It does need to be done. It may not be something we could actually do for 2.2 processing depending on when it really starts in earnest but it could always be done in reprocessing.

We should probably follow up with @plazas to understand the state of the project mask/defect code now.

cwwalter commented 4 years ago

@plazas Could you give us an update on the status of making defect masks (we are curious when they might exist from the BOT that we could then use while processing DC2)?

Also, to give us a quantitive sense of how much masking/interpolation we are missing by not including the masks for documentation, can you tell us what sort of fraction of hot pixel and column statistics you have been seeing?

Thanks!