Outputs for displayQA

[HironaoMiyatake]

As we have discussed before, we decided to use displayQA for showing a bunch of systematics plots. I am thinking about what is needed for this. Here is the list. Currently I restrict myself to tests for each field (not for coadd).

1. A Visit-level plot for a summary plot (a left large figure on the displayQA webpage): This can be obtained by running StileVisit.py.
2. CCD-level plots: This can be obtained by running StileCCD.py. However, we have to run Stile for each CCD which duplicates the time for reading data given that we run StileVisit.py as well. This can be avoided by adding an option to run CCD-level tests as well to StileVisit.py.
3. Statistics for each CCD: We want mean/median and scatter for making table under the summary plot and the bad CCD indicator on the right of the displayQA webpage. StatSysTest could do that, but we have to add the functionality to write down the results to file.

I think 3 is urgent for the displayQA work. For 2, it is not obvious. This feature to StileVisit.py can be duplicate as we already have StileCCD.py. Anyway, please let me know what you think.

[rmandelb]

I think 3 seems like the most urgent. For 2, I like the idea of adding that option to StileVisit.py.

[msimet]

I can work on 3, but FYI it's going to require some code refactoring so it might take a little time.

[rmandelb]

We should perhaps compare this in priority to other ongoing Stile work. Do you want to chat about this when we meet tomorrow?

[msimet]

Yep, sounds good.

[HironaoMiyatake]

I thought about 3 a little bit more. Suppose we need whisker plots, scatter plots, and some correlation functions.

3-1. For whisker plots, the code does not calculate statistics internally, so we need to calculate those by using StatSysTest. 3-2. For scatter plots, actually the code calculates median and scatter inside, so we could add an option to write down those numbers to ScatterPlotSysTest. However, this functionality to calculate statistics duplicates with StatSysTest, so it might be cleaner to call some methods inside StatSysTest, but we could leave this for a future work. 3-3. For correlation functions, it might be nice to write down the correlation function itself calculate by TreeCree into a file to make use of Rachel's requirement which is defined by a correlation function at 10'.

So, we might need three separate works. I am happy to work on 3-1 and 3-2. For 3-3, I need Melanie's help. Anyway, I know we should put priority with other ongoing works. Please discuss it tomorrow, and please let me know.

[msimet]

Er, sorry, I meant 2! Getting to do CCD-level stuff while collecting the data for Visit-level stuff.

3-3 is easy--the data is in an array returned by the SysTest, the stuff that's sent to the .plot() method. And we have table-writing methods already, that's file_io.WriteTable (or file_io.WriteASCIITable so it doesn't use FITS). We should maybe add a line to WriteASCIITable to output a header line with column descriptions, but that too is pretty easy. So if you want to start a branch for this, Hironao, I can do 3-3 quickly tomorrow, and then think about 2 with Rachel in our meeting.

[HironaoMiyatake]

Great! Please work on 3-3. I'll work on 3-1 and 3-2 this week. Once we have 3, I can start writing the displayQA codes.

[HironaoMiyatake]

I tried to look into 3-1. It seems that it is linked to #32. Currently there is only one 'sys_tests' which uses StatSysTest in the configurable list of sys_tests. Do we want to work on #32 first?

Meanwhile, I'll work on 3-2.

[msimet]

If all you want is statistics of quantities you're using in the whisker plot, you can use a Stile StatSysTest directly, without an Adapter wrapper. The Adapter versions are just so the driver knows which columns and masks to use. Since we already have the arrays, you don't need it.

[HironaoMiyatake]

Okay, I'll call StatSysTest inside of WhiskerPlotSysTest.

[HironaoMiyatake]

I'll merge #39 to this, as I want to work with the faster code.

[HironaoMiyatake]

@msimet I added rho1 statistics (StarPSFResidXStarPSFResidShear). I might not have done this because I am not a person who wrote the correlation function classes... Can you just take a look?

[msimet]

Hi Hironao,

Just left a comment on the code (formatting updated on GitHub). You could also just name it Rho1 if you like, since that's shorter and most people will know what it means (we can document it).