Automate Data Processing

[sumanchapai]

The regular data processing that all our images for each night go through currently go through a manual but perhaps automate-able process... We'll see how far we can get about automating that!

[sumanchapai]

@wilkerje , we'll dicuss/document this process here and several of the related YET_TO_BE_CREATED issues…

[minhviet0308]

Here is the current state of things w.r.t. Data processing. We tested the extraction module independently of data processing to diagnose the bugs more easily, and found them and were able to fix them.

Likewise, we were also able to do the same to normalization code.

Fixing bugs here and there, and re-running the data processing, this is the result we got....

Normalized ADU for a star from running IDL code were as follows:

   8264340.00
   8272134.50
   8203955.50
   8157302.50
   8067181.50

Normalized ADU for a star from running python code were as follows:

8271920.49
8260667.23
8190466.17
8126589.06
8085918.76

The norm factors for that night from IDL code was:

1.040383
1.018559
1.021395
0.987418
0.976323

and from python code was:

1.04074
1.01886
1.01815
0.98858
0.97675

@wilkerje @isaachroberts43

[minhviet0308]

We also verified (by testing each module independently) that the difference we have currently is coming from alignment module.

[minhviet0308]

Here is the result after running data processing in the new python code from start to finish. The curves below are normalization intra night (june 12 2021) for norm factor vs image through night

The one in the left is from python.

[minhviet0308]

[minhviet0308]

Also, stars look generally brighter in IDL code... something as much as 6%... please check the graph for three stars we sampled...

[minhviet0308]

Normalization factors for a different night in 2021 in py and idl

[minhviet0308]

@wilkerje Here is a normalization factor file for a processed night on June 19, 2022.

[sumanchapai]

the issue is titled automate data processing, this is pretty much completed… closing this now