Tutorial: Identifying Instrument Noise

[astrobel]

This notebook will be part of a series on creating and correcting custom light curves for Kepler and K2 data. Users will learn about instrumental noise through practical examples, which will be useful background knowledge for the following tutorials on light curve correction.

Assumed knowledge:

• Familiarity with Lightkurve, particularly the handling of LightCurve objects.
• A working knowledge of normalised light curve data and periodograms.
• Understanding terminology relating to the Kepler/K2 missions and the practical applications of that terminology, e.g. quarter/campaign, TPF, pointing.

Learning goals:

1. A thorough understanding of the main sources of noise in both Kepler and K2 missions.
2. The ability to identify instrument noise in light curves and periodograms.
3. Understanding that there are different noise features across quarters/campaigns, and being able to check the Archive Manual to confirm what you're observing.

The original proposal for this tutorial suggests looking at pixel noise as well; however, it'll be hard to explain noise phenomena at the pixel level without using Lightkurve's interactive functions, so for these goals I've kept the focus on identifying noise in light curves and periodograms. Since we'd be including periodograms (for looking at known spurious frequencies and blended signals), it might make sense to put this tutorial later in the series, or split it into two.

I'd be interested if anyone has any thoughts on this, or would like to collaborate on writing the tutorial!

[barentsen]

This sounds great @astrobel!!

Have you thought about which sources of noise to explain/cover? Needless to say, there are so many types of noise, including:

• spacecraft motion (at myriad frequencies, both inter- and intra-cadence)
• velocity aberration (somewhat similar to motion)
• psf changes
• cosmic rays
• sudden pixel sensitivy dropouts (spsd)
• crosstalk between detectors
• rolling bands (= readout electronics interference)
• aliases of read-out electronics in periodograms
• neighbor contamination (blends, saturation spikes, ...)
• asteroid contamination
• etc...

It seems very hard to cover everything in a single tutorial.

Just thinking out loud, perhaps we could split it up as follows:

1. An introductory rogues' gallery that shows a short & sweet example of every type of noise, and provides references for futher reading. (A limited attempt to gather examples for such a gallery started in this issue: https://github.com/KeplerGO/lightkurve/issues/160)
2. A tutorial explaining generic techniques to verify that a signal is real (e.g. check that a signal is not also present in a neighbor star, not present in every pixel, etc..).
3. A set of more detailed tutorials about specific types of noise.

@astrobel Does this make sense?

Useful references will include (but are certainly not limited to):

• Section 5 of the Kepler data characteristics handbook.
• Section 6 of the Kepler instrument handbook (focused on electronic sources of noise)

[astrobel]

@barentsen This sounds like a good breakdown! My initial idea was to focus on instrument-level noise - specifically things like spacecraft motion, aberration, PSF; not so much contamination - which I think would work well in one of the detailed tutorials for point three. That said, I would love to start work on the second point, and I think it could tie in with a Lightkurve project I've been meaning to propose for ages, which is incorporating the pixel periodograms plot, as in my chance alignments repo (though this code needs a big clean-up!).

[barentsen]

@astrobel That sounds good :+1:

[astrobel]

I've been combing through the instrument/data characteristics handbooks and working on a structure for some of these tutorials, and I'd welcome any feedback! I've split "instrumental noise" into things you'd observe in TPFs and light curves, based on where a casual user is more likely to encounter these different sources of noise. This means at the moment I'm considering two tutorials, structured as follows:

Instrumental Noise in Kepler and K2 TPFs

1. Introduction & background
2. DVA & Image Motion
3. Cosmic Rays
4. Undershoot
5. Crosstalk
6. Ghosts & Scattered Light
7. K2-specific effects

Instrumental Noise in Kepler and K2 Light Curves

1. Introduction & background
2. Thermal Phenomena: safe modes, momentum desaturation
3. Pointing Phenomena: loss of fine pointing, attitude tweaks, reaction wheel zero crossings, downlink Earth point
4. Other Quality Issues: cosmic rays/SPSD, Argabrightening, onboard errors, other manual exclusions
5. Quarter boundary flux discontinuities
6. Variable FGS stars in Q0-2
7. Focus Changes
8. Short cadence-specific effects: requantisation gaps, spurious frequencies
9. K2-specific effects

Some things I'm deliberating on now:

• Should the K2 features tutorials be separate?
• Is any of this too detailed?
• Would some of these topics which are less likely to come up (e.g. crosstalk, ghosts) be better suited to the rogue's gallery notebook? Or would it be good to cover them in both?

Looking forward to anybody's thoughts!

[barentsen]

Wow! It would be insanely awesome if we managed to cover all these effects!! Thank you for going through the handbook @astrobel! :+1:

I wonder if these tutorials and the rogues' gallery could be the same and one thing. I say this because the easiest way to cover all these topics, within the limited amount of time we have, may be to show examples. Doing so would provide significant added value to the handbooks which are more focused on explaining the effects rather than "demonstrating" them.

I'm not sure we can cleanly separate the sources of noise as being "noise in tpfs" versus "noise in light curves", because ultimately all noise is present at the pixel level.

I wonder if we could split the noise by theme, e.g.

• Noise due to spacecraft motion and jitter
• Noise due to spacecraft events & temperature changes (safe modes, zero wheel crossings, focus changes, etc)
• Noise due to CCD electronics (rolling bands, crosstalk, cosmic rays, etc)
• Noise due to time sampling / quantization effects

...so, perhaps the goal could be to make 4 or 5 rogues' galleries, with lots of deep links to the handbooks inside them? Just thinking out loud!

[astrobel]

Thanks for your feedback @barentsen! I think it's a great idea to combine the rogue's gallery and introduction to different types of instrument noise. I'm definitely happy to split it by theme; I'll get started, and hopefully have an example ready to chat about at the meeting tomorrow.

[astrobel]

Here's another possible structure for these tutorials/rogue's galleries:

• Gaps in the data & bad cadences (safe modes, momentum desaturation, downlink Earth point, loss of fine pointing, attitude tweaks, RW zero crossings, Argabrightening, onboard errors, manual exclusions)
• Spurious signals and time sampling effects (focus changes, variable FGS stars, attenuation of long period signals in PDC, short cadence requantisation gaps, short cadence spurious frequencies)
• Seasonal and aperture effects (DVA & image motion, quarter boundary discontinuities, ghosts & scattered light)
• Electronic noise (crosstalk, rolling bands, undershoot, cosmic rays/SPSD)

I was thinking that a beginner might not know what to look for when it comes to different sources of noise, so I largely put these together based on how the noise manifests. What do you think?

[barentsen]

@astrobel I like this structure a lot! :+1:

[ojhall94]

Wow this is awesome @astrobel, I think I would have really benefited from such tutorials when learning to work with Kepler data, and will definitely benefit from it now. My opinion is that I feel like K2 data has very specific noise properties that do not appear in Kepler and that they might be better treated separately, but then I also think that the layout you have in the comment above for 4 tutorials works really well.

I think organising based on how the noise manifests is a good idea. I'm pretty new to this as well to be honest, and it makes it easier for me to understand as well. Excited to read & provide comments!

[astrobel]

Hi all! As you can see, I've added the "help wanted" label to this issue. I've done some intensive work on collating examples and putting together tutorials for instrumental effects in Kepler data, and I would really appreciate some help when it comes to K2. I have no experience working with K2 light curves so even just pointing a few EPIC numbers in my direction to get me started would be great! Things like data gaps and spurious signals are especially appreciated. I'm okay on the electronic/detector effects front for now. Cheers!

[astrobel]

Went ahead and added some K2 sections, but as before, please let me know if there are any glaring errors! I've also added the draft ready label: the first two tutorials in this series are now available as complete first drafts.

[barentsen]

Silly question: is the Colab notebook associated with this issue named Data_gaps_and_bad_cadences.ipynb or Spurious_signals_and_time_sampling_effects_in_Kepler_and_K2.ipynb (or both?)

[astrobel]

They're both associated with this issue!

[mustaric]

I've just started reading through these notebooks! These are wonderfully useful, great job.

[astrobel]

Hi all, I've gone ahead and changed the label on this issue from "draft ready" to "ready for review." There are now (99%) complete drafts of all four instrumental noise notebooks in the google drive folder. The only notebook missing something is number 1, where I'm on the lookout for an example of reaction wheel zero crossings & momentum desaturation. However, it might be fine without an example! Any and all feedback is appreciated!

[barentsen]

I finished reviewing the first notebook in this series, which has the filename: Instrumental noise in Kepler and K2 #1: data gaps and quality flags.ipynb.

It is an excellent tutorial. Thank you @astrobel! I believe it is in a great shape and ready for copy-editing.

I made five small changes:

1. I added a cell in Section 1.1 to demonstrate how cadences can be selected for a specific quality flag, i.e. I added the following:

   
   We can use Python's "bitwise and" operator (&) to select the cadence numbers affected
   by a specific quality flag as follows:

tpf.cadenceno[(tpf.quality & 64) > 0] # cadence numbers flagged for "Argabrightening" (flag 64)

2. Similarly, elsewhere in the notebook I replaced instances of e.g. `tpf.quality==4160` with `(tpf.quality & 4160) > 0` to demonstrate the use of the "bitwise and" operation. This is important because the `==` operation only selects cadences which exclusively have flag 4160, where the `&` operation also allows other flags to be present.
3. I added a sentence to Section 2.1 to explain that the temperature changes are associated with telescope focus changes (which tends to explain the systematic), i.e. I added the words: `For example, a change in temperature can slightly change the telescope focus.`
4. I removed the TPF plot intended to show the cosmic ray for now, because I was not able to find any obviously-visible cosmic ray examples in the TPF either.
5. I could not find good visual examples of zero crossings / momentum desat artifacts either.  Because our time is limited, I think we can go ahead without those specific examples for now!

Thanks again Isabel. This tutorial is awesome and much-needed! :+1:

[barentsen]

I finished reviewing the second notebook in this series, filename:

Instrumental noise in Kepler and K2 #2: spurious signals and time sampling effects.ipynb

I believe this notebook is brilliant and very well-written.

I only made a few minor edits to remove the bold-faced pieces of text which are intended to become links to other tutorials in the future. Because none of the tutorials have been merged into the stsci repository yet, it is best to proceed with copy-editing without cross-linking the other tutorials for now. We can add links later.

Thanks so much @astrobel! This is a great notebook! :+1:

[barentsen]

I finished reviewing the third notebook in this series, filename:

Instrumental noise in Kepler and K2 #3: seasonal and detector effects.ipynb

This may be my favorite notebook in the series, because the code examples are so fun and smart! Love it!

I made a few very minor edits (e.g. one or two typos).

The most notable edit is the addition of the plt.tight_layout(); instruction in Section 3's code example, which prevents the labels from overlapping.

In the same cell, I made the code a bit more Pythonic by replacing:

i = 0
for q in np.arange(8, 12):
    ...
    i+=0

with the slightly more elegant:

for idx, quarter in enumerate(range(8, 12)):
   ...

As you can tell, I was nit-picking. I love this tutorial!! :+1:

[barentsen]

I completed my review of the fourth and final notebook in this series, with the filename:

Instrumental noise in Kepler and K2 #4: electronic noise.ipynb

It is a fabulous and valuable tutorial. It provides examples and context for electronic effects which significantly affect Kepler data, but to which few or none online resources have been dedicated in the past. I love it and have no significant corrections or suggestions! 💯

[barentsen]

I believe all four notebooks in this series are ready for copy-editing, and I intend to move them to the "ready-for-copy-editing" folder shortly.

The filenames are:

• Instrumental noise in Kepler and K2 #1: data gaps and quality flags.ipynb
• Instrumental noise in Kepler and K2 #2: spurious signals and time sampling effects.ipynb
• Instrumental noise in Kepler and K2 #3: seasonal and detector effects.ipynb
• Instrumental noise in Kepler and K2 #4: electronic noise.ipynb

[lglattly]

@astrobel and @barentsen I've finished copy editing all four of these notebooks! They were all in really great shape — I proofread and made minor copy edits throughout all of them. I believe all four notebooks are all set on my end and ready to go!

[barentsen]

Thanks @lglattly!

[barentsen]

This notebook has successfully been merged into spacetelescope/notebooks. Thanks everyone! Closing the issue...