Clarify docs on how to handle gappy data

[shicks-seismo]

Hi Callum,

I'm just getting started with EQcorrscan, and am having good success so far - great work!

However, I have gappy data and am finding that I am getting high-CC detections both in the middle of gaps, and at the edge of gaps (typically, these gaps are minutes long). I am using the latest development version of EQCS, cloned directly from here, and running it on OSX. I see that you recent made some PRs over this: (https://github.com/eqcorrscan/EQcorrscan/pull/230; https://github.com/eqcorrscan/EQcorrscan/pull/227; https://github.com/eqcorrscan/EQcorrscan/issues/224). Any ideas why I'm still getting gaps?

Thanks!

[calum-chamberlain]

Would you be able to share some data and a template that have the issue (and the script you are running)?

[shicks-seismo]

Hi Callum, Attached are the files you need. Execute run.py to make the templates and to do the detection. Data comes from an FDSNWS source that is called in the script, so I've not provided you with any data. I get numerous detections in the two main regions of gaps within the hourly data segment requested. Archive4Callum.zip.

[calum-chamberlain]

Thanks Steve, I will have a play. Thanks for flagging this as well - the more people that flag issues the better for everyone in the long run!

Just to check, you say the latest development version, I assume you mean the develop branch? This branch should raise an error if there are issues with the correlations (unfortunately error flags were getting lost, but this was patched in #254). Did you get errors at your end?

On an initial run I did get errors here (on Linux, py 3.6):

Error in C code (possible normalisation error)
Maximum cccs 1.213420 at (2, 32764)
Minimum cccs -1.015720 at (7, 33218)
Internal correlation error
We made a total of 0 detections

and I will investigate further today.

[calum-chamberlain]

Looks like the issue stems from line 94 of your script, where you fill the gaps with zeros:

st.merge(fill_value=0)

Previous versions of EQcorrscan did require data to be padded with zeros, but for version 0.3.x it now requires gaps to be maintained, so that they can be properly handled internally. This is not clear in the docs at all, so I'm not surprised you ended up doing this.

When I change that line to:

st.merge()

I don't get any obviously strange correlations (no errors and no detections).

Would you be able to make that change on your end and play around some more? I would advise checking out the develop branch - I have made a further patch since the last release (#254) that attempts to handle another gap-related issue (gaps followed by spikes were producing horrible correlations, and errors were not properly passed back to Python).

On my end (or if you wanted to make a PR on develop), we need to:

• [ ] make it explicit in the docs that gaps should be maintained before passing to pre_processing functions.
• [ ] remove the reference to two-channel naming in pre_processing doc-strings, this is out-dated.
• [ ] It would be really good if some variance check could be made on the data to ensure that, if someone does pass gap-filled data, it is caught and a warning is raised. This could be a moving window variance calculation... It would slow things down though.

Let me know what you find @shicks-seismo, and sorry for the confusion, this needs to be clearer in the docs! If you have thoughts on where things should go in the docs that would be appreciated and if you have noticed other things that are wrong in the docs I would love to know. It's easy to miss things when you wrote them.

[shicks-seismo]

Great - fixed! Thanks a lot. Yes, I didn't have the development branch (even though I had tried st.merge() on the main release which still produced the artifacts).

[calum-chamberlain]

Hey @shicks-seismo, I'm going to keep this open until I make those docs changes for version 0.3.2, which I really must release soon!