chime-experiment / ch_util

CHIME utilities
https://chime-experiment.github.io/ch_util
MIT License
3 stars 3 forks source link

Parallel RFI filtering #35

Closed ljgray closed 2 years ago

ljgray commented 2 years ago

In its current implementation, the RFI Filter step of the daily pipeline takes ~25 minutes. Most of this time is spent taking a rolling median, and it is done in a single process. This PR should address the issue by pre-calculating a median across frequencies to flatten the auto vis. data, then distributing across the frequency axis to distribute the computation across processes.

ljgray commented 2 years ago

Although the products aren't quite right yet, this reduces the RFIFilter step down to ~15 seconds in a daily pipeline run

ljgray commented 2 years ago

Depends on radiocosmology/caput#211

ljgray commented 2 years ago

Interestingly, the mask being produced is different from that produced by the previous method.

ljgray commented 2 years ago

@jrs65 this produces the same mask now as previous revisions, so it should be good to go unless there are additional changes.