Fix RMS maps offset in corrections and select non-variable sources

[AkashA98]

Currently, given the linear flix scale model, the RMS maps are corrected by an offset parameter, which they shouldn't. This fix reverses that.

Also, when comparing the data with the RACS epoch, if there are variable sources, then can influence the linear fit that estimates the scale and offset, a basic sigma clipping is implemented to filter out > 3 -sigma variable sources.

[AkashA98]

Ok, @ddobie, @dlakaplan I've changed the default behavior to hold the intercept to 0 and just fit for slope; and pushed the changes here. Please do let me know if anything improvements are needed. Thanks...

[ddobie]

I think this looks fine. Have you tested it?

[AkashA98]

I think this looks fine. Have you tested it?

Nope. testing this code is a bit difficult because downloading all the necessary images/RMS maps and catalogs onto a local machine is tedious. Maybe the best way is to run this on one epoch and then I can inspect the images and catalogs?

[ddobie]

This should have a small subset of the data that I use for testing locally: https://liveswinburneeduau-my.sharepoint.com/:u:/g/personal/ddobie_swin_edu_au/EQJMZCDSqQBJn1Ly0ipE9NsBKa45B1GmAJ_6HDBYdxqITQ?e=z7rbKX

[ddobie]

Just tested it and immediately got an error:

  File "/import/ada2/ddob1600/.conda/envs/vast-post-processing-dev/bin/vast_post_processing", line 6, in <module>
    sys.exit(app())
  File "/import/ada2/ddob1600/.conda/envs/vast-post-processing-dev/lib/python3.9/site-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "/import/ada2/ddob1600/.conda/envs/vast-post-processing-dev/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/import/ada2/ddob1600/.conda/envs/vast-post-processing-dev/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/import/ada2/ddob1600/.conda/envs/vast-post-processing-dev/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/import/ada2/ddob1600/.conda/envs/vast-post-processing-dev/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/import/ada2/ddob1600/.conda/envs/vast-post-processing-dev/lib/python3.9/site-packages/typer/main.py", line 532, in wrapper
    return callback(**use_params)  # type: ignore
  File "/import/ada2/ddob1600/vast-post-processing/vast_post_processing/cli/vast_post_processing.py", line 109, in main
    core.run(
  File "/import/ada2/ddob1600/vast-post-processing/vast_post_processing/core.py", line 857, in run
    corrected = corrections.correct_field(
  File "/import/ada2/ddob1600/vast-post-processing/vast_post_processing/corrections.py", line 828, in correct_field
    ) = vast_xmatch_qc(
  File "/import/ada2/ddob1600/vast-post-processing/vast_post_processing/corrections.py", line 171, in vast_xmatch_qc
    sigma_clip_mask = sigma_clip(
  File "/import/ada2/ddob1600/.conda/envs/vast-post-processing-dev/lib/python3.9/site-packages/astropy/stats/sigma_clipping.py", line 886, in sigma_clip
    return sigclip(
  File "/import/ada2/ddob1600/.conda/envs/vast-post-processing-dev/lib/python3.9/site-packages/astropy/stats/sigma_clipping.py", line 673, in __call__
    return self._sigmaclip_noaxis(
  File "/import/ada2/ddob1600/.conda/envs/vast-post-processing-dev/lib/python3.9/site-packages/astropy/stats/sigma_clipping.py", line 460, in _sigmaclip_noaxis
    filtered_data.mask |= np.logical_or(
  File "/import/ada2/ddob1600/.conda/envs/vast-post-processing-dev/lib/python3.9/site-packages/astropy/utils/masked/core.py", line 715, in __array_ufunc__
    raise TypeError("cannot write to unmasked output")
TypeError: cannot write to unmasked output

[ddobie]

Okay I've fixed that error. Seems to mostly run okay now

[ddobie]

I've tested it on the complete epoch 41 and from what I can tell, everything looks fine and normal. I think we can merge this.