search

caracal-pipeline / caracal

Containerized Automated Radio Astronomy Calibration (CARACal) pipeline
GNU General Public License v2.0
28 stars 6 forks source link

imcontsub crashing #1455

Closed paoloserra closed 6 months ago

paoloserra commented 1 year ago

The error looks like this

2022-11-21 08:33:46 CARACal INFO: Subtracting continuum in the image domain for target 0
2022-11-21 08:34:57 CARACal ERROR: expected non-empty vector for x [TypeError]
2022-11-21 08:34:57 CARACal INFO:   More information can be found in the logfile at output/logs-20221121-083239/log-caracal.txt
2022-11-21 08:34:57 CARACal INFO:   You are running version 1.0.6
2022-11-21 08:34:57 CARACal ERROR: Traceback (most recent call last):
2022-11-21 08:34:57 CARACal ERROR:   File "/home/pserra/Astro/caracal-venv-3.7/lib/python3.7/site-packages/caracal/main.py", line 189, in __run
2022-11-21 08:34:57 CARACal ERROR:     pipeline.run_workers()
2022-11-21 08:34:57 CARACal ERROR:   File "/home/pserra/Astro/caracal-venv-3.7/lib/python3.7/site-packages/caracal/workers/worker_administrator.py", line 441, in run_workers
2022-11-21 08:34:57 CARACal ERROR:     worker.worker(self, recipe, config)
2022-11-21 08:34:57 CARACal ERROR:   File "/home/pserra/Astro/caracal-venv-3.7/lib/python3.7/site-packages/caracal/workers/line_worker.py", line 1522, in worker
2022-11-21 08:34:57 CARACal ERROR:     clobber=True,
2022-11-21 08:34:57 CARACal ERROR:   File "/home/pserra/Astro/caracal-venv-3.7/lib/python3.7/site-packages/caracal/workers/utils/image_contsub.py", line 133, in imcontsub
2022-11-21 08:34:57 CARACal ERROR:     np.flip(np.ma.polyfit(x, incubus_data_flat, polyorder)))
2022-11-21 08:34:57 CARACal ERROR:   File "/home/pserra/Astro/caracal-venv-3.7/lib/python3.7/site-packages/numpy/ma/extras.py", line 1924, in polyfit
2022-11-21 08:34:57 CARACal ERROR:     return np.polyfit(x[not_m], y[not_m], deg, rcond, full, w, cov)
2022-11-21 08:34:57 CARACal ERROR:   File "<__array_function__ internals>", line 6, in polyfit
2022-11-21 08:34:57 CARACal ERROR:   File "/home/pserra/Astro/caracal-venv-3.7/lib/python3.7/site-packages/numpy/lib/polynomial.py", line 599, in polyfit
2022-11-21 08:34:57 CARACal ERROR:     raise TypeError("expected non-empty vector for x")
2022-11-21 08:34:57 CARACal ERROR: TypeError: expected non-empty vector for x
2022-11-21 08:34:57 CARACal INFO: exiting with error code 1

@gigjozsa rings a bell?

Currently investigating the cause ...

paoloserra commented 1 year ago

@gigjozsa

This is what I think is happening. Channels masked in at least 1 LOS of the cube are excluded from the polynomial fit of all LOS's. Only channels with no masked LOS's at all are included in the fit of each and every LOS. Those channels are used to build the independent variable of the fit x mentioned in the error message above.

In the case of my error, all channels are masked in at least 1 LOS (but by no means in all LOS's), with the result that x is empty.

I think this is not the desired behaviour. Each LOS should get its own x. The fact that a channel is masked in a given LOS does not mean that it should be masked in all LOS's.

In fact, that's one of the main advantages of image-based continuum subtraction compared to UVLIN. A channel with line emission at x1,y1 should be excluded from the fit at x1,y1 but can be used at all other positions in the cube.

What do you think?

paoloserra commented 1 year ago

@gigjozsa kind reminder

Fil8 commented 9 months ago

I think it may be a problem of np.ma.polyfit as discussed here: https://github.com/numpy/numpy/issues/9193

paoloserra commented 9 months ago

Yep, agreed, which just means we need a different implementation altogether.

Athanaseus commented 6 months ago

Will be implemented in stimela2 and in the next caracal updates.