Sharpener woes

[molnard89]

I'm trying to extract spectra towards continuum sources with sharpener, but I bumped into some issues.

Firstly, lines setting the input folder names had to be changed. So in lines

https://github.com/caracal-pipeline/caracal/blob/e9061386407eb8f282443f77f79a219fd3926174/caracal/workers/line_worker.py#L1152 and https://github.com/caracal-pipeline/caracal/blob/e9061386407eb8f282443f77f79a219fd3926174/caracal/workers/line_worker.py#L1167

pipeline.output has to be removed, because the output folder path is already included in image_cube_list[uu]).split('/')[:-1].

With that done, I still get different errors when trying to use NVSS or PYBDSM-extracted source catalogues. With PYBDSM selected I get:

2020-07-31 08:11:44 CARACal.Stimela.remove_cube_stokes_axis-1 INFO: job complete at 2020-07-31 08:11:44.603217 after 0:00:00.055920
2020-07-31 08:11:44 CARACal.Stimela.continuum-spectral_extraction-0 INFO: job started at 2020-07-31 08:11:44.666543
# 0e60bfc3a6722321870cdfdf042add9d6690d79b28d24da696489590a6cae04d
# /stimela_mount/code/run.py:26: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
#   list_doc = yaml.load(f)
# Traceback (most recent call last):
#   File "/stimela_mount/code/run.py", line 61, in <module>
#     subprocess.check_call(shelx.split(_runc))
# NameError: name 'shelx' is not defined
2020-07-31 08:11:46 CARACal.Stimela.continuum-spectral_extraction-0 ERROR: docker returns error code 1
2020-07-31 08:11:46 CARACal.Stimela.continuum-spectral_extraction-0 ERROR: job failed at 2020-07-31 08:11:46.286124 after 0:00:01.619581
2020-07-31 08:11:46 CARACal ERROR: Job 'continuum-spectral_extraction-0:: Continuum Spectral Extraction' failed: docker returns error code 1 [PipelineException]
2020-07-31 08:11:46 CARACal INFO:   More information can be found in the logfile at output/logs-20200731-081141/log-caracal.txt
2020-07-31 08:11:46 CARACal INFO: exiting with error code 1

and with NVSS

2020-07-31 08:13:53 CARACal.Stimela.continuum-spectral_extraction-0 INFO: job started at 2020-07-31 08:13:53.880816
2020-07-31 08:13:53 CARACal.Stimela.continuum-spectral_extraction-0 ERROR: Parameter thr is unknown. Run 'stimela cabs -i sharpener' to get help on this cab
2020-07-31 08:13:53 CARACal.Stimela.continuum-spectral_extraction-0 ERROR: job failed at 2020-07-31 08:13:53.881343 after 0:00:00.000527
2020-07-31 08:13:53 CARACal ERROR: Job 'continuum-spectral_extraction-0:: Continuum Spectral Extraction' failed: Parameter thr is unknown. Run 'stimela cabs -i sharpener' to get help on this cab [PipelineException]
2020-07-31 08:13:53 CARACal INFO:   More information can be found in the logfile at output/logs-20200731-081350/log-caracal.txt
2020-07-31 08:13:53 CARACal INFO: exiting with error code 1

I have a fresh installation of the latest caracal as of today. I implemented fixes locally for the first issue I mentioned, and will make it into a PR once the rest is sorted, but for now I'm pretty lost on how to proceed. Thanks!

[molnard89]

Following the recommendation of @KshitijT I changed the line

https://github.com/caracal-pipeline/caracal/blob/e9061386407eb8f282443f77f79a219fd3926174/caracal/workers/line_worker.py#L1161

from params["thr"] into params["thresh"], which is now the correct input for sharpener. This turned my NVSS issue into the same error as the one above using PYBDSM.

[KshitijT]

This line in Stimela: https://github.com/ratt-ru/Stimela/blob/master/stimela/cargo/cab/sharpener/src/run.py#L61

    subprocess.check_call(shelx.split(_runc))

needs to read:

    subprocess.check_call(shlex.split(_runc))

[KshitijT]

This is fixed in Stimela now; @molnard89 , do you still encounter this error? If not , please close this issue.