flag uzero broken with apptainer

[Fil8]

The upgrade to this (from /workers/utils/flag_uzeros.yml)

    @extras(packages=["astropy", "scipy"])
    def __init__(self, config):
        from astropy import units as u
        from astropy.coordinates import SkyCoord
        import astropy.visualization as astviz
        from astropy.wcs import WCS
        from astropy.table import Table, Column
        from astropy.io import fits
        import astropy.io.ascii as astasc
        import scipy.optimize as optimize
        import scipy.constants as scconstants
        from scipy import stats

        self.config = config

tragically fails:

# Cleaning up temporary files...
2023-06-02 16:23:17 CARACal.Stimela.makeCube INFO: job complete at 2023-06-02 16:23:17.159716 after 0:02:02.061811
2023-06-02 16:23:17 CARACal.Stimela.flagUzeros INFO: Saving pipeline information in .last_flagUzeros.json
2023-06-02 16:23:17 CARACal.Stimela.flagUzeros INFO: Recipe executed successfully
2023-06-02 16:23:17 CARACal INFO: Image Done
2023-06-02 16:23:17 CARACal INFO: Making FFT of image
2023-06-02 16:23:17 CARACal ERROR: name 'fits' is not defined [NameError]
2023-06-02 16:23:17 CARACal INFO:   More information can be found in the logfile at output/logs-20230601-042430/log-caracal.txt
2023-06-02 16:23:17 CARACal INFO:   You are running version 1.1.1
2023-06-02 16:23:17 CARACal ERROR: Traceback (most recent call last):
2023-06-02 16:23:17 CARACal ERROR:   File "/iranet/arcsoft/caracal/caracal-master/lib/python3.9/site-packages/caracal/main.py", line 189, in __run
2023-06-02 16:23:17 CARACal ERROR:     pipeline.run_workers()
2023-06-02 16:23:17 CARACal ERROR:   File "/iranet/arcsoft/caracal/caracal-master/lib/python3.9/site-packages/caracal/workers/worker_administrator.py", line 436, in run_workers
2023-06-02 16:23:17 CARACal ERROR:     worker.worker(self, recipe, config)
2023-06-02 16:23:17 CARACal ERROR:   File "/iranet/arcsoft/caracal/caracal-master/lib/python3.9/site-packages/caracal/utils/requires.py", line 25, in inner_func
2023-06-02 16:23:17 CARACal ERROR:     return func(*args, **kw)
2023-06-02 16:23:17 CARACal ERROR:   File "/iranet/arcsoft/caracal/caracal-master/lib/python3.9/site-packages/caracal/workers/line_worker.py", line 716, in worker
2023-06-02 16:23:17 CARACal ERROR:     uZeros.run_flagUzeros(pipeline, all_targets, msname_Flag)
2023-06-02 16:23:17 CARACal ERROR:   File "/iranet/arcsoft/caracal/caracal-master/lib/python3.9/site-packages/caracal/workers/utils/flag_Uzeros.py", line 838, in run_flagUzeros
2023-06-02 16:23:17 CARACal ERROR:     inFFTData, inFFTHeader = self.makeFFT(outCubeName)
2023-06-02 16:23:17 CARACal ERROR:   File "/iranet/arcsoft/caracal/caracal-master/lib/python3.9/site-packages/caracal/workers/utils/flag_Uzeros.py", line 302, in makeFFT
2023-06-02 16:23:17 CARACal ERROR:     with fits.open(inCube) as hdul:
2023-06-02 16:23:17 CARACal ERROR: NameError: name 'fits' is not defined
2023-06-02 16:23:17 CARACal INFO: exiting with error code 1

[paoloserra]

@Fil8 maybe the standard test does not try the flagging at u=0?

[Fil8]

I fear it may be more complicated than this. I am running a check and update soon.

[Fil8]

It is an issue not in the code itself, but in the images. In the meergas cluster I don't get this error, but in the IRA cluster (where I load the images with this variable: STIMELA_PULLFOLDER) I get the error above

[Fil8]

so comparing the log-files between meergas cluster and ira's I find that in IRA's (where I get the error above), these info appear:

# INFO:    Environment variable SINGULARITYENV_HOME is set, but APPTAINERENV_HOME is preferred
# INFO:    Environment variable SINGULARITYENV_OUTPUT is set, but APPTAINERENV_OUTPUT is preferred
# INFO:    Environment variable SINGULARITYENV_MSDIR is set, but APPTAINERENV_MSDIR is preferred
# INFO:    Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
# INFO:    Environment variable SINGULARITYENV_CONFIG is set, but APPTAINERENV_CONFIG is preferred
# INFO:    Environment variable SINGULARITYENV_INPUT is set, but APPTAINERENV_INPUT is preferred
# INFO:    Environment variable SINGULARITYENV_STIMELA_MOUNT is set, but APPTAINERENV_STIMELA_MOUNT is preferred
# WARNING: Overriding HOME environment variable with SINGULARITYENV_HOME is not permitted
# INFO:    squashfuse not found, will not be able to mount SIF

while on the meergas cluster I don't get this error.

[Fil8]

this is a command I get on these IRA's machines when loading the caracal-environment.

fmaccagni@merlino2:/local/work/MAGNHIFFIC/rawdata/NGC3100$ caracal-setup
[irainit] 'apptainer' version '1.1.5' is now ready on 
[ To run : caracal -ct singularity -c ${your-configuration-file} ]
[irainit] 'caracal' version 'default' is now ready on 

[Fil8]

When using apptainer the astropy image has issues and therefore the error above.

[Athanaseus]

@Fil8 did you install with pip install .[all]? astropy including a few dependencies (scipy, regions, astroquery) are now not installed by default but can be included with the above line.

[mstagni]

Hi there, Matteo from Bologna IRA. I have re-pulled the stimela images using singularity 3.8.3 - same as ASTRON as @Fil8 suggested. Pulling using apptainer seems to be causing issues. What is the recommended version of singularity to use with stimela images?

[Athanaseus]

Hi @mstagni , are you still experiencing this issue with the latest caracal release?

[mstagni]

It seems to be working with apptainer too now in v1.1.3 . Thanks @Athanaseus.