IndexError: list index out of range

[AlexKurek]

Using latest SIngularity conatiner (https://github.com/mhardcastle/ddf-pipeline/issues/312#issuecomment-1527010957) Im getting:

Successful readonly open of default-locked table L658346_SB001_uv_avg_12C2BC993t_121MHz.pre-cal.ms/OBSERVATION: 31 columns, 1 rows
../4C29.30.ds9.reg
[130.00975000deg,29.81742500deg]
Correcting boxfile for the local north
Using these observations  ['L658346']
Traceback (most recent call last):
  File "/usr/local/src/ddf-pipeline/scripts/sub-sources-outside-region.py", line 585, in <module>
    DOut=SummaryToVersion("summary.txt")
  File "/usr/local/src/ddf-pipeline/scripts/sub-sources-outside-region.py", line 574, in SummaryToVersion
    l=L[iLine]
IndexError: list index out of range
Traceback (most recent call last):
  File "/usr/local/bin/CleanSHM.py", line 26, in <module>
    from DDFacet.compatibility import range
  File "/usr/local/lib/python3.9/dist-packages/DDFacet/__init__.py", line 23, in <module>
    __version__ = pkg_resources.require("DDFacet")[0].version
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 886, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 777, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (numpy 1.24.3 (/usr/local/lib/python3.9/dist-packages), Requirement.parse('numpy<1.24,>=1.18'), {'numba'})
 25 columns, 6578850 rows
/storage/akurek/extractPy/4C29.30_timeavg1/4C29.30/P129+29/L693959_SB413_uv.pre-cal_12D524E44t_156MHz.pre-cal.ms 0.26963945066386985
Successful readonly open of default-locked table /storage/akurek/extractPy/4C29.30_timeavg1/4C29.30/P129+29/L693959_SB423_uv.pre-cal_12D524E44t_158MHz.pre-cal.ms: 25 columns, 6578850 rows
/storage/akurek/extractPy/4C29.30_timeavg1/4C29.30/P129+29/L693959_SB423_uv.pre-cal_12D524E44t_158MHz.pre-cal.ms 0.2744160681578087
Successful readonly open of default-locked table /storage/akurek/extractPy/4C29.30_timeavg1/4C29.30/P129+29/L693959_SB432_uv.pre-cal_12D524E44t_160MHz.pre-cal.ms: 25 columns, 6578850 rows
/storage/akurek/extractPy/4C29.30_timeavg1/4C29.30/P129+29/L693959_SB432_uv.pre-cal_12D524E44t_160MHz.pre-cal.ms 0.17283014508614727
Successful readonly open of default-locked table /storage/akurek/extractPy/4C29.30_timeavg1/4C29.30/P129+29/L693959_SB442_uv.pre-cal_12D524E44t_162MHz.pre-cal.ms: 25 columns, 6578850 rows
/storage/akurek/extractPy/4C29.30_timeavg1/4C29.30/P129+29/L693959_SB442_uv.pre-cal_12D524E44t_162MHz.pre-cal.ms 0.19027707730074406
Successful readonly open of default-locked table /storage/akurek/extractPy/4C29.30_timeavg1/4C29.30/P129+29/L693959_SB452_uv.pre-cal_12D524E44t_164MHz.pre-cal.ms: 25 columns, 6578850 rows
/storage/akurek/extractPy/4C29.30_timeavg1/4C29.30/P129+29/L693959_SB452_uv.pre-cal_12D524E44t_164MHz.pre-cal.ms 0.24146511168365292
Successful readonly open of default-locked table /storage/akurek/extractPy/4C29.30_timeavg1/4C29.30/P129+29/L693959_SB461_uv.pre-cal_12D524E44t_166MHz.pre-cal.ms: 25 columns, 6578850 rows
/storage/akurek/extractPy/4C29.30_timeavg1/4C29.30/P129+29/L693959_SB461_uv.pre-cal_12D524E44t_166MHz.pre-cal.ms 0.5271044103452731

�[91m============================= Running subtraction  =============================�[0m

�[92mRunning: sub-sources-outside-region.py  --timeavg=1 --overwriteoutput --ncpu=28 -b ../4C29.30.ds9.reg -p 4C29.30�[0m
�[91mFAILED to run sub-sources-outside-region.py  --timeavg=1 --overwriteoutput --ncpu=28 -b ../4C29.30.ds9.reg -p 4C29.30: return value is 1�[0m
Traceback (most recent call last):
  File "/usr/local/src/ddf-pipeline/scripts/extraction.py", line 116, in <module>
    run(executionstr,database=False)
  File "/usr/local/src/ddf-pipeline/utils/auxcodes.py", line 68, in run
    die('FAILED to run '+s+': return value is '+str(retval),database=database)
  File "/usr/local/src/ddf-pipeline/utils/auxcodes.py", line 51, in die
    raise RuntimeError(s)
RuntimeError: FAILED to run sub-sources-outside-region.py  --timeavg=1 --overwriteoutput --ncpu=28 -b ../4C29.30.ds9.reg -p 4C29.30: return value is 1

[mhardcastle]

This is because of an error in the logic in the parsing of the summary file -- shouldn't matter unless the summary file is corrupt. Can you try building a singularity using the bugfixes branch of the pipeline and see if it's resolved?

[cyriltasse]

@mhardcastle should I merge that bugfixes branch (which?) onto ddfFixesApril23_issue331 and recompile ddf_dev.sif?

[mhardcastle]

I think there's a lot of duplication in the two branches so they probably won't merge cleanly. I'm just looking at this.

[mhardcastle]

... so I think the bugfixes branch (just bugfixes) probably now contains all the changes that you made in your ddfFixes... branch, but in a more flexible way (e.g. should still work with old versions of PyBDSF). It also fixes other issues. I'm rebuilding your singularity image with that version of ddf-pipeline now and will test.

[mhardcastle]

Should be fixed, closing.