search

wfau / ScienceArchives

0 stars 0 forks source link

Bug in cu34 - missing PROGID? #589

Open wfastrononomer opened 1 month ago

wfastrononomer commented 1 month ago

2024-07-17 12:18:47.46: [14071] Getting all programme IDs.

2024-07-17 12:18:47.65: str: [14071] ValueError('Map /disk62/sys/test/products/map/20240713_v6/v20140224_00217_st_map160_2_1.fits should have known programmeID: probably a missing PROGID key',)

Traceback (most recent call last): File "/unas/njc/VDFS/ListDrivenClean/src/curation/invocations/cu4/cu4.py", line 370, in _onRun catsForProg = self.getProgIDs() File "/unas/njc/VDFS/ListDrivenClean/src/curation/invocations/cu4/cu4.py", line 745, in getProgIDs self.getProgramIDs(self._shortListFile.name) File "/unas/njc/VDFS/ListDrivenClean/src/curation/wsatools/DbConnect/IngCuSession.py", line 988, in getProgramIDs "probably a missing PROGID key" % fileName)

2024-07-17 12:18:47.65: [14071] Log written to /disk78/genOps/vsa/logs/TestVSAnjcUVDR6/cu34id14071.log

wfastrononomer commented 1 month ago

Can't see any of the header values from updateHeader in the catalogue FITS: MAPID, CATLGID, VERSNUM, PROGID.

Has it gone to ingest before completion? MapFrameStatus - bitProcessingFlag --> complete, but why not updateHeaders: catID=3720. Check logs multiframeID=4850203 Weird - frame seems to have been created on disk >= 20240713, but only entry in logs is cuEventID=1651, 20240530.

But, /disk62/sys/test/products/map/20240713_v6/v20180110_00204_st_map160_2_1.fits does have these keys in the extension headers.

updateHeaders in filterCat bit of code --> check FITS files to see if this has been run

TODO: add more checks incase network issues - make sure updates have happened before updating bitProcessingFlag

wfastrononomer commented 1 month ago

Rerun after refactoring still gives this error. Check why.

Also pipeline should fall over, rather than continuing --> check why it doesn't - this is cu34 code.

wfastrononomer commented 1 month ago

Mostly updated the pipeline. Running to test it. Mostly working. Think last bug was a network issue, but doing more checks to be sure.

tms-epcc commented 3 weeks ago

14/AUG/25 @wfastrononomer reported

wfastrononomer commented 2 weeks ago

2024-08-16 13:24:34.38: [15251] Log written to /disk78/genOps/vsa/logs/TestVSAnjcUVDR6/cu34id15251.log

2024-08-16 13:40:53.98: str: [15211] IOError(11, 'Resource temporarily unavailable')

Traceback (most recent call last): File "/unas/njc/VDFS/ListDrivenClean/src/curation/invocations/cu4/cu4.py", line 483, in _onRun histories) File "/unas/njc/VDFS/ListDrivenClean/src/curation/wsatools/File.py", line 297, in pickleWrite ofh = open(self.name, 'w')

2024-08-16 13:40:53.98: [15211] Log written to /disk78/genOps/vsa/logs/TestVSAnjcUVDR6/cu34id15211.log