Out of date LBCS VO server

[rvweeren]

The script uses an old version of the LBCS catalog on the Astron VO server (resulting in zero sources found and an error)

https://github.com/lmorabit/long_baseline_pipeline/blob/e29d33b2c8b8252125dfe07827801d95bb4b5950/plugins/PipelineStep_DownloadCats.py#L127

Good sources - 0 Target field not in LoTSS coverage yet! Only writing /net/rijn/data2/rvweeren/A2256_LB/prefactor_output//LB-Delay-Calibration/delay_calibrators.csv 2019-10-07 20:19:06 ERROR genericpipeline: 2019-10-07 20:19:06 ERROR genericpipeline: Failed pipeline run: LB-Delay-Calibration 2019-10-07 20:19:06 ERROR genericpipeline: Detailed exception information: 2019-10-07 20:19:06 ERROR genericpipeline: <type 'exceptions.AttributeError'> 2019-10-07 20:19:06 ERROR genericpipeline: 'NoneType' object has no attribute 'write' 2019-10-07 20:19:06 ERROR genericpipeline: 2019-10-07 20:19:06 ERROR genericpipeline: LOFAR Pipeline finished unsuccesfully. 2019-10-07 20:19:06 WARNING genericpipeline: recipe genericpipeline completed with errors Results:

[lmorabit]

Which branch are you using? I think you're on the master when it would be better to use the new.

The error message you are getting has to do with the LoTSS coverage, not LBCS. Is there anything written in /net/rijn/data2/rvweeren/A2256_LB/prefactor_output//LB-Delay-Calibration/delay_calibrators.csv ?

[rvweeren]

Using "new"

[rvweeren]

/net/rijn/data2/rvweeren/A2256_LB/prefactor_output//LB-Delay-Calibration/delay_calibrators.csv

is not created.

https://vo.astron.nl/lbcs/lobos/cone/form does not find any LBCS source nearby (I think because this region is not covered in this catalog version) and that causes the error I think (and delay_calibrators.csv not being created)

[rvweeren]

was using the options:

! do_download = True ! continue_without_lotss = True

[rvweeren]

@tikk3r also mentioned having an issue with this....

[lmorabit]

Hmm. I see in the "new" branch that line 127 of long_baseline_pipeline/plugins/PipelineStep_DownloadCats.py should be:

if os.path.isfile( outfile ):

Regardless, if delay_calibrators.csv is not created then there isn't LBCS coverage. That is the correct VO link. You can get around this by creating your own delay_calibrators.csv file. The new branch checks for the existence of the file first but if you really want to force it you can set

! do_download = False

[tikk3r]

I have seen it happen. The "Good sources" part comes from the LBCS query. Querying Neal's catalogue at http://www.jb.man.ac.uk/~njj/lbcs_stats.sum directly yields more sources (it also contains duplicates I noticed).

[lmorabit]

I am hesitant to link a pipeline to a personal unvetted database. I think it's fine if you know what you are doing but Neal takes the lbcs data physically on a hard drive to astron to update the VO. It probably hasn't happened yet for your field.

[tikk3r]

Oh yeah, I didn't mean to imply we should point the pipeline there :) Good to know the VO is alive and gets updated once in a while!

[lmorabit]

@rvweeren if you need, myself or @tikk3r can provide an example delay_calibrators.csv file for you to copy the format if you want to use Neal's catalogue for this.

[rvweeren]

I now use a manual delay_calibrators.csv file from @tikk3r and set do_download = False. Then the pipeline indeed continues fine. A "print/screen output" that the user should provide a manual csv file (with an example) might be helpful, unless of course we expect this to never occur once the VO server is update to date (not sure how good LBCS coverage is now...)

[lmorabit]

LBCS should almost be complete, but I am not sure when the VO will be updated next. We do need to clean up some of the error messages (not high priority at the moment but will eventually get around to it).

[nealjackson]

LBCS is very nearly (97%) complete now (see http://www.jb.man.ac.uk/~njj/lbcs_stats.png and lbcs_stats.sum). The archive is now in Hertfordshire at the UK-LOFAR site, since we can copy all the data products there easily from Manchester. It is awaiting discussion with Martin about a VO-type server to do cone searches. The version in ASTRON is rather out of date now, as I haven't been there for a while.

[mhardcastle]

Yeah, we should have that discussion, sorry! It's a very small amount of work to implement a web-based cone search... happy to talk about getting it done and integrated into the pipeline.

[lmorabit]

Fixed with commit 18039d