search

askap-vast / vast-tools

A collection of tools that are useful for the VAST project and for exploration of results from the VAST Pipeline.
https://vast-survey.org/vast-tools/
MIT License
8 stars 0 forks source link

Incorrect epoch numbers in pipeline #321

Open ddobie opened 2 years ago

ddobie commented 2 years ago

@ajfox123 has pointed out that the epoch column in the pipeline source dataframe is incorrect. For example,

from vasttools.pipeline import Pipeline
pipe = Pipeline()
piperun = pipe.load_run('combined')
my_source = piperun.get_source(5338477)
meas = my_source.measurements
display(meas.epoch)

The offending line is here but off the top of my head I'm not sure of the best way to get the correct epoch - it should be easy to pull it from the image path, but I'm sure there's nicer ways to do it.

ddobie commented 2 years ago

@marxide you're probably the person to ask - are the epoch names/numbers stored anywhere in the pipeline files? From a quick search my assumption is no.

ajstewart commented 2 years ago

Ah yes this was a conscious 'feature' on my part as you're right in that the pipeline has no context of the epoch numbers that have been assigned to the pilot survey. It is instead intended to reflect what 'epoch number' the measurement is of the respective source. So really epoch might be an unfortunate naming.

Really the pipeline was designed as a generic ASKAP one so the idea of epochs wasn't built in to be recorded. It does have an 'epoch mode' for image entry but again this is not recorded.

If you wanted this functionality you'd probably have to write some specific get epoch code in vast tools for the pilot data. I think the epoch might be stated in the image path?

ddobie commented 2 years ago

It's probably not worth actually adding that functionality now that I think about it. But I might just rename the column for clarity.

ddobie commented 2 years ago

Actually that's not necessarily easy because the epoch column is necessary for everything else...

ajstewart commented 2 years ago

Ah yes you're right, I see where the disconnect comes from and then user confusion of always dealing with the epoch names that have been given to the pilot data and then not seeing them in the pipeline data to 'access' the measurements.

Given that the epochs are so ingrained it might be worth writing something to attempt to pull the epoch numbers from the image path attached to the source and replace the epoch values. I think the directory structure is standardised so it might not be too difficult. Probably something like a check_for_pilot_epochs flag in the get_source that is default True.

You'd have to deal with the duplicates with the tiles and non epoch mode combined however, in that a source can be picked up multiple times in one 'epoch'. Code to rename the epochs appropriately I think already exists in the planet stuff.

You'd also have to hope that the names continue to be standardised as what happens if one epoch can't be found? I guess just fall back to the numbers.

Edit: Here is the duplicate epoch function: https://github.com/askap-vast/vast-tools/blob/a4b30e51ecdf6bbad591e9d2b691e63342f54cc3/vasttools/query.py#L1258-L1282