search

Keck-DataReductionPipelines / KCWI_DRP

KCWI python DRP
BSD 3-Clause "New" or "Revised" License
10 stars 14 forks source link

create kiwi.proc without executing pipeline #33

Open astronomeara opened 3 years ago

astronomeara commented 3 years ago

in the new pipeline, there is no equivalent to the initial step in the IDL that creates the kiwi.proc file. Insopecting this file before running the whole pipeline is very useful.

MNBrod commented 3 years ago

Definitely worth looking into. This should be a fairly straightforward feature to add, just add a script (or option) that only ingests files and doesn't proceed to any other primitives

lucarizzi commented 3 years ago

It would be useful to know what kind of information you are used to retrieve from kcwi.proc.

The reason why that file is useful is because sometimes the old pipelines could not correctly associate data with cals, for example. The new pipeline uses the stateID, not the instrument configuration, so it's hard to imagine that it would provide useful information on that.

Also, the old pipeline is driven by kcwi.proc. There really is no file that drives the current pipeline. We use kcwi.proc to archive calibrations and retrieve them.

So let's start by figuring out what an observer might want to know before running the pipeline and let's make sure that we provide all the required information.

astronomeara commented 3 years ago

The main reason is to see a listing of the files and to make sure the pipeline understands the files' purposes correctly. Absent something like this, is there an easy way, without actually running the whole pipeline, to see if the pipeline understands the files? A good example case was the February run: we knew something was amiss (or at least I did) because none of the files were showing up in kcwi.proc.

My primary goal is not to change the workflow, but instead to avoid instances where we have to run the pipeline end to end before we find something is wrong. Perhaps that is just less common in the new framework, and again, I am being too attached to the old workflow.

lucarizzi commented 3 years ago

Very good. In this case we should be looking at the functionality required which is:

I think we can implement that.

Using your example, the reason why the IDL pipeline failed during that night is that it infers the cal association from instrument-generated keywords. In the specific example: the cal unit was not positioned correctly, which caused the CALTYPE keyword to be incorrect.

Luckily, we also have declarative keywords, generated by the observing system. The corresponding keyword would be IMTYPE, instead of CALTYPE. The IMTYPE, being declared, was correct, which is how we eventually fixed the files.

The new pipeline uses the declarative keywords, not the inferred keywords.

Still, problems might always happen so I think John makes a good case for having a way to look at possible calibration association issues before we run the pipeline. Let’s move this to a KCWI_DRP project feature request and add it to our to do list.

From: astronomeara @.> Date: Tuesday, April 6, 2021 at 7:41 PM To: Keck-DataReductionPipelines/KCWI_DRP @.> Cc: Luca Rizzi @.>, Assign @.> Subject: Re: [Keck-DataReductionPipelines/KCWI_DRP] create kiwi.proc without executing pipeline (#33)

The main reason is to see a listing of the files and to make sure the pipeline understands the files' purposes correctly. Absent something like this, is there an easy way, without actually running the whole pipeline, to see if the pipeline understands the files? A good example case was the February run: we knew something was amiss (or at least I did) because none of the files were showing up in kcwi.proc.

My primary goal is not to change the workflow, but instead to avoid instances where we have to run the pipeline end to end before we find something is wrong. Perhaps that is just less common in the new framework, and again, I am being too attached to the old workflow.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/Keck-DataReductionPipelines/KCWI_DRP/issues/33#issuecomment-814617650, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAEAFHJBNPCXLGNZZYPS66DTHPV67ANCNFSM42NNJD3A.