search

njcuk9999 / apero-drs

A PipelinE to Reduce Observations - The DRS for SPIRou (CFHT)
MIT License
12 stars 0 forks source link

0.7.288 : in .ini file SCIENCE_TARGETS requires *all* listed objects to be in the astrometric db ? #780

Open larnoldgithub opened 3 weeks ago

larnoldgithub commented 3 weeks ago

Hi Neil Follow up after the email I sent you - I'm opening this ticket

07:10:46.034-!!|PROC| Could not find all objects in SCIENCE_TARGETS in astrometric database. Please
07:10:46.034-!!|PROC|   check SCIENCE_TARGETS, add missing objects to astrometric database or remove
07:10:46.034-!!|PROC|   from SCIENCE_TARGETS and try again.
07:10:46.037-!!|PROC| objnames=blablabla
07:10:46.314-**|PROC| ***************************************************************************
07:10:46.334-@!|PROC| W[40-003-00005]: Recipe apero_processing has NOT been successfully completed
07:10:46.345-**|PROC| ***************************************************************************

Its seems that with SCIENCE_TARGETS= obj1, obj2, obj3 and with obj3 not in the astrometric db, the processing fails to start. How to handle a list of objects with some not worth being in the astrometric db ?

I do want to process only a limited number of objects, with a bunch not in the astrometric db (too faint objects).

larnoldgithub commented 3 weeks ago

additional question: does SCIENCE_TARGETS accept DRSOBJN ?

njcuk9999 commented 3 weeks ago

objects with some not worth being in the astrometric db

They should all be in the astrometric database - for lbl and air if they have no proper motion you can use one of the arguments (see the --help) please comment of these for now and I'll add a tag to them for NO_PM - in v0.8 we'll use this tag

SCIENCE_TARGETS crossmatches against the database - the preferred name is the DRSOBJN but any alias will work.

larnoldgithub commented 3 weeks ago

I have 150 stars to process for 24A, some of them aren't in Simbad. How can I proceed then ?

larnoldgithub commented 3 weeks ago

those are 'custom targets' entered by PI in our Kealahou system.

larnoldgithub commented 3 weeks ago

I'm running another Apero for the quicklook, just for one night, with SCIENCE_TARGETS=All and I get

00:42:11.031-@!|PP_SCI[00159]| W[10-010-00005]: Object LP_695M351 is not in the astrometric database. Using
00:42:11.031-@!|PP_SCI[00159]|  header values for astrometric parameters

This is the usual warning, so just a warning, not an error as with the processing mentioned above.

It seems that SCIENCE_TARGETS=All and SCIENCE_TARGETS=obj1, obj2,obj3 aren'n processed the same way ?

njcuk9999 commented 3 weeks ago

I have 150 stars to process for 24A, some of them aren't in Simbad. How can I proceed then ?

Can you not process all targets in 24A? Why do you need such a big list, how many targets are not in your list?

It seems that SCIENCE_TARGETS=All and SCIENCE_TARGETS=obj1, obj2,obj3 aren'n processed the same way ?

I'm not sure what you mean here. SCIENCE_TARGETS=All does not filter by target name so just gets all .fits for the raw directory. For SCIENCE_TARGETS=obj1,obj2,obj3 I have to look in the database for aliases hence why they have to be there. How does APERO know obj1 is not an alias to another target? It is dangerous to process objects not in the database because it means no one has checked for an entry already existing under another name. That being said I should be able to just try to find any object name you have on disk - right now I don't know how much code change this would be, I'll have to check.

larnoldgithub commented 3 weeks ago

I have 150 stars to process for 24A, some of them aren't in Simbad. How can I proceed then ? Can you not process all targets in 24A? Why do you need such a big list, how many targets are not in your list?

I only want to process the objects of 24A (new or already observed before); I'll recalculate the template for those, so I set INCLUDE_OBS_DIRS =All. If I do SCIENCE_TARGETS=All too, then I'll repreocess all SPI data, which I don't want. I only want to reprocess the 24A objects and update the template for those.

Is there better way to proceed for that ?

larnoldgithub commented 3 weeks ago

It seems that SCIENCE_TARGETS=All and SCIENCE_TARGETS=obj1, obj2,obj3 aren'n processed the same way ? I'm not sure what you mean here. SCIENCE_TARGETS=All does not filter by target name so just gets all .fits for the raw directory. For SCIENCE_TARGETS=obj1,obj2,obj3 I have to look in the database for aliases hence why they have to be there. How does APERO know obj1 is not an alias to another target? It is dangerous to process objects not in the database because it means no one has checked for an entry already existing under another name. That being said I should be able to just try to find any object name you have on disk - right now I don't know how much code change this would be, I'll have to check.

What I meant is what you're explaining: SCIENCE_TARGETS=All doesn't filter, ='obj1,obj2,obj3' does filter. So I understand that if I would set SCIENCE_TARGETS=All instead of the long list of objects, I would have no issues reducing all objects, but as I said this is not what I want to do.

Let me know if I'll still unclear!

njcuk9999 commented 2 weeks ago

Is there better way to proceed for that ?

Currently no objects have to be in the astrometric database.

larnoldgithub commented 6 days ago

Hi @njcuk9999 good to hear this morning that fixing this issue is in your immediate todo list ;)

A full reprocessing may be worth with the new CCF of the .7.290. Do you need a test of the .290 with the minidata2 ? I may be able to run a test in the coming days. Or could I run the CCF on the existing .288 files ?

njcuk9999 commented 5 days ago

Urm there were some changes to the background which start with the preprocessing so probably best to run the mini data fresh.

You can of course try to run existing files but I can't promise you wont get errors.

njcuk9999 commented 1 day ago

So I've been looking into doing this, so I can do this but:

In the future, there may be other reasons why these won't work as having an object database is being used in multiple places (not just for the BERV).

I would still strongly recommend adding all objects to the object database.

It will also be much slower to find the targets as instead of currently just looking in the object database I would have to do the following:

  1. Make sure the file index database for all raw observations is up-to-date
  2. Get the reject list
  3. Check all objects against the reject list
  4. Get a list of unique objects
  5. Check against the database and all aliases

This takes a lot of time every time I want to find "unique" objects when one could just get a list of objects from the object database very quickly.

Also there is no way to know whether the user made a typo or really wants these targets - I've found it very useful to have a "catch" when writing the wrong target name in SCIENCE_TARGETS (i.e. WASP-90 instead of WASP-80)

njcuk9999 commented 1 day ago

okay one thing I could do is when checking whether they are all in the database ask the user to confirm if targets that are missing from the database are okay to be added to processing... you'll get a pause and have to confirm each one that was found to be missing but otherwise, I see no way to "check" these are the objects you want.