0.7.288 something broken with apero_astrometrics.py and new targets

larnoldgithub commented 6 months ago

Hi Neil We have just completed the observations of an discretionary program and I wanted to process the data asap with the 0.7.288.

I set SCIENCE_TARGETS =nameofnewobj in my ini.file but get an error message from Apero :

I added the target in the obj db (although it's not relevant for this H>12 star) but still get the error message.

The weird thing is that if I set SCIENCE_TARGETS = All, it works, no error message. But I don't want (can't yet) to process all stars, only that new single object for the moment. I don't understand why Apero is behaving this way.

L

njcuk9999 commented 6 months ago

So the astrometric database is used for more than just the pm now, we rely on science targets being there for multiple sets including the template creation (mostly for the name).

Its a weekend so I'm not near my computer but I guess nameofnewobj is not your target name and you redacted this information from github? If so I assume your object is the last one in the list (a TOI object) I see it looks like you entered a Gaia DR2 id as the input name- I would suggest you check exactly what the name is in the header (you could use precheck to tell you what apero is looking for - it will tell you the name.... maybe you need to add whitespace between i.e. TOI XXXXXYYYYY I see that only TOI-XXXXXYYYYY is in the aliase list, I always recommend adding a few alaises - that might work (and precheck will tell you when infact it is in the database.

If you confirm with precheck that it is in the database then we might have a bug. Again its a weekend I'm not going to debug it now. But if you really want data now you never HAVE to use apero processing you can do the steps manually (apero_processing is just a convience).

i.e. you could do the following commands:

apero_preprocess_spirou.py {night} {filename}

apero_extract_spirou.py {night} {filename}

apero_fit_tellu_spirou.py {night} {filename}

If you are unsure which commands to run you can run apero_processing.py for another object for one night in test mode --test=True

and it will print a list of commands that will be run - you can easily run these one by one for your filename(s) or add them to a run.in file as follows:

This is the full run.ini file


# Note this is a example file
#   Please copy this before making changes
# FOR USE WITH VERSION 0.7.288 (2023-09-06)

# -----------------------------------------------------------------------------
# Core variables
# -----------------------------------------------------------------------------
# Define run name
RUN_NAME = My custom commands

# Define whether to send email (required yagmal)
#   to install yagmail: pip install yagmail
# Note this currently is untested
SEND_EMAIL = False

# Define email address to send/recieve (send to self)
# Note this currently is untested
EMAIL_ADDRESS = None

# Define single observation directory (for all nights to "All")
RUN_OBS_DIR = All

# Define observation directories to exclude ("None" means no filter)
#    EXCLUDE_OBS_DIRS = 2018-07-31, 2018-08-05, 2018-12-18, 2018-12-19, 2019-01-17
EXCLUDE_OBS_DIRS = All
# Define observation directories to include ("All" means no filter)
#    INCLUDE_OBS_DIRS = 2018-07-31, 2018-08-01, 2018-08-02, 2018-08-03
INCLUDE_OBS_DIRS = All

# Define a pi name list for filtering by ("All" means no filter)
PI_NAMES = All

# define reference observation directory
REF_OBS_DIR = 2020-08-31

# Number of cores (if negative uses N-ABS(CORES), if zero uses N-1 cores)
CORES = 5

# Stop on exception
STOP_AT_EXCEPTION = False

# Run in test mode (does not run codes)
TEST_RUN = False

# Whether to process engineering data
USE_ENGINEERING = False

# Whether this is a trigger run
TRIGGER_RUN = False

# Use odometer reject list
USE_REJECTLIST = True

# Recalculate templates if template already exists
RECAL_TEMPLATES = True

# Update object database from googlesheet - only recommended if doing a
#    full reprocess with all data (will cause inconsistencies if googlesheet
#    has been updated and you are not reprocessing all new data)
UPDATE_OBJ_DATABASE = False

# Update reject database from googlesheet
UPDATE_REJECT_DATABASE = True

# Update the index database. WARNING only do this if precheck or a previous
#   apero_processing has been run. This will result in bad things if the
#   index database is not update to date.
UPDATE_FILEINDEX_DATABASE = True

# Select which databases to update. WARNING. You must be 100% sure that
#   databases left out of this list are up to date, otherwise APERO will
#   fail in bad ways. Options are "All", or any combination of the following:
#   "raw", "tmp', "red", "out"
UPDATE_IDATABASE_NAMES = All

# -----------------------------------------------------------------------------
# Run information
# -----------------------------------------------------------------------------
#
#   Format:
#
#   id##### = command
#
#   Must start with "id" followed by a number
#     The number identifies the order to process in
#
#   If command is a sequence will process all in defined sequence
#      valid sequences are:
#           pp_seq, pp_seq_opt, full_seq,
#           limited_seq, ref_seq, calib_seq,
#           tellu_seq, science_seq, eng_seq

id00000 = apero_preprocess_spirou.py {night} {filename}

id00001 = apero_extract_spirou.py {night} {filename}

id00002 = apero_fit_tellu_spirou.py {night} {filename}

you can then run this through apero_processing - it wont use a sequence therefore you shouldn't have the problem above.

Obviously update the stuff above "Run information" with the default stuff you use from another run.ini - but you do not need any RUN_XXXX or SKIP_XXXX stuff (as no sequences are being used) apero should just run your commands in parallel where possible (i.e. it will run all your preprocess if the id's are in sequential order)

njcuk9999 commented 6 months ago

It may also have to do with this, we just got an error updating the object database:

20:14:04.342-**|DatabaseError|E[00-002-00047] Pandas.to_sql <class 'sqlalchemy.exc.DataError'>: (mysql.connector.errors.DataError) 1406 (22001): Data too long for column 'TEFF_SOURCE' at row 1380

It seems your TEFF_SOURCE was too long (it needs to fit in a header so < 80 characters), I've edited the row.

If you weren't getting this error it means your astrometric database is not up-to-date, so another fix may be to make sure the following is True in your run.ini file:

# Update object database from googlesheet - only recommended if doing a
#    full reprocess with all data (will cause inconsistencies if googlesheet
#    has been updated and you are not reprocessing all new data)
UPDATE_OBJ_DATABASE = True

If it isn't you need to update your astrometric database yourself i.e. with:

apero_database.py --update --dbkind='astrom'

In general we keep the database "offline" in case mistakes are made - so you do need to update it if you add objects!

larnoldgithub commented 6 months ago

Thanks Neil ! the UPDATE_OBJ_DATABASE = True solved the issue.
Sorry for the nameofnewobj I was not sure this ticket on github was public or not.

it's good to know I can have lines like id00000 = apero_preprocess_spirou.py {night} {filename} in an ini file.

njcuk9999 commented 6 months ago

It is public so yes probably best to leave the name out as you did!

njcuk9999 / apero-drs

0.7.288 something broken with apero_astrometrics.py and new targets #747