Custom header keywords trough chimera plugins

wschoenell commented 9 years ago

This PR fixes #50. It implements a way to have flexible headers for each observatory by the means of controller plugins. The changes needed on chimera code (and done here) are:

Move the metadata information of the camera instrument from _saveImage() to a new method getMetadata() to be consistent with the other instruments structure.
Add a new variable to the ChimeraObject class named __metadataOverrideMethod__. This variable, if different of None, will store the location of the object which must contain a getMetadata() method and will replace the instrument's (and site) original getMetadata() method.
Add two new methods on ChimeraObject: setMetadataMethod() and getMetadataOverride(). The first one is used if one wants to override the default metadata method of a instrument by another object on chimera's namespace. The usage is something like:

myInstrument = getManager().getProxy('/Instrument/name', lazy=True)
myInstrument.setMetadataMethod('/HeaderOverrider/name')

The second one should be called on all getMetadata() methods before the instrument's metadata itself being written. The call would be, for example:

def getMetadata(self, request):
    # Check first if there is metadata from an metadata override method.
    md = self.getMetadataOverride(request)
    if md is not None:
        return md
    # If not, just go on with the instrument's default metadata.
    return [(... instrument s metadata here ...)]

Other minor changes:

Corrected minor typos and PEP8 on fakecamera.py and site.py
Added CHECKSUM and DATASUM header keywords to all files saved by chimera.
Moved extras to a class variable and created a method to extract it. I think that extras option on _saveImage() could go away sometime, but now I left this way for compatibility with the existent camera drivers. Maybe when doing the wcs move to the Image class (#93)

An template of a controller that does modifications on a fits header can be seen here: https://www.github.com/astroufsc/chimera-headers

When running chimera with that controller configured for all the fake instruments, one haves a fits header like the below. All headers with Custom. in the beginning of the comment where changed by that controller. The others are static and cannot be changed.

SIMPLE  =                    T / conforms to FITS standard                      
BITPIX  =                   16 / array data type                                
NAXIS   =                    2 / number of array dimensions                     
NAXIS1  =                  512                                                  
NAXIS2  =                  512                                                  
EXTEND  =                    T                                                  
BZERO   =                32768                                                  
DATE    = '2015-07-29T15:15:22.233742' / date of file creation                  
AUTHOR  = 'chimera '           / Observatory Automation System                  
OBJECT  = 'object  '           / name of observed object                        
SITE    = 'T80S    '           / Custom. Site name (in config)                  
LATITUDE= '-70:48:20.480'      / Custom. Site latitude                          
LONGITUD= '-30:10:04.310'      / Custom. Site longitude                         
ALTITUDE= '2187    '           / Custom. Site altitude                          
DATE-OBS= '2015-07-29T15:15:21.082973' / Custom. Date exposure started          
EXPTIME =                  1.0 / Custom. exposure time in seconds               
IMAGETYP= 'object  '           / Custom. Image type                             
SHUTTER = 'OPEN    '           / Custom. Requested shutter state                
INSTRUME= 'Fake Cameras Inc.'  / Custom. Name of instrument                     
CCD     = 'Fake CCDs Inc.'     / Custom. CCD Model                              
CCD_DIMX=                  512 / Custom. CCD X Dimension Size                   
CCD_DIMY=                  512 / Custom. CCD Y Dimension Size                   
CCDPXSZX=                    9 / Custom. CCD X Pixel Size [micrometer]          
CCDPXSZY=                    9 / Custom. CCD Y Pixel Size [micrometer]          
CRPIX1  =                  255 / Custom. coordinate system reference pixel      
CRPIX2  =                  255 / Custom. coordinate system reference pixel      
CD1_1   = 0.000515662015617741 / Custom. transformation matrix element (1,1)    
CD1_2   =                 -0.0 / Custom. transformation matrix element (1,2)    
CD2_1   =                  0.0 / Custom. transformation matrix element (2,1)    
CD2_2   = 0.000515662015617741 / Custom. transformation matrix element (2,2)    
DOME_MDL= 'Fake Domes Inc.'    / Custom. Dome Model                             
DOME_TYP= 'Classic '           / Custom. Dome Type                              
DOME_TRK= 'Track   '           / Custom. Dome Tracking/Standing                 
DOME_SLT= 'Closed  '           / Custom. Dome slit status                       
FOCUSER = 'Fake Focus v.1'     / Custom. Focuser Model.                         
FOCUS   =                 3500 / Custom. Focuser position used for this observat
TELESCOP= 'Fake Telescopes Inc.' / Custom. Telescope Model                      
OPTICS  = 'Newtonian'          / Custom. Telescope Optics Type                  
MOUNT   = 'Mount type Inc.'    / Custom. Telescope Mount Type                   
APERTURE=                100.0 / Custom. Telescope aperture size [mm]           
F_LENGTH=               1000.0 / Custom. Telescope focal length [mm]            
F_REDUCT=                  1.0 / Custom. Telescope focal reduction              
RA      = '08:29:03.740'       / Custom. Right ascension of the observed object 
DEC     = '-03:00:00.000'      / Custom. Declination of the observed object     
EQUINOX =               2000.0 / Custom. coordinate epoch                       
ALT     = '+21:09:51.623'      / Custom. Altitude of the observed object        
AZ      = '+340:17:31.456'     / Custom. Azimuth of the observed object         
WCSAXES =                    2 / Custom. wcs dimensionality                     
RADESYS = 'ICRS    '           / Custom. frame of reference                     
CRVAL1  =    127.2655845654916 / Custom. coordinate system value at reference pi
CRVAL2  =   -3.000000000000017 / Custom. coordinate system value at reference pi
CTYPE1  = 'RA---TAN'           / Custom. name of the coordinate axis            
CTYPE2  = 'DEC--TAN'           / Custom. name of the coordinate axis            
CUNIT1  = 'deg     '           / Custom. units of coordinate value              
CUNIT2  = 'deg     '           / Custom. units of coordinate value              
CHECKSUM= 'U8a3U5Z3U5a3U5Y3'   / HDU checksum updated 2015-07-29T17:15:23       
DATASUM = '192873200'          / data unit checksum updated 2015-07-29T17:15:23 
CCD-TEMP=    20.30499427111227 / Custom. CCD Temperature at Exposure Start [deg.
CHM_ID  = '0ab51b1a2599217c3cfc2e650621477d2c'                                  
END

wschoenell commented 9 years ago

We (me and @tribeiro) have been using this code for a while without any problem. Will give 2 weeks more to people complain, then merge.

phsilva commented 9 years ago

Can someone explain what problem this PR solves? I really don't get it.

wschoenell commented 9 years ago

The ChimeraObject now haves __metadataOverrideMethod__ attribute which can be set externally to a chimera location i.e. /MyHeaderController/name which contains an alternative getMetadata method.

When you take an exposure, the camera in the process will ask the getProxy(__metadataOverrideMethod__).getMetadata(request) instead the instrument's default getMetadata method if __metadataOverrideMethod__ is not None.

This is all to be able to set custom fits headers based on custom getMetadata methods. This is one of the requirements to make the fits header files produced by T80 (and any other robotic telescope) to be compliant with the pipeline software.

An practical example of how the override process works can be seen here: https://github.com/astroufsc/chimera-headers/blob/ab708c92560cc8e6b6866f3e0da37f427db2664f/chimera_headers/controllers/headers.py#L86-L87

phsilva commented 9 years ago

To be compliant with pipeline software we need headers, in any way, even manually if someone would like this task.

All you want is to get all headers from everything possibly running that affects that exposure, right!?

This Header instrument as it is now will cause huge performance issues by moving the getMetadata class to its own control. Every call to self.instrument.X is a remote call now instead of a local one if the responsible for it was the original instrument.

I think Site is the one to gather all info about what instruments compose an observatory or part of it, if you take an image using a camera, it asks the Header controller to get its headers, Header controller goes on the site containing that camera (it must be only unique) and iterate over all instruments listed there asking getMetadata on that instrument (which knows how to compute it locally). That seems sane in terms of performance and in terms of coupling.

PS: Site is weird and maybe a misnomer, in this context we can think of site as a group of instruments that make sense together.

wschoenell commented 9 years ago

On Thu, Aug 6, 2015, 21:17 Paulo Henrique Silva notifications@github.com wrote:

To be compliant with pipeline software we need headers, in any way, even manually if someone would like this task.

All you want is to get all headers from everything possibly running that affects that exposure, right!?

Yes and no. What we want is to have headers of everything but also to have them on the standard which meets the observatory/pipeline software standards. If, for example, the observatory standard is EXP-TIME for exposure time and the chimera standard is EXPTIME, things will not work. Of course this is a silly example. I can give you more realistic ones.

This Header instrument as it is now will cause huge performance issues by moving the getMetadata class to its own control. Every call to self.instrument.X is a remote call now instead of a local one if the responsible for it was the original instrument.

Maybe I did not got point but I don't see how this could interfere on the performance of chimera. We have been using many remote calls all over the code. Every call to self.instrument.getMetadata() which is only for metadata purposes will be a to getManager().getProxy(self.__metadataOverrideMethod__).getMetadata(request) if there is an metadata controller. This is the way that it is done now is doing a remote call likewise the call over an controller... No?

https://github.com/astroufsc/chimera/blob/3056296fc951d09a3765b06499a9c8a001363129/src/chimera/controllers/imageserver/imagerequest.py#L147-L158

I think Site is the one to gather all info about what instruments compose an observatory or part of it, if you take an image using a camera, it asks the Header controller to get its headers, Header controller goes on the site containing that camera (it must be only unique) and iterate over all instruments listed there asking getMetadata on that instrument (which knows how to compute it locally). That seems sane in terms of performance and in terms of coupling.

PS: Site is weird and maybe a misnomer, in this context we can think of site as a group of instruments that make sense together.

I don't really see how to do this without changing all the way how chimera gets headers from the instruments.

— Reply to this email directly or view it on GitHub https://github.com/astroufsc/chimera/pull/97#issuecomment-128478856.<img src=" https://ci4.googleusercontent.com/proxy/zHxB5hP5kJWclZO5wMlf67Px3befD42ccwXw_7rqLwyVE8vr7Nuatnv9u1jTJyF3nZJIwpa-soe9DczxwiDWI3ZYGZoQdGkrZFcruzg5PWbhSsBv9LtQSWlUEY2MdmnOEKTUB7mNsZoeI66Zra5xJU736AxcsQ=s0-d-e1-ft#https://github.com/notifications/beacon/AByXk27T3_Z0IkwI-rZo5J_AJfvv9WHOks5ok6pPgaJpZM4FiHmR.gif ">

phsilva commented 9 years ago

hard to see what I wrote and what you answered...

wschoenell commented 9 years ago

Again, let's see if the parser gets it now.

On Thu, Aug 6, 2015 at 9:17 PM Paulo Henrique Silva < notifications@github.com> wrote:

To be compliant with pipeline software we need headers, in any way, even manually if someone would like this task.

All you want is to get all headers from everything possibly running that affects that exposure, right!?

Yes and no. What we want is to have headers of everything but also to have them on the standard which meets the observatory/pipeline software standards. If, for example, the observatory standard is EXP-TIME for exposure time and the chimera standard is EXPTIME, things will not work. Of course this is a silly example. I can give you more realistic ones.

This Header instrument as it is now will cause huge performance issues by moving the getMetadata class to its own control. Every call to self.instrument.X is a remote call now instead of a local one if the responsible for it was the original instrument.

Maybe I did not got point but I don't see how this could interfere on the performance of chimera. We have been using many remote calls all over the code. Every call to self.instrument.getMetadata() which is only for metadata purposes will be a to getManager().getProxy(self.__metadataOverrideMethod__).getMetadata(request) if there is an metadata controller. This is the way that it is done now is doing a remote call likewise the call over an controller... No?

https://github.com/astroufsc/chimera/blob/3056296fc951d09a3765b06499a9c8a001363129/src/chimera/controllers/imageserver/imagerequest.py#L147-L158

I think Site is the one to gather all info about what instruments compose an observatory or part of it, if you take an image using a camera, it asks the Header controller to get its headers, Header controller goes on the site containing that camera (it must be only unique) and iterate over all instruments listed there asking getMetadata on that instrument (which knows how to compute it locally). That seems sane in terms of performance and in terms of coupling.

I don't really see how to do this without changing all the way how chimera gets headers from the instruments.

PS: Site is weird and maybe a misnomer, in this context we can think of site as a group of instruments that make sense together.

— Reply to this email directly or view it on GitHub https://github.com/astroufsc/chimera/pull/97#issuecomment-128478856.

wschoenell commented 8 years ago

We have been using this since last year on T80 with no impact on the performance of the exposures. Merging until we don't have any more elegant way out to change the way chimera does header on images.

astroufsc / chimera

Custom header keywords trough chimera plugins #97