search

aps-8id-dys / ipython-8idiuser

8-ID-I ipython configuration for bluesky (and other)
1 stars 1 forks source link

handle metadata per initial request #25

Closed prjemian closed 5 years ago

prjemian commented 5 years ago

Per initial request in #9, handle the existing metadata.

prjemian commented 5 years ago

Note this previous remark from @sureshnaps:

Regarding Sinisa's system, we will discuss before we work on it as I am doing some developments to aid this as well.

prjemian commented 5 years ago

here is the python file that writes the metadata using PVs written by spec to registers /home/beams/8IDIUSER/Python_HDF5_DataExchange/Create_DataExchange_HDF5_8idi.py

spec macros that write to registers: /home/beams/8IDIUSER/local_macros/CCD_macros/write_CCD_MetaData_hdf.mac

Data movement part is handled by the DM system using the 2 macros defined here: /home/beams/8IDIUSER/local_macros/CCD_macros/DM_Workflow_macros.mac

prjemian commented 5 years ago

Summarizing what SPEC does:

  1. contains local source of data to be provided to DM system
  2. writes data to bespoke EPICS PVs
  3. runs python code as unix command to read the PVs and write the HDF5 file
  4. runs python code as unix command to initiate the DM workflow
prjemian commented 5 years ago

An important consideration is that Step 3 must complete before Step 4.

@sureshn: Must Step 4 complete before a new scan starts?

prjemian commented 5 years ago
write_CCD_MetaData_hdf.mac

``` ##some functions used here like max requires ##https://subversion.xray.aps.anl.gov/trac/spec/browser/common/aps/math.mac ##copied to s8spec def write_CCD_MetaData_Registers '{ global batch_pars user_data_folder atten_set attvalue BURST_MODE epics_put("8idi:Reg101",batch_pars["col_beg"]) epics_put("8idi:Reg102",batch_pars["col_end"]) epics_put("8idi:Reg103",batch_pars["row_beg"]) epics_put("8idi:Reg104",batch_pars["row_end"]) epics_put("8idi:Reg105",batch_pars["cols"]) epics_put("8idi:Reg106",batch_pars["rows"]) epics_put("8idi:Reg107",batch_pars["kinetics"]) epics_put("8idi:Reg108",batch_pars["kinwinsize"]) epics_put("8idi:Reg109",batch_pars["slicetop"]) epics_put("8idi:Reg110",attvalue) ##Reg111 and Reg112 are filled down, conditional on dark frames epics_put("8idi:Reg113",batch_pars["ndata0"]) epics_put("8idi:Reg114",batch_pars["ndataend"]) epics_put("8idi:Reg115",batch_pars["preset"]) epics_put("8idi:Reg116",batch_pars["preset_period"]) epics_put("8idi:Reg117",-1) epics_put("8idi:Reg118",batch_pars["spec_scanN"]) epics_put("8idi:Reg119",batch_pars["ccdx"]) epics_put("8idi:Reg120",batch_pars["ccdz"]) epics_put("8idi:Reg121",batch_pars["ring_i_beg"]) epics_put("8idi:Reg122",batch_pars["ring_i_end"]) #epics_put("8idi:Reg123",batch_pars["I0Mon"]) epics_put("8idi:Reg124",batch_pars["burst_mode"]) epics_put("8idi:Reg125",batch_pars["number_of_bursts"]) epics_put("8idi:Reg126",batch_pars["first_usable_burst"]) epics_put("8idi:Reg127",batch_pars["last_usable_burst"]) ##string registers epics_put("8idi:StrReg1",DATAFILE) epics_put("8idi:StrReg2",batch_pars["parent"]) epics_put("8idi:StrReg3",user_data_folder) epics_put("8idi:StrReg4",batch_pars["child"]) epics_put("8idi:StrReg5",rmquotes(batch_pars["datafilename"])) epics_put("8idi:StrReg6",rmquotes(batch_pars["start_time"])) epics_put("8idi:StrReg7",rmquotes(batch_pars["end_time"])) #check to make sure these exist because in case of compression, these are not defined if (!exists("batch_pars","ndark0")) epics_put("8idi:Reg111",-1) if (!exists("batch_pars","ndarkend")) epics_put("8idi:Reg112",-1) if (exists("batch_pars","ndark0")) epics_put("8idi:Reg111",batch_pars["ndark0"]) if (exists("batch_pars","ndarkend")) epics_put("8idi:Reg112",batch_pars["ndarkend"]) if (!exists("batch_pars","beam_i_vacuum")) epics_put("8idi:Reg9",1.0) if (!exists("batch_pars","beam_i")) epics_put("8idi:Reg10",1.0) if (exists("batch_pars","beam_i_vacuum")) epics_put("8idi:Reg9", max(1.0,batch_pars["beam_i_vacuum"])) if (exists("batch_pars","beam_i")) epics_put("8idi:Reg10", max(1.0,batch_pars["beam_i"])) }' def write_CCD_MetaData_hdf '{ global HDF5_METADATA_FILE FULL_INFOFILE_NO_EXT local cmd fn tmp1 ##call python to create hdf metadatafile that can replace batchinfo HDF5_METADATA_FILE = FULL_INFOFILE_NO_EXT ".hdf"; cmd=sprintf("/APSshare/anaconda/x86_64/bin/python /home/beams/8IDIUSER/Python_HDF5_DataExchange/Create_DataExchange_HDF5_8idi.py %s", HDF5_METADATA_FILE); if (PRINT_DEBUG_MSG == 2) printf("%s...\n",cmd) unix(cmd); }' ```

Create_DataExchange_HDF5_8idi.py

``` ##This file is meant to be called from SPEC so that it will create an hdf file ##that can replace .batchinfo ##Reads data from camera and puts it into an hdf5 file import h5py import epics import sys import PythonDict import math import datetime import time class EigerHDF5: def __init__(self): self.index=0 self.obj = PythonDict.PythonDict() def begin(self): self.create_hdf5_file() def create_hdf5_file(self): """ Creates a new file based on name supplied via commandline and populates some metadata. """ filename = sys.argv[1] try: self.f = h5py.File(filename, "w-") except Exception, ex: print "Error: %s" % ex sys.exit(1) # Gets Python Dict stored in other file masterDict = self.obj.returnMasterDict() manufacturerDict = self.obj.returnManufacturerDict() # Metadata dt = h5py.special_dtype(vlen=unicode) data = 0 # Gets Detector Number from a PV (for static fields) detNum = epics.caget('8idi:Reg2') # Gets Geometry Number from a PV (for static fields) geometry_num = epics.caget('8idi:Reg3') # Gets kinetics state from a PV (for static fields) kinetics_state = epics.caget('8idi:Reg107') burst_mode_state = epics.caget('8idi:Reg124') # Gets compression mode state from a PV (for static fields) compression = epics.caget('8idi:Reg8') # get a version number so we can make changes without breaking client code temp = self.f.create_dataset("/hdf_metadata_version", (1,1)) temp[(0,0)] = epics.caget('8idi:Reg1') #same as batchinfo_ver for now ##version 15 (May 2019) is start of burst mode support (rigaku) #######/measurement/instrument/acquisition #######some new acq fields to replace batchinfo temp = self.f.create_dataset("/measurement/instrument/acquisition/dark_begin", (1,1),dtype='uint64') temp[(0,0)] = epics.caget('8idi:Reg111') temp = self.f.create_dataset("/measurement/instrument/acquisition/dark_end", (1,1),dtype='uint64') temp[(0,0)] = epics.caget('8idi:Reg112') temp = self.f.create_dataset("/measurement/instrument/acquisition/data_begin", (1,1),dtype='uint64') temp[(0,0)] = epics.caget('8idi:Reg113') temp = self.f.create_dataset("/measurement/instrument/acquisition/data_end", (1,1),dtype='uint64') temp[(0,0)] = epics.caget('8idi:Reg114') temp = self.f.create_dataset("/measurement/instrument/acquisition/specscan_dark_number", (1,1),dtype='uint64') temp[(0,0)] = epics.caget('8idi:Reg117') temp = self.f.create_dataset("/measurement/instrument/acquisition/specscan_data_number", (1,1),dtype='uint64') temp[(0,0)] = epics.caget('8idi:Reg118') temp = self.f.create_dataset("/measurement/instrument/acquisition/attenuation", (1,1)) temp[(0,0)] = epics.caget('8idi:Reg110') temp = self.f.create_dataset("/measurement/instrument/acquisition/beam_size_H", (1,1)) temp[(0,0)] = epics.caget('8idi:Slit2Hsize.VAL') temp = self.f.create_dataset("/measurement/instrument/acquisition/beam_size_V", (1,1)) temp[(0,0)] = epics.caget('8idi:Slit3Vsize.VAL') self.f["/measurement/instrument/acquisition/specfile"] = epics.caget('8idi:StrReg1',as_string=True) self.f["/measurement/instrument/acquisition/root_folder"] = epics.caget('8idi:StrReg2',as_string=True) self.f["/measurement/instrument/acquisition/parent_folder"] = epics.caget('8idi:StrReg3',as_string=True) self.f["/measurement/instrument/acquisition/data_folder"] = epics.caget('8idi:StrReg4',as_string=True) self.f["/measurement/instrument/acquisition/datafilename"] = epics.caget('8idi:StrReg5',as_string=True) ##standard fields continue temp = self.f.create_dataset("/measurement/instrument/acquisition/beam_center_x", (1,1)) temp[(0,0)] = epics.caget('8idi:Reg11') temp = self.f.create_dataset("/measurement/instrument/acquisition/beam_center_y", (1,1)) temp[(0,0)] = epics.caget('8idi:Reg12') temp = self.f.create_dataset("/measurement/instrument/acquisition/stage_zero_x", (1,1)) temp[(0,0)] = epics.caget('8idi:Reg13') temp = self.f.create_dataset("/measurement/instrument/acquisition/stage_zero_z", (1,1)) temp[(0,0)] = epics.caget('8idi:Reg14') temp = self.f.create_dataset("/measurement/instrument/acquisition/stage_x", (1,1)) temp[(0,0)] = epics.caget('8idi:Reg119') temp = self.f.create_dataset("/measurement/instrument/acquisition/stage_z", (1,1)) temp[(0,0)] = epics.caget('8idi:Reg120') if compression == 1: self.f["/measurement/instrument/acquisition/compression"] = 'ENABLED' else : self.f["/measurement/instrument/acquisition/compression"] = 'DISABLED' if geometry_num == 1: ##reflection geometry temp = self.f.create_dataset("/measurement/instrument/acquisition/xspec", (1,1)) temp[(0,0)] = epics.caget('8idi:Reg15') temp = self.f.create_dataset("/measurement/instrument/acquisition/zspec", (1,1)) temp[(0,0)] = epics.caget('8idi:Reg16') temp = self.f.create_dataset("/measurement/instrument/acquisition/ccdxspec", (1,1)) temp[(0,0)] = epics.caget('8idi:Reg18') temp = self.f.create_dataset("/measurement/instrument/acquisition/ccdzspec", (1,1)) temp[(0,0)] = epics.caget('8idi:Reg17') temp = self.f.create_dataset("/measurement/instrument/acquisition/angle", (1,1)) temp[(0,0)] = epics.caget('8idi:Reg19') if geometry_num == 0: ##transmission geometry temp = self.f.create_dataset("/measurement/instrument/acquisition/xspec", (1,1)) temp[(0,0)] = -1 temp = self.f.create_dataset("/measurement/instrument/acquisition/zspec", (1,1)) temp[(0,0)] = -1 temp = self.f.create_dataset("/measurement/instrument/acquisition/ccdxspec", (1,1)) temp[(0,0)] = -1 temp = self.f.create_dataset("/measurement/instrument/acquisition/ccdzspec", (1,1)) temp[(0,0)] = -1 temp = self.f.create_dataset("/measurement/instrument/acquisition/angle", (1,1)) temp[(0,0)] = -1 #/measurement/instrument/source_begin temp = self.f.create_dataset("/measurement/instrument/source_begin/beam_intensity_incident", (1,1)) temp[(0,0)] = epics.caget('8idi:Reg9') temp = self.f.create_dataset("/measurement/instrument/source_begin/beam_intensity_transmitted", (1,1)) temp[(0,0)] = epics.caget('8idi:Reg10') temp = self.f.create_dataset("/measurement/instrument/source_begin/current", (1,1)) temp[(0,0)] = epics.caget('8idi:Reg121') temp = self.f.create_dataset("/measurement/instrument/source_begin/energy", (1,1)) temp[(0,0)] = epics.caget('8idimono:sm2.RBV') self.f["/measurement/instrument/source_begin/datetime"] = epics.caget('8idi:StrReg6',as_string=True) #/measurement/instrument/source_end (added in January 2019) temp = self.f.create_dataset("/measurement/instrument/source_end/current", (1,1)) temp[(0,0)] = epics.caget('8idi:Reg122') self.f["/measurement/instrument/source_end/datetime"] = epics.caget('8idi:StrReg7',as_string=True) ############################################################################################################### #/measurement/instrument/sample temp = self.f.create_dataset("/measurement/sample/thickness", (1,1)) temp[(0,0)] = 1.0 temp = self.f.create_dataset("/measurement/sample/temperature_A", (1,1)) temp[(0,0)] = epics.caget('8idi:LS336:TC4:IN1') temp = self.f.create_dataset("/measurement/sample/temperature_B", (1,1)) temp[(0,0)] = epics.caget('8idi:LS336:TC4:IN1') temp = self.f.create_dataset("/measurement/sample/temperature_A_set", (1,1)) temp[(0,0)] = epics.caget('8idi:LS336:TC4:OUT1:SP') ##temp[(0,0)] = epics.caget('8idi:pid1.VAL') temp = self.f.create_dataset("/measurement/sample/temperature_B_set", (1,1)) temp[(0,0)] = epics.caget('8idi:LS336:TC4:OUT1:SP') temp = self.f.create_dataset("/measurement/sample/translation", (1,3)) ##x,y,z temp[(0,0)] = epics.caget('8idi:m54.RBV') temp[(0,1)] = epics.caget('8idi:m49.RBV') temp[(0,2)] = epics.caget('8idi:m50.RBV') ##new dataset added on Oct 15,2018 (2018-3) to additionally add table params temp = self.f.create_dataset("/measurement/sample/translation_table", (1,3)) ##x,y,z temp[(0,0)] = epics.caget('8idi:TI3:x.VAL') temp[(0,1)] = epics.caget('8idi:TI3:z.VAL') temp[(0,2)] = epics.caget('8idi:TI3:y.VAL') temp = self.f.create_dataset("/measurement/sample/orientation", (1,3)) ##pitch,roll.yaw temp[(0,0)] = epics.caget('8idi:m52.RBV') temp[(0,1)] = epics.caget('8idi:m53.RBV') temp[(0,2)] = epics.caget('8idi:m51.RBV') #######/measurement/instrument/detector######################### try: self.f["/measurement/instrument/detector/manufacturer"] = manufacturerDict[detNum] except KeyError: self.f["/measurement/instrument/detector/manufacturer"] = 'UNKNOWN' ##self.f["/measurement/instrument/detector/model"] = 'UNKNOWN' ##self.f["/measurement/instrument/detector/serial_number"] = 'UNKNOWN' temp = self.f.create_dataset("/measurement/instrument/detector/bit_depth", (1,1),dtype='uint32') temp[(0,0)] = math.ceil(math.log(masterDict[detNum]["saturation"],2)) temp = self.f.create_dataset("/measurement/instrument/detector/x_pixel_size", (1,1)) temp[(0,0)] = masterDict[detNum]["dpix"] temp = self.f.create_dataset("/measurement/instrument/detector/y_pixel_size", (1,1)) temp[(0,0)] = masterDict[detNum]["dpix"] temp = self.f.create_dataset("/measurement/instrument/detector/x_dimension", (1,1),dtype='uint32') temp[(0,0)] = int(masterDict[detNum]["ccdHardwareColSize"]) temp = self.f.create_dataset("/measurement/instrument/detector/y_dimension", (1,1),dtype='uint32') temp[(0,0)] = masterDict[detNum]["ccdHardwareRowSize"] temp = self.f.create_dataset("/measurement/instrument/detector/x_binning", (1,1),dtype='uint32') temp[(0,0)] = 1 temp = self.f.create_dataset("/measurement/instrument/detector/y_binning", (1,1),dtype='uint32') temp[(0,0)] = 1 temp = self.f.create_dataset("/measurement/instrument/detector/exposure_time", (1,1)) temp[(0,0)] = epics.caget('8idi:Reg115') temp = self.f.create_dataset("/measurement/instrument/detector/exposure_period", (1,1)) temp[(0,0)] = epics.caget('8idi:Reg116') temp = self.f.create_dataset("/measurement/instrument/detector/burst/number_of_bursts", (1,1),dtype='uint32') if burst_mode_state == 1: temp[(0,0)] = epics.caget("8idi:Reg125") else : temp[(0,0)] = 0 temp = self.f.create_dataset("/measurement/instrument/detector/burst/first_usable_burst", (1,1),dtype='uint32') if burst_mode_state == 1: temp[(0,0)] = epics.caget("8idi:Reg126") else : temp[(0,0)] = 0 temp = self.f.create_dataset("/measurement/instrument/detector/burst/last_usable_burst", (1,1),dtype='uint32') if burst_mode_state == 1: temp[(0,0)] = epics.caget("8idi:Reg127") else : temp[(0,0)] = 0 temp = self.f.create_dataset("/measurement/instrument/detector/distance", (1,1)) temp[(0,0)] = epics.caget('8idi:Reg5') if masterDict[detNum]["flatfield"] == 1: self.f["/measurement/instrument/detector/flatfield_enabled"] = 'ENABLED' else : self.f["/measurement/instrument/detector/flatfield_enabled"] = 'DISABLED' if masterDict[detNum]["blemish"] == 1: self.f["/measurement/instrument/detector/blemish_enabled"] = 'ENABLED' else : self.f["/measurement/instrument/detector/blemish_enabled"] = 'DISABLED' temp = self.f.create_dataset("/measurement/instrument/detector/efficiency", (1,1)) temp[(0,0)] = masterDict[detNum]["efficiency"] temp = self.f.create_dataset("/measurement/instrument/detector/adu_per_photon", (1,1)) temp[(0,0)] = masterDict[detNum]["adupphot"] temp = self.f.create_dataset("/measurement/instrument/detector/lld", (1,1)) if masterDict[detNum]["lld"] < 0: temp[(0,0)] = abs(masterDict[detNum]["lld"]) else: temp[(0,0)] = 0 temp = self.f.create_dataset("/measurement/instrument/detector/sigma", (1,1)) if masterDict[detNum]["lld"] > 0: temp[(0,0)] = masterDict[detNum]["lld"] else: temp[(0,0)] = 0 temp = self.f.create_dataset("/measurement/instrument/detector/gain", (1,1),dtype='uint32') temp[(0,0)] = 1 if geometry_num == 0: self.f["/measurement/instrument/detector/geometry"] = 'TRANSMISSION' elif geometry_num == 1: self.f["/measurement/instrument/detector/geometry"] = 'REFLECTION' else : self.f["/measurement/instrument/detector/geometry"] = 'UNKNOWN' if kinetics_state == 1: self.f["/measurement/instrument/detector/kinetics_enabled"] = 'ENABLED' else : self.f["/measurement/instrument/detector/kinetics_enabled"] = 'DISABLED' if burst_mode_state == 1: self.f["/measurement/instrument/detector/burst_enabled"] = 'ENABLED' else : self.f["/measurement/instrument/detector/burst_enabled"] = 'DISABLED' #######/measurement/instrument/detector/kinetics/###### temp = self.f.create_dataset("/measurement/instrument/detector/kinetics/first_usable_window", (1,1),dtype='uint32') if kinetics_state == 1: temp[(0,0)] = 2 else : temp[(0,0)] = 0 temp = self.f.create_dataset("/measurement/instrument/detector/kinetics/last_usable_window", (1,1),dtype='uint32') if kinetics_state == 1: temp1slicetop = epics.caget("8idi:Reg109") temp1size = epics.caget("8idi:Reg108") temp[(0,0)] = int(temp1slicetop/temp1size)-1 else : temp[(0,0)] = 0 temp = self.f.create_dataset("/measurement/instrument/detector/kinetics/top", (1,1),dtype='uint32') if kinetics_state == 1: temp[(0,0)] = epics.caget('8idi:Reg109') else : temp[(0,0)] = 0 temp = self.f.create_dataset("/measurement/instrument/detector/kinetics/window_size", (1,1),dtype='uint32') if kinetics_state == 1: temp[(0,0)] = epics.caget('8idi:Reg108') else : temp[(0,0)] = 0 #######/measurement/instrument/detector/roi/###### temp = self.f.create_dataset("/measurement/instrument/detector/roi/x1", (1,1),dtype='uint32') temp[(0,0)] = epics.caget('8idi:Reg101') temp = self.f.create_dataset("/measurement/instrument/detector/roi/y1", (1,1),dtype='uint32') temp[(0,0)] = epics.caget('8idi:Reg103') temp = self.f.create_dataset("/measurement/instrument/detector/roi/x2", (1,1),dtype='uint32') temp[(0,0)] = epics.caget('8idi:Reg102') temp = self.f.create_dataset("/measurement/instrument/detector/roi/y2", (1,1),dtype='uint32') temp[(0,0)] = epics.caget('8idi:Reg104') ############################################################################################### # Close file self.f.close() newObject=EigerHDF5() newObject.begin() ############################################################################################### ```

DM_Workflow_macros.mac

``` """ **Note the Block commenting syntax: Starting in January 2018, we are trying to move towards SDM Group product as part of DM or Data Management developed by Sinisa Veseli. This primarily has tools to move data to the storage server with cataloging tools. Additionally, this has workflow processing tools which are all done in python along with shell scripts. With Faisal's new xpcs analysis toolkit, we will be using Sun Grid Engine (SGE) for job submission. So it is time to move away from the Active MQ pipeline with the actors, etc. Sinisa has set up workflows: one for data transfer using GridFTP and the other to do full analysis. He has also developed SGE submission and monitoring which is very helpful. The downside as of now is that there is no GUI that shows the job status which was a plus with the old pipeline. A GUI will be developed in the near future. These workflows are stored in ~8idiuser/DM_Workflows/ and in https://subversion.xray.aps.anl.gov/xpcs/DM_Workflows/ """ global DM_WORKFLOW_DATA_TRANSFER, DM_WORKFLOW_DATA_ANALYSIS global DM_WORKFLOW_DATA_TRANSFER_CMD, DM_WORKFLOW_DATA_ANALYSIS_CMD DM_WORKFLOW_DATA_TRANSFER = "xpcs8-01" DM_WORKFLOW_DATA_ANALYSIS = "xpcs8-02" ###DM_WORKFLOW_DATA_TRANSFER = "xpcs8-01-nos8iddata" ###DM_WORKFLOW_DATA_ANALYSIS = "xpcs8-02-nos8iddata" def DM_Workflow_DataTransfer '{ local cmd, hdf_in_data_folder_with_fullpath hdf_with_fullpath = $1 ##usually this is saved in the global variable HDF5_METADATA_FILE global DM_WORKFLOW_DATA_TRANSFER cmd = sprintf("source /home/dm/etc/dm.setup.sh; dm-start-processing-job --workflow-name=%s filePath:%s",DM_WORKFLOW_DATA_TRANSFER,hdf_with_fullpath); DM_WORKFLOW_DATA_TRANSFER_CMD = cmd; printf("DM Workflow call is made for DATA transfer: %s----%s\n",hdf_with_fullpath,date()); unix(cmd); }' def DM_Workflow_DataAnalysis '{ local cmd, hdf_with_fullpath, qmapfile_with_fullpath global DM_WORKFLOW_DATA_ANALYSIS, QMAP_FOLDER_PATH, XPCS_QMAP_FILENAME hdf_with_fullpath = $1 ##usually this is saved in the global variable HDF5_METADATA_FILE if ($# < 2) { qmapfile_with_fullpath = QMAP_FOLDER_PATH XPCS_QMAP_FILENAME } else { qmapfile_with_fullpath = $2 } if ($# < 3) { xpcs_group_name = "/xpcs" } else { xpcs_group_name = $3 } cmd = sprintf("source /home/dm/etc/dm.setup.sh; dm-start-processing-job --workflow-name=%s filePath:%s qmapFile:%s xpcsGroupName:%s",DM_WORKFLOW_DATA_ANALYSIS,hdf_with_fullpath,qmapfile_with_fullpath,xpcs_group_name); DM_WORKFLOW_DATA_ANALYSIS_CMD = cmd; printf("DM Workflow call is made for XPCS Analysis: %s,%s----%s\n",hdf_with_fullpath,qmapfile_with_fullpath,date()); unix(cmd); }' def DM_Workflow_ListJobs '{ printf("*****************************************\n"); unix("source /home/dm/etc/dm.setup.sh; dm-list-processing-jobs --display-keys=startTime,endTime,sgeJobName,status,stage,runTime,id | sort -r |head -n 10"); printf("*****************************************\n"); p date(); printf("*****************************************\n"); }' ```

prjemian commented 5 years ago

This SPEC macro also uses the batch_pars associative array (dict):

write_shortfile_log.mac

``` """ This is meant to be a replacement for the .short file. This will store the same info in a single row per data folder and hopefully will be more useful than the older version. (January 25, 2018) """ def write_shortfile_log '{ local tmp_qmapfilename global DATA_LOG_FILE tmp_qmapfilename = "null" if (SUBMIT_XPCS_JOB_FLAG) tmp_qmapfilename = XPCS_QMAP_FILENAME; ##open_shortfile_log #creates the file if it does not exist and write a line of labels open(DATA_LOG_FILE); fprintf(DATA_LOG_FILE,"batch_name: %s, start_time: %s, end_time: %s, samx: %5.3f, samy: %5.3f, samz: %5.3f, sampit: %5.3f, samchi: %5.3f, samth: %5.3f, ti3_x: %5.3f, ti3_z: %5.3f, ccdx: %5.3f, ccdz: %5.3f, current_pos: %d, Attenuation: %02d, Transmitted_flux: %4.3g, start_frame: %d, end_frame: %d, specfile: %s, scan_num: %d, current_begin: %5.2f, current_end: %5.2f, qmap_filename: %s\n", batch_name,rmquotes(batch_pars["start_time"]),rmquotes(batch_pars["end_time"]),A[samx],A[samy],A[samz],A[sampit], A[samchi], A[samth],A[ti3_x],A[ti3_z],A[ccdx],A[ccdz],(nextpos-1),atten_set, batch_pars["beam_i"],batch_pars["ndata0"], batch_pars["ndataend"], DATAFILE, SCAN_N,batch_pars["ring_i_beg"],batch_pars["ring_i_end"],tmp_qmapfilename); close(DATA_LOG_FILE); printf("Measurement conditions and Motor positions are written to a CSV file: %s\n",DATA_LOG_FILE); }' """ ##the below was used to label the columns in the file at the top, decided later to have the format as above of key: ##value, this is much easier to render to html for example global _LOG; # associative array with values of scan header entries global _L_COUNT; # counter of number of beam line info to record def open_shortfile_log '{ global DATA_LOG_FILE global _LOG global _L_COUNT; if (file_info(DATA_LOG_FILE,"-e")) { # Log file exists, no need to write labels } else { # Log file does not exist, write header withlabels open(DATA_LOG_FILE) _L_COUNT=0 _LOG[_L_COUNT++] = "batch_name" _LOG[_L_COUNT++] = "start_time" _LOG[_L_COUNT++] = "end_time" _LOG[_L_COUNT++] = "samx" _LOG[_L_COUNT++] = "samy" _LOG[_L_COUNT++] = "samz" _LOG[_L_COUNT++] = "ti3_x" _LOG[_L_COUNT++] = "ti3_z" _LOG[_L_COUNT++] = "ccdx" _LOG[_L_COUNT++] = "ccdz" _LOG[_L_COUNT++] = "atten_set" _LOG[_L_COUNT++] = "beam_i" _LOG[_L_COUNT++] = "ndata0" _LOG[_L_COUNT++] = "ndataend" # add more labels here as needed # Write label header for (i=0; i<_L_COUNT; i++) { fprintf(DATA_LOG_FILE, "%s,", _LOG[i]) ; # using comma delimited } fprintf(DATA_LOG_FILE, "\n"); close(DATA_LOG_FILE); } }' """ ```

prjemian commented 5 years ago

Too complicated. Take a simpler approach.

next plan: Ophyd Device to interface with the PVs, bluesky plan to write the HDF5 file.

prjemian commented 5 years ago

Create_DataExchange_HDF5_8idi.EigerHDF5.create_hdf5_file() will be called as part of a bluesky plan. Plan execution must wait for this step to complete. Seems we have two choices how to code this.

  1. a bluesky plan (with yield from *something()*)
  2. a non-plan (no yield from ...) -- in which case it will need to be waited on for completion.

First way sounds easier today. Experience with this situation suggests the first way. Most of the method will look the same. The only difference is that a plan must have at least one yield from ... statement (it needs to provide bluesky's RunEngine at least one bluesky Message). We can add yield from bps.null() to satisfy that requirement.

prjemian commented 5 years ago

Another Message is bps.checkpoint() which marks a re-wind point for the RunEngine. I prefer null at this point since it does nothing.

prjemian commented 5 years ago

Test the new HDF5 writer with this command:

RE(dm_pars.dm_workflow.create_hdf5_file("test.h5", as_bluesky_plan=True))

Produces a file with this HDF5 tree structure:

!punx tree test.h5

``` In [2]: !punx tree test.h5 !!! WARNING: this program is not ready for distribution. /home/beams10/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/test.h5 hdf_metadata_version:float64[1,1] = __array __array = [ [15.0] ] measurement instrument acquisition angle:int64[1,1] = __array __array = [ [-1] ] attenuation:float64[1,1] = __array __array = [ [1.0] ] beam_center_x:float64[1,1] = __array __array = [ [674.9085255354202] ] beam_center_y:float64[1,1] = __array __array = [ [157.15317133443162] ] beam_size_H:float64[1,1] = __array __array = [ [60.0] ] beam_size_V:float64[1,1] = __array __array = [ [15.000000000000002] ] ccdxspec:int64[1,1] = __array __array = [ [-1] ] ccdzspec:int64[1,1] = __array __array = [ [-1] ] compression:CHAR = ENABLED dark_begin:uint64[1,1] = __array __array = [ [0] ] dark_end:uint64[1,1] = __array __array = [ [0] ] data_begin:uint64[1,1] = __array __array = [ [1] ] data_end:uint64[1,1] = __array __array = [ [17500] ] data_folder:CHAR = A135_SiO2_100nm_A2_56p_S1_oscshear_5Hz_2500_att0_Lq1_001/ datafilename:CHAR = A135_SiO2_100nm_A2_56p_S1_oscshear_5Hz_2500_att0_Lq1_001_00001-17500.imm parent_folder:CHAR = weichen201908 root_folder:CHAR = /data/2019-2/weichen201908/ specfile:CHAR = weichen20190813 specscan_dark_number:uint64[1,1] = __array __array = [ [0] ] specscan_data_number:uint64[1,1] = __array __array = [ [680] ] stage_x:float64[1,1] = __array __array = [ [207.7] ] stage_z:float64[1,1] = __array __array = [ [30.05000000000001] ] stage_zero_x:float64[1,1] = __array __array = [ [209.0] ] stage_zero_z:float64[1,1] = __array __array = [ [31.5] ] xspec:int64[1,1] = __array __array = [ [-1] ] zspec:int64[1,1] = __array __array = [ [-1] ] detector adu_per_photon:float64[1,1] = __array __array = [ [1.0] ] bit_depth:uint32[1,1] = __array __array = [ [12] ] blemish_enabled:CHAR = ENABLED burst_enabled:CHAR = DISABLED distance:float64[1,1] = __array __array = [ [3930.0] ] efficiency:float64[1,1] = __array __array = [ [1.0] ] exposure_period:float64[1,1] = __array __array = [ [0.0311] ] exposure_time:float64[1,1] = __array __array = [ [0.03] ] flatfield_enabled:CHAR = ENABLED gain:uint32[1,1] = __array __array = [ [1] ] geometry:CHAR = TRANSMISSION kinetics_enabled:CHAR = DISABLED lld:int64[1,1] = __array __array = [ [0] ] manufacturer:CHAR = LAMBDA sigma:int64[1,1] = __array __array = [ [0] ] x_binning:uint32[1,1] = __array __array = [ [1] ] x_dimension:uint32[1,1] = __array __array = [ [1556] ] x_pixel_size:float64[1,1] = __array __array = [ [0.055] ] y_binning:uint32[1,1] = __array __array = [ [1] ] y_dimension:uint32[1,1] = __array __array = [ [516] ] y_pixel_size:float64[1,1] = __array __array = [ [0.055] ] burst first_usable_burst:uint32[1,1] = __array __array = [ [0] ] last_usable_burst:uint32[1,1] = __array __array = [ [0] ] number_of_bursts:uint32[1,1] = __array __array = [ [0] ] kinetics first_usable_window:uint32[1,1] = __array __array = [ [0] ] last_usable_window:uint32[1,1] = __array __array = [ [0] ] top:uint32[1,1] = __array __array = [ [0] ] window_size:uint32[1,1] = __array __array = [ [0] ] roi x1:uint32[1,1] = __array __array = [ [0] ] x2:uint32[1,1] = __array __array = [ [1555] ] y1:uint32[1,1] = __array __array = [ [0] ] y2:uint32[1,1] = __array __array = [ [515] ] source_begin beam_intensity_incident:float64[1,1] = __array __array = [ [1.0] ] beam_intensity_transmitted:float64[1,1] = __array __array = [ [573112333.6404556] ] current:float64[1,1] = __array __array = [ [102.21508800860398] ] datetime:CHAR = Tue Aug 20 23:05:39 2019 energy:float64[1,1] = __array __array = [ [10.925] ] source_end current:float64[1,1] = __array __array = [ [102.067107528604] ] datetime:CHAR = Tue Aug 20 23:14:43 2019 sample orientation:float64[1,3] = __array __array = [ [0.0, -4.440892098500626e-16, 0.0] ] temperature_A:float64[1,1] = __array __array = [ [-273.15] ] temperature_A_set:float64[1,1] = __array __array = [ [26.0] ] temperature_B:float64[1,1] = __array __array = [ [-273.15] ] temperature_B_set:float64[1,1] = __array __array = [ [26.0] ] thickness:float64[1,1] = __array __array = [ [1.0] ] translation:float64[1,3] = __array __array = [ [0.0, 0.0, 3.552713678800501e-15] ] translation_table:float64[1,3] = __array __array = [ [0.0, -0.0008900000000000001, 10.0] ] ```

This is the file:

snow% pwd
/home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup
snow% ls test.h5 
test.h5

test.zip

prjemian commented 5 years ago

In case you were wondering, took about 25 ms to write this file:

In [4]: t0 = time.time(); RE(dm_pars.dm_workflow.create_hdf5_file("test.h5", as_bluesky_plan=True)); print(time.time() - t0)                                          
0.025174856185913086
prjemian commented 5 years ago

Pull the dm_workflow attribute out of the dm_pars structure because it is simpler to see and resolve problems. Too much optimization.

sureshnaps commented 5 years ago

This is great news. In response to an earlier message, step 4 need not complete before next scan starts, as long as the params are saved to be written later, it is ok.

sureshnaps commented 5 years ago

will test test.h5 in the afternoon today

sureshnaps commented 5 years ago

i used your test.h5 to replace the same metadata .hdf created before the beam went down. This was my last dataset this cycle as that is what is in the registers. I could use your file to do everything. /net/wolf/data/xpcs8/2019-2/weichen201908/A135_SiO2_100nm_A2_56p_S1_oscshear_5Hz_2500_att0_Lq1_001

snow% ls -rtla total 6164300 drwxr-sr-x 563 8idiuser clusers 77824 Aug 20 23:14 .. -rw-rw-r-- 1 8idiuser clusers 41992 Aug 20 23:15 A135_SiO2_100nm_A2_56p_S1_oscshear_5Hz_2500_att0_Lq1_001_0001-17500.hdforig -rw-rw-r-- 1 8idiuser clusers 1871 Aug 20 23:15 A135_SiO2_100nm_A2_56p_S1_oscshear_5Hz_2500_att0_Lq1_001_0001-17500.batchinfo -rw-rw-r-- 1 8idiuser clusers 6312042946 Aug 20 23:15 A135_SiO2_100nm_A2_56p_S1_oscshear_5Hz_2500_att0_Lq1_001_00001-17500.imm -rw-r--r-- 1 8idiuser clusers 41896 Aug 26 11:01 A135_SiO2_100nm_A2_56p_S1_oscshear_5Hz_2500_att0_Lq1_001_0001-17500.hdf drwxr-sr-x 2 8idiuser clusers 4096 Aug 26 11:01 .

.hdf is your test.h5 and .hdforig is the original one. they both worked. did h5diff: (will check when on site what these mean)

snow% h5diff A135_SiO2_100nm_A2_56p_S1_oscshear_5Hz_2500_att0_Lq1_001_0001-17500.hdf A135_SiO2_100nm_A2_56p_S1_oscshear_5Hz_2500_att0_Lq1_001_0001-17500.hdforig dataset: </measurement/instrument/acquisition/beam_center_x> and </measurement/instrument/acquisition/beam_center_x> 1 differences found dataset: </measurement/instrument/acquisition/beam_center_y> and </measurement/instrument/acquisition/beam_center_y> 1 differences found dataset: </measurement/instrument/acquisition/beam_size_V> and </measurement/instrument/acquisition/beam_size_V> 1 differences found dataset: </measurement/instrument/acquisition/stage_x> and </measurement/instrument/acquisition/stage_x> 1 differences found dataset: </measurement/instrument/acquisition/stage_z> and </measurement/instrument/acquisition/stage_z> 1 differences found dataset: </measurement/instrument/detector/exposure_period> and </measurement/instrument/detector/exposure_period> 1 differences found dataset: </measurement/instrument/detector/exposure_time> and </measurement/instrument/detector/exposure_time> 1 differences found dataset: </measurement/instrument/detector/x_pixel_size> and </measurement/instrument/detector/x_pixel_size> 1 differences found dataset: </measurement/instrument/detector/y_pixel_size> and </measurement/instrument/detector/y_pixel_size> 1 differences found dataset: </measurement/instrument/source_begin/beam_intensity_transmitted> and </measurement/instrument/source_begin/beam_intensity_transmitted> 1 differences found dataset: </measurement/instrument/source_begin/current> and </measurement/instrument/source_begin/current> 1 differences found dataset: </measurement/instrument/source_begin/energy> and </measurement/instrument/source_begin/energy> 1 differences found dataset: </measurement/instrument/source_end/current> and </measurement/instrument/source_end/current> 1 differences found dataset: </measurement/sample/orientation> and </measurement/sample/orientation> 2 differences found dataset: </measurement/sample/temperature_A> and </measurement/sample/temperature_A> 1 differences found dataset: </measurement/sample/temperature_A_set> and </measurement/sample/temperature_A_set> 1 differences found dataset: </measurement/sample/temperature_B> and </measurement/sample/temperature_B> 1 differences found dataset: </measurement/sample/temperature_B_set> and </measurement/sample/temperature_B_set> 1 differences found dataset: </measurement/sample/translation_table> and </measurement/sample/translation_table> 3 differences found

Some objects are not comparable

Use -c for a list of objects. snow%

prjemian commented 5 years ago

I hope these differences are in the values and not the structure.

sureshnaps commented 5 years ago

After doing a field by field comparison, the following were the only discrepancies.

% /measurement/instrument/acquisition
ccdxspec - should be 64-bit float
ccdzspec - should be 64-bit float
xspec  - should be 64-bit float
zspec - should be 64-bit float
% /measurement/instrument/detector
lld - should be 64-bit float (this is a lower level discriminator and could be integer and can be argued)
sigma - should be 64-bit float

it seems the orientation is swapped but is hard to tell as they are all close to zero, very small 1e-16 or so. Something to check

temp = self.f.create_dataset("/measurement/sample/orientation", (1,3)) ##pitch,roll.yaw
            temp[(0,0)] = epics.caget('8idi:m52.RBV')
            temp[(0,1)] = epics.caget('8idi:m53.RBV')
            temp[(0,2)] = epics.caget('8idi:m51.RBV')