offset_calculation and offset_calibration do not work together

[dneise]

I just tried to execute offset_calculation.py in order to get the constants, and then offset_calibration.py in order to apply the calibration to a folder full of raw files. I did actually only want one file to be calibrated, but I did not know how, since I may only specify a <raw_datafile_directory> and not e.g. _raw_datafilepath.

What I did was:

mkdir caliboffsets
mkdir calibrated_output
ipython offset_calculation.py ../IP192.168.1.1/Pedestal/ caliboffsets/
ipython offset_calibration.py ../IP192.168.1.1/Gain/ caliboffsets/ calibrated_output/

I got this error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/home/dneise/LST/dbtb_new/offset_calibration.py in <module>()
    144     scan_datafile_amount(raw_datafile_directory)
    145     is_calibration_constants_existent(calibration_constants_directory)
--> 146     store_calibrated_data(output_directory, apply_offset_calibration(raw_datafile_directory, calibration_constants_directory))

/home/dneise/LST/dbtb_new/offset_calibration.py in apply_offset_calibration(raw_datafile_directory, calibration_constants_directory)
     65                         const_roi = calibration_constants[calib_const_array_pos][stop_cell[stop_cell_array_pos]:stop_cell[stop_cell_array_pos]+event.roi]
     66 
---> 67                         calibrated_data.append(np.subtract(event.data[gaintype][pixelindex], const_roi))
     68                         event_header.append([filename[len(raw_datafile_directory):], gaintype, pixelindex, stop_cell[stop_cell_array_pos]])
     69 

ValueError: operands could not be broadcast together with shapes (1024,) (307,) 

[dneise]

How is one supposed to use these two programs together? Can you post a working example please?

[dneise]

I get this error on the current master and on the current mario_offsetcalculation branches.

[MHoerbe]

Hey, sorry for being late. Next to a minor fix it seems like all the ../Gain data cannot be treated by my routine. I assume the data structure if all the regarding data to cause the exceptions.

You used the offset_calculation entirely correct. offset_calibration works perfectly fine on /Jitter data, for instance (your usage of the latter was correct, too.)

Currently investigating on the bugs.

[MHoerbe]

As expected, it seems like the data structure of all directories is not homogeneous. It seems like data in, e.g. /CoherentNoise and /Gain is saved as

event = ([event1...] , [event2...])

whereas /Jitter ans /Pedestal only contain events of the structure

event = ([event...])

I am not sure about the interpretation of the former structure. Maybe we should ask the group for their opinion.

[MHoerbe]

Update:

It seems like /Gain data consists of events of the kind event = ([event1...] , [event2...]) event = ([event1...] , [event2...], [event3...])

I could recover the calibration method's functionality by calling the 0th/1st/2nd entry but that leaves 2/3 of the raw data untreated. Why are there tuples of events per 'event'?!

[MHoerbe]

Program should work for more kinds of data, have a try ;-)

[dneise]

This is a non issue, since the code it refeers to was largely rewritten