How do I generate the calibration constants for one run?

[benkrikler]

What are the modules files I should use to generate the current set of calibration constants (pedestal, noise and TDiffs) for a single run? Could we add these to the configurations database directory?

I know there's a script to convert the CSV into the sqlite table, but is it ready and can I work around this by hand at this point?

[AndrewEdmonds11]

What are the modules files I should use to generate the current set of calibration constants (pedestal, noise and TDiffs) for a single run?

I think it has to be done in two runs since the TDiff calculation needs to know the pedestal.

John and I are currently merging stuff in. Can this wait until after the meeting? We should then be in a position to not need to do anything by hand and do a full Al50(b) run.

[benkrikler]

It can wait, I just wanted to look at the TMEs with properly calibrated data. It would be good to include them in the run, but I wanted to check things properly first.

[benkrikler]

Or if you had a reasonable database I could use instead that would work for me...

[AndrewEdmonds11]

I have nothing, I'm afraid. I'm not entirely sure what state my code is in at the moment (in terms of whether I have the renamings in) so was just waiting for everything to be merged.

We should definitely have the calibration config files stored in the repository like our production one.

And I think we should add the TMEs for this run. As long as it doesn't crash, we can check whether it makes sense afterwards.

[benkrikler]

Ok then, I'll leave it for now.

[benkrikler]

So is there an update on this? I want to test some things but we now depend on the calibration DB a lot. What are the modules and / or options I need?

[jrquirk]

I just finished rerunning the peds/noises. There's a problem with run scripts that I'm working on. Anyway, it's not done yet. If you want to test things just use the MaxBin timing method and the MaxBin timing offset by grabbing the offset via TSetupData instead of the SetupNavigator.

[benkrikler]

I'm trying to test a new generator to look at the pile-up a bit more, so pedestal values would be very helpful. I'll hard code things for now then.

[jrquirk]

I've generated the stuff at ~quirk_j/AlcapDAQ/analyzer/rootana/calibration.db for all the Al50b dataset. The thing is the timing offsets can only be loaded for the FirstComplete generator with a certain value for the config string, so I've pushed the calibration config files with the name rootana/configurations/*_calib.cfg. So if you set the constant_fraction argument for the Ge-S to, say, 50%, you'll get a map::at() error because that timing offset doesn't exist. It only exists for Ge-S at 60%.

The steps I took for a run were

$ ./rootana -i tree.root -o out.root -m configurations/pedestalsandnoise_calib.cfg -c
...
$ scripts/merge_calibration_database.py ./calibration.db calib.run?????.PedestalAndNoise.csv
$ ./rootana -i tree.root -o out.root -m configurations/timeoffset_calib.cfg -c
...
$ scripts/merge_calibration_database.py ./calibration.db calib.run?????.CoarseTimeOffset.csv

(Edit: Things in the code block above have been reordered.)

I did spot checks, but honestly it's hard to tell what's reasonable and what isn't and non of us have the time to go through this line by line. So I also uploaded a script to make a quick gnuplot at rootana/scripts/make_trend_plots.sh which produces two *_trend.png pics.

All that's left is to test the job submission scripts.

[jrquirk]

Also it's important to note the python merging script uses the name of the calib.run..csv file as table names and the such, instead of storing that info in the file itself.

[benkrikler]

I'm trying to produce the calibration database over 1 run in the SiR2 active target dataset, but don't seem to be able to get things running. I'm trying the following command:

./rootana -c -i ~/data/tree/tree02140.root -o output.root -m configurations/timeoffset_calib.cfg 

but its still looking into maps in GetPedestal. I thought this had been disabled if I run with -c but it doesn't seem to be the case, given that things are crashing (the pedestal map is empty so the lookup throws an exception).

Am I not using things correctly, was this behaviour removed or has it been broken ( / did I break it)?

[benkrikler]

I'm not sure what's changed then, but GetPedestal and GetNoise ignore the -c option (don't check for a calib run or not), so they throw exceptions. Was this different before or was the check for -c implemented in the callers of the Get methods?

Also, what's a reasonable value to return if we are in a calib run (with -c)? Just 0?

[benkrikler]

Could this be because we need to have done the pedestal and noise calibration in order to do the timeoffset calibration which I hadn't done?

[jrquirk]

Could this be because we need to have done the pedestal and noise calibration in order to do the timeoffset calibration which I hadn't done?

Yep. This was intended behavior, though whether or not it's the best choice is another question. The -c flag serves two purposes.

1. If the SetNav can't find a table in the DB, it normally throws an exception immediately. But here it would pass the buck because it expects to be doing some sort of calibration.
2. If a SetNav setter is called for a DB value, normally nothing happens but a warning message. Here the setters are enabled.

The CF timing needs the pedestal value, as you suggest.

[benkrikler]

Yep. This was intended behavior, though whether or not it's the best choice is another question.

Ok. I was folllowing this comment from a while back where you put:

The steps I took for a run were

$ ./rootana -i tree.root -o out.root -m configurations/timeoffset_calib.cfg -c
...
$ scripts/merge_calibration_database.py ./calibration.db calib.run?????.PedestalAndNoise.csv
$ ./rootana -i tree.root -o out.root -m configurations/pedestalsandnoise_calib.cfg -c
...
$ scripts/merge_calibration_database.py ./calibration.db calib.run?????.CoarseTimeOffset.csv

You say you run rootana with the timeoffset_calib.cfg config file first and then ran again for the pedesals and noise. I hadn't spotted that merging into the DB was in reverse order else I'd have realised one of these was wrong.

So how should we handle the SetupNavigator Getters if they're called with -c ? Should I throw something or return 0?

[jrquirk]

So that's very much a mistake on my part, I'll fix it.

Right now some getters should be callable by some calibrations. It should throw if the value hasn't been loaded from the DB because a mistake has been made (a module had an unmet expectation), but shouldn't generally throw just because a getter and the -c option are used. And as of now now default (like 0) should be returned.

This is all within the context of the calib modules we have now.

[benkrikler]

I've started adding some more exceptions. At the moment the NoCalibDB exception was thrown when the file didn't exist AND when the tables were missing. So I've now split it up so that this is only thrown for a missing file and a missing table throws MissingTable.

Secondly, I'm going to make sure the Read...Values methods checks whether it's read anything for this run from the DB. If no values at all are read, I'm going to throw an UncalibratedRun exception if this is not a -c run and just print a warning otherwise. This should stop things getting as far as the getters so be a bit clearer.

Also, the behaviour at the moment with -c is to check for the table. If it exists read the values in, if it doesn't then print a warning but otherwise continue. That means that if we want to recalibrate data and the DB already contains values, we'll be calibrating things on-top of the previous values. Is this what we want to happen? If we're doing a calibration run, it seems we should ignore existing calibration values, no? The difficulty is that we then need to be able to specify which parts of the calibration to ignore since we need calibrated pedestals and noises to be able to get the TDiff values which.

Does the merge calibration script check for previously existing values? What does it do in this case?