search

cms-gem-daq-project / gem-light-dqm

GEM light DQM code
0 stars 17 forks source link

[feature] Automatically convert xdaq raw data into a gemTreeStructure format #39

Open BenjaminRS opened 5 years ago

BenjaminRS commented 5 years ago

Brief summary of issue

Need to create a daemon which runs a script that takes the raw format from xdaq (*.dat) for all shelves slots and links, unpacks the data and for each ohkey() converts it into a gemTreeStructure format.

Types of issue

Expected Behavior

The idea is for a new executable, gemTreeTranslator, to use the GEMUnpacker.cc in a similar fashion to gemTreeReader and create a set of TFiles (one per AMC/OH pair).

It will fill the following branches: the following branches:

Brian mentions that "The rest of the parameters could be filled with dummy values or obtained from a DB query of the conditions/configuration DB at runtime"

Current Behavior

Does not currently exist.

Context

Should be done within approximately 1 week.

jsturdy commented 5 years ago

This really should not belong in cmsgemos, rather ldqm (which already can run as a daemon) or gem-light-dqm, however I will reserve final judgement until this issue is more fully fleshed out

BenjaminRS commented 5 years ago

not sure if everyone gets automatically pinged - but I added more info the issue description

BenjaminRS commented 5 years ago

Hi @bdorney, in the meeting today you were suggesting some techniques to take the meta data required for the gemTreeTranslator. Can you give some more details here please? @ram1123 is looking into using a python script to take some metadata but I am not sure if it is the correct ones, from chatting to you this morning.

jsturdy commented 5 years ago
ram1123 commented 5 years ago

hi @jsturdy and @bdorney

Currently we are filling the following to the output tree. Please correct us if there is anything wrong.

Questions:

  1. what is the difference between amc13.m_amcs.gebd.vfatd.fSlotNumber and amc13.m_amcs.gebd.vfatd.fPos?

@korostysh, @BenjaminRS

jsturdy commented 5 years ago

Issue moved to correct repo

jsturdy commented 5 years ago

hi @jsturdy and @bdorney

Currently we are filling the following to the output tree. Please correct us if there is anything wrong.

OK, there are some corrections below, but there seems to be a fundamental misunderstanding about this translation. Fundamentally, to create this new output TTree, you will be required to do an analysis of the input TTree. This is because in the new way of data-taking, the tree is event-based, while in the old way of data-taking the tree is point-based, and the goal of this translator is to take the event-based tree and translate it into the point-based tree structure.

  • link : amc13.m_amcs.gebd.m_InputID

Yes

  • Nev : amc13.m_amcs.gebd.vfatd.fEC

No, every entry in the TTree is the aggregate of multiple "events" from the input file. Nev is the sum total of the number of events for a given VFAT/Channel/scan point

  • shelf : amc13.m_AMC_no

No,

  • slot : amc13.m_amcs.m_AMCnum

Maybe, @mexanick?

  • vfatCH : total number of non-zero bits obtained from amc13.m_amcs.gebd.vfatd.lsData and amc13.m_amcs.gebd.vfatd.msData.

No, vfatCH is the index of a given bit in the lsData and msData; it is literally the index of a given VFAT channel

  • What we will do when there are two differents hits on two different VFAT say VFAT0 and VFAT23?

This is expected to be the case

  • vfatID : amc13.m_amcs.gebd.vfatd.fSlotNumber

No, this should be the ChipID but this is not stored in the data itself. It will have to come from the (eventual) meta-data

  • vfatN : amc13.m_amcs.gebd.vfatd.fPos

Maybe, I don't know what the distinction between fPos and fSlotNumber is...

  • latency : amc13.m_amcs.m_Param1

Possibly, but probably will be redefined, is there simply a field m_Params?

  • vcal : amc13.m_amcs.gebd.vfatd.fisBlockGood

No, this will come from the same as latency in the case of an s-curve scan

  • vthr : amc13.m_amcs.m_Param2

Not likely to be stored in the data format, must come from meta-data

  • vth1 : amc13.m_amcs.m_Param3

Not likely to be stored in the data format, must come from meta-data

  • scan type: amc13.m_amcs.m_Rtype

Maybe, @mexanick?

  • Nhits: Number of differet VFATs having non-zero channels having hits??

Depends on the scan type, but generally as below, out of all the events at the scan point of interest:

Questions:

  1. what is the difference between amc13.m_amcs.gebd.vfatd.fSlotNumber and amc13.m_amcs.gebd.vfatd.fPos?

You should look at the code, but I would use fPos for the VFAT position on the GEB, @mexanick?

@korostysh, @BenjaminRS

bdorney commented 5 years ago

In addition to the responses from @jsturdy where pretty conclusive but to be more specific:

  • latency : amc13.m_amcs.m_Param1
  • vcal : amc13.m_amcs.gebd.vfatd.fisBlockGood
  • vthr : amc13.m_amcs.m_Param2
  • vth1 : amc13.m_amcs.m_Param3
  • scan type: amc13.m_amcs.m_Rtype

The meaning of the RunParams words in the AMC13 header will depend on the Rtype following the discussion @jsturdy and myself had here:

https://github.com/cms-gem-daq-project/cmsgemos/issues/230

As you can see these values of latency, CFG_CAL_DAC, and CFG_THR_ARM_DAC will only be stored in the RunParams if the RunType states that they will. Otherwise they will, just like all other register info, need to be fetched from the (eventual) meta-data.

mexanick commented 5 years ago

All assumptions made by @jsturdy are correct.

amc13.m_amcs.gebd.vfatd.fSlotNumber and amc13.m_amcs.gebd.vfatd.fPos

There's no difference in meaning, just different field names used between V2 and V3 VFATs and I'm sticking to VFAT manual on this.

korostysh commented 5 years ago

And for the final tree, the one VFAT should be an Event or each trigger channel should be a separate event? @bdorney

bdorney commented 5 years ago

And for the final tree, the one VFAT should be an Event or each trigger channel should be a separate event? @bdorney

I don't understand the question; can you elaborate?

bdorney commented 5 years ago

See updated discussion on the meaning of the RUN_PARAMS words:

https://github.com/cms-gem-daq-project/cmsgemos/issues/230#issuecomment-518642741

ram1123 commented 5 years ago

Hi @bdorney , @jsturdy , @mexanick

I am analyzing the root file

/data/bigdisk/GEM-Data-Taking/GE11_QC8/Cosmics/run000033_Cosmic_CERNQC8_2019-02-12.raw.root

I was just checking the two variables SlotNumber and Pos.

IMG_20190807_133848 IMG_20190807_133855 IMG_20190807_133908

What I expected is that for each GEB, Pos should vary from 0 to 23. But, it seems random. And I could not find a pattern. Is this expected? or it's the issue of the previous step that gives the *.raw.root?

bdorney commented 5 years ago

What's the goal of using this particular file and not say, the newer ones I've pointed you and @BenjaminRS to?

ram1123 commented 5 years ago

What's the goal of using this particular file and not say, the newer ones I've pointed you and @BenjaminRS to?

Hi @bdorney ,

I tried to look at the new root file (run000214_allLumi_index000000.raw.root ) that you provided.

By looking at this I observe following things:

bdorney commented 5 years ago

But did you try with run000217 that I circulated earlier this week?

jsturdy commented 5 years ago

@ram1123, you should definitely use only the latest run, as until that run, there were various issues with the data taking that could have had some impact on the underlying data (though I wouldn't expect anything so drastic as what you report below)

What's the goal of using this particular file and not say, the newer ones I've pointed you and @BenjaminRS to?

Hi @bdorney ,

I tried to look at the new root file (run000214_allLumi_index000000.raw.root ) that you provided.

By looking at this I observe following things:

* Some events AMC size is zero.

This sounds like an issue unpacking, which may be the case as per the note above.

* For some events where is finds AMC and GEB it shows numbere of VFAT more than 1000 and sometime only few. While I expect it should always have size of 24.

This sounds like an unpacking error, also, for the data that @bdorney is providing, I think you should only expect 17 VFATs. Additionally, in the raw data, you don't have any expectation that the VFATs will appear in order, but furthermore, at the level you're analyzing, it's already been unpacked an put into some tree structure

(edited)

ram1123 commented 5 years ago

Thanks @bdorney and @jsturdy .

Now I tried to see the behaviour of two runs: run000217_allLumi_index000000.raw.root and run000205_Cosmic_CERNQC8_2019-07-16_chunk_1.raw.root

For run000217_allLumi_index000000.raw.root, I observe following:

For run000205_Cosmic_CERNQC8_2019-07-16_chunk_1.raw.root, I observe following:

[1]

   GEB:0: local_link = 8
   Found VFATs having size 17
   VFAT:0: vfat_Pos = 2
   VFAT:0: lsData = 2
   VFAT:0: msData = 0
   VFAT:1: vfat_Pos = 3
   VFAT:1: lsData = 2
   VFAT:1: msData = 0
   VFAT:2: vfat_Pos = 4
   VFAT:2: lsData = 2
   VFAT:2: msData = 0
   VFAT:3: vfat_Pos = 5
   VFAT:3: lsData = 2
   VFAT:3: msData = 0
   VFAT:4: vfat_Pos = 6
   VFAT:4: lsData = 2
   VFAT:4: msData = 0
   VFAT:5: vfat_Pos = 7
   VFAT:5: lsData = 2
   VFAT:5: msData = 0
   VFAT:8: vfat_Pos = 11
   VFAT:8: lsData = 2
   VFAT:8: msData = 0
   VFAT:9: vfat_Pos = 12
   VFAT:9: lsData = 2
   VFAT:9: msData = 0

[2]

  236 VFAT:1: local_Pos = 17
  237 VFAT:1: lsData = 12                                                                                              
  238 VFAT:1: msData = 0
  239 VFAT:7: local_Pos = 23
  240 VFAT:7: lsData = 0
  241 VFAT:7: msData = 4611686018427387904
ram1123 commented 5 years ago

Hi @bdorney and @jsturdy ,

I forgot to mention one more thing in the Run000217 the order in which VFAT appears varies from event to event like below[1]. Is this expected?

[1]

########################################

    Event loop

########################################
Get a vector of AMC13 having size 1
Found AMC having size 1
Found GEB having size 1
-----
GEB:0: local_link = 8
Found VFATs having size 17
VFAT:0: local_Pos = 10
VFAT:1: local_Pos = 11
VFAT:2: local_Pos = 12
VFAT:3: local_Pos = 14
VFAT:4: local_Pos = 15
VFAT:5: local_Pos = 18
VFAT:6: local_Pos = 19
VFAT:7: local_Pos = 22
VFAT:8: local_Pos = 0
VFAT:9: local_Pos = 1
VFAT:10: local_Pos = 2
VFAT:11: local_Pos = 3
VFAT:12: local_Pos = 4
VFAT:13: local_Pos = 5
VFAT:14: local_Pos = 6
VFAT:15: local_Pos = 7
VFAT:16: local_Pos = 9
########################################

    Event loop

########################################
Get a vector of AMC13 having size 1
Found AMC having size 1
Found GEB having size 1
-----
GEB:0: local_link = 8
Found VFATs having size 17
VFAT:0: local_Pos = 0
VFAT:1: local_Pos = 1
VFAT:2: local_Pos = 2
VFAT:3: local_Pos = 3
VFAT:4: local_Pos = 4
VFAT:5: local_Pos = 5
VFAT:6: local_Pos = 6
VFAT:7: local_Pos = 7
VFAT:8: local_Pos = 9
VFAT:9: local_Pos = 10
VFAT:10: local_Pos = 11
VFAT:11: local_Pos = 12
VFAT:12: local_Pos = 14
VFAT:13: local_Pos = 15
VFAT:14: local_Pos = 18
VFAT:15: local_Pos = 19
VFAT:16: local_Pos = 22
bdorney commented 5 years ago

  • Whenever there is hit, it is always shown bylsData. I mean msData was always zero. As shown below [1].

    • My expectation: sometime lsData should be non-zero and sometime msData or both.
  • For many events many VFAT shows that there is hit on exactly same strip for each of them. See [1].

Check the associate elog for this run: http://cmsonline.cern.ch/cms-elog/1088166

Also consider what RunType is set too.

I forgot to mention one more thing in the Run000217 the order in which VFAT appears varies from event to event like below[1]. Is this expected?

You'll need to provide more debugging info. It's not clear how you obtained this output. e.g. what was your input, what are you doing, what code is being used, etc...

jsturdy commented 5 years ago

I forgot to mention one more thing in the Run000217 the order in which VFAT appears varies from event to event like below[1]. Is this expected?

Additionally, in the raw data, you don't have any expectation that the VFATs will appear in order, but furthermore, at the level you're analyzing, it's already been unpacked an put into some tree structure

I edited my earlier message as I had left out a key word, but at the raw level, there is no expectation of VFATs coming "in order" (nor GEBs, for that matter, though I do believe that the AMC payloads will be in order). Since you're analyzing the raw.root tree, @mexanick should be able to confirm (and I'd like to know what your non-local indices are, since you should only use what you print above as "local" to refer to the HW positions), I don't know, but I suspect you're looping over entries in the tree, and these are filled as they are unpacked, so they will also not come with any expectation of being "in order"

mexanick commented 5 years ago

The order of VFAT packets is arbitrary, there's no guarantee that it will be preserved from one event to another. Moreover, in case of zero-suppression there's no fixed number of the VFAT packets (i.e. if a chip didn't have a hit, it won't appear in the data stream). So the behavior you observe is the correct one.

ram1123 commented 5 years ago

I edited my earlier message as I had left out a key word, but at the raw level, there is no expectation of VFATs coming "in order" (nor GEBs, for that matter, though I do believe that the AMC payloads will be in order). Since you're analyzing the raw.root tree, @mexanick should be able to confirm (and I'd like to know what your non-local indices are, since you should only use what you print above as "local" to refer to the HW positions), I don't know, but I suspect you're looping over entries in the tree, and these are filled as they are unpacked, so they will also not come with any expectation of being "in order"

Yes. Here I am looping over entries in the tree. And "local" is just my convention for just checking the value of the VFAT position.

ram1123 commented 5 years ago

I forgot to mention one more thing in the Run000217 the order in which VFAT appears varies from event to event like below[1]. Is this expected?

You'll need to provide more debugging info. It's not clear how you obtained this output. e.g. what was your input, what are you doing, what code is being used, etc...

From @jsturdy and @mexanick, response it is now clear that this behavirour is expected.

BenjaminRS commented 5 years ago

Hi all - I updated the code with a commit here. However there are still two points I need to work on: 1) it is not pre-fetching how many AMC13/AMC/GEBs there are; 2) It is not properly counting the NeV (i.e. recording for each VFAT/Channel/scan point) --> this is the point of switching from event-based tree to the point-based tree. I was looking into recording the information at the end of the class, but then was trying to think of a better way to does this with the root structure.