digi2raw test

[lietava]

Please take latest ctpdev and test digi2raw:

• produce digits as before -run digi2raw

[mbombara]

I updated the installation with the latest ctpdev. I run these commands:

o2-sim -n 10 -m FT0 FV0 ITS CTP -g pythia8 -j10
o2-sim-digitizer-workflow -b
o2-ctp-digi2raw

running o2-ctp-digi2raw produces file CTPraw.cfg with this content:

#[defaults]
#dataOrigin = CTP
#dataDescription = RAWDATA
#readoutCard = CRU

Nothing else. Was this desired output? I have not found any .raw file.

[lietava]

Marek, thanks. It should produce raw data somewhere. Please, check. There should be also log somewhere. Have s look in digi2raw.cxx. @shahor02 any comments ? Cheers, Roman.

[mbombara]

Even if I set the output directory like:

mkdir ctpoutput
o2-ctp-digi2raw -o ctpoutput

there is only CTPraw.cfg in ctpoutput. I can see in digi2raw.cxx that output file should be ctp.raw. In the code, the function processDigits reads ctpdigits.root but I am not sure where the writing raw data happens.

[shahor02]

Hi, I also run ctpdev and see problems, left in-line comments in the PR.

[shahor02]

Also, could you update you do

git checkout dev
git pull
git checkout ctpdev
git rebase dev
git push -f cptdev

otherwise it takes ages to recompile your branch.

[lietava]

@mbombara there is new version which produces nonempty file. Please, have a look.

[lietava]

o2-sim -n 10 -m FT0 FV0 ITS CTP -g pythia8 -j10 - fails on my computer now: [INFO] Process 20408 EXITED WITH CODE 0 SIGNALED 1 SIGNAL 6 o2-sim -n 10 -m FT0 FV0 ITS CTP -j10 runs ok @shahor02 Any hint ?

[mbombara]

I cannot pull:

mbombara@MacBook-Pro-3 O2 % git pull origin ctpdev
remote: Enumerating objects: 1076, done.
remote: Counting objects: 100% (719/719), done.
remote: Compressing objects: 100% (70/70), done.
remote: Total 1076 (delta 655), reused 680 (delta 645), pack-reused 357
Receiving objects: 100% (1076/1076), 510.71 KiB | 2.81 MiB/s, done.
Resolving deltas: 100% (755/755), completed with 329 local objects.
From https://github.com/lietava/AliceO2
 * branch                  ctpdev     -> FETCH_HEAD
 + 399afc700c...11a701b295 ctpdev     -> origin/ctpdev  (forced update)
CONFLICT (add/add): Merge conflict in Detectors/CTP/simulation/src/digi2raw.cxx
Auto-merging Detectors/CTP/simulation/src/digi2raw.cxx
CONFLICT (add/add): Merge conflict in Detectors/CTP/simulation/src/Digits2Raw.cxx
Auto-merging Detectors/CTP/simulation/src/Digits2Raw.cxx
Auto-merging Detectors/CTP/simulation/src/Digitizer.cxx
CONFLICT (add/add): Merge conflict in Detectors/CTP/simulation/include/CTPSimulation/Digits2Raw.h
Auto-merging Detectors/CTP/simulation/include/CTPSimulation/Digits2Raw.h
CONFLICT (add/add): Merge conflict in Detectors/CTP/macro/CreateCTPConfig.C
Auto-merging Detectors/CTP/macro/CreateCTPConfig.C
Auto-merging DataFormats/Detectors/CTP/src/Configuration.cxx
CONFLICT (content): Merge conflict in DataFormats/Detectors/CTP/src/Configuration.cxx
Auto-merging DataFormats/Detectors/CTP/include/DataFormatsCTP/Digits.h
Auto-merging DataFormats/Detectors/CTP/include/DataFormatsCTP/Configuration.h
CONFLICT (content): Merge conflict in DataFormats/Detectors/CTP/include/DataFormatsCTP/Configuration.h
Removing Analysis/Tasks/PWGHF/qaTaskSimple.cxx
Removing Analysis/Tasks/PWGDQ/tableMakerMuon_pp.cxx
Automatic merge failed; fix conflicts and then commit the result.
mbombara@MacBook-Pro-3 O2 % git remote -v
origin  https://github.com/lietava/AliceO2.git (fetch)
origin  https://github.com/lietava/AliceO2.git (push)
upstream    https://github.com/AliceO2Group/AliceO2 (fetch)
upstream    https://github.com/AliceO2Group/AliceO2 (push)
mbombara@MacBook-Pro-3 O2 % git pull origin ctpdev
error: Pulling is not possible because you have unmerged files.
hint: Fix them up in the work tree, and then use 'git add/rm <file>'
hint: as appropriate to mark resolution and make a commit.
fatal: Exiting because of an unresolved conflict.
mbombara@MacBook-Pro-3 O2 %

Is it on my side?

[lietava]

try git pull --rebase or just remove your local repository and clone fresh.

[lietava]

git reset --hard origin/

[lietava]

git fetch --all (before reset)

[lietava]

sorry, so: git fetch --all git reset --hard origin/ (also rm and clone should work)

[shahor02]

pythia8 was deprecated, use pythia8pp.

For the test: once you have produced the raw data, please run

o2-raw-file-check --input-conf <your_cfg_file>

there should be no any error message.

[lietava]

[O2/latest-ctpdev-o2] ~/o2tests2 $> o2-raw-file-check --input-conf CTPraw.cfg [INFO] RawDataHeader v6 is assumed [INFO] apply check for Mismatch between flagged and calculated new TF start [INFO] apply check for No SOX found on 1st page [INFO] ignore check for TF does not start by new superpage [INFO] apply check for Wrong HBF orbit increment [INFO] apply check for Number of TFs is less than expected [INFO] apply check for Number of HBFs per TF not as expected [INFO] apply check for Data does not start with TF/HBF [INFO] apply check for New HBF starts w/o closing old one [INFO] ignore check for RDH.stop set of 1st HBF page [INFO] apply check for Wrong RDH.pageCnt increment [INFO] apply check for Wrong RDH.packetCounter increment [INFO] perform check for /Wrong RDH.packetCounter increment/ [INFO] perform check for /Wrong RDH.pageCnt increment/ [INFO] perform check for /New HBF starts w/o closing old one/ [INFO] perform check for /Data does not start with TF/HBF/ [INFO] perform check for /Number of HBFs per TF not as expected/ [INFO] perform check for /Number of TFs is less than expected/ [INFO] perform check for /Wrong HBF orbit increment/ [INFO] perform check for /No SOX found on 1st page/ [INFO] perform check for /Mismatch between flagged and calculated new TF start/ [INFO] File 0 : 22126720 bytes scanned, 3586 RDH read for 2 links from /home/rl/o2tests2/ctp.raw [INFO] Summary of preprocessing: [INFO] Lnk0 | Link CTP/RAWDATA/0x00000000 RO: Trig FEE:0x0000 CRU: 0 Lnk: 0 EP:0 RDHv6 Src:NIL | SPages: 1 Pages: 1793 TFs: 1 with 256 HBF in 256 blocks (0 err) [INFO] Lnk1 | Link CTP/RAWDATA/0x00000001 RO: Trig FEE:0x0001 CRU: 0 Lnk: 1 EP:0 RDHv6 Src:NIL | SPages: 1 Pages: 1793 TFs: 1 with 256 HBF in 256 blocks (0 err) [INFO] First orbit: 0, Last orbit: 255 [INFO] Largest super-page: 11063360 B, largest TF: 11063360 B Real time 0:00:00, CP time 0.010 [O2/latest-ctpdev-o2] ~/o2tests2 $>

[lietava]

@shahor02 I dont see error ?

[shahor02]

ok, on this level it is fine. Surely, the final judgement is by decoding.

[mbombara]

Thanks. The git commands

git fetch --all
git reset --hard origin/<branch_name>

worked! After building I run:

o2-sim -n 10 -m FT0 FV0 ITS CTP -g pythia8pp -j10
o2-sim-digitizer-workflow -b
o2-ctp-digi2raw

The file ctp.raw was indeed created. What surprised me was the size: ctpdigits.root has 7kB, ctp.raw has 22MB. Do you have some macro for reading the raw file?

[lietava]

@mbombara no @shahor02 I realised now it is more natural to move CTP classes calculations to Digitizer so going to fix it

[lietava]

we need to connect meaning of CTP inputs with detector inputs (eg V0, T0). Conceptually it can be done either by using input names or also code number - so called signature - we have in run2. Techically I guess I need to access detector config file in database.

[lietava]

HI @mbombara , please, take the latest ctpdev and test: o2-sim -n 10 -m FT0 FV0 ITS CTP -g pythia8pp -j10 o2-sim-digitizer-workflow -b => ctpdigits.root o2-ctp-digi2raw o2-raw-file-reader-workflow --input-conf CTPraw.cfg | o2-ctp-reco-workflow -bls -ltr => o2_ctpdigits.root Clearly it does not work. So I suggest to start from beginning: 1.) check if digits are consistent with inputs (you already started this, I guess) and in correct format 2.) check raw data are consistent with digits - here I can see if to use some o2 tools or just binary dump 3.) raw to digits - let us discuss later Cheers, Roman.

[mbombara]

It does not work because: [ERROR] error parsing options of o2-ctp-reco-workflow: unrecognised option '-bls' Is it the same with you? I am going to check 1.)

[shahor02]

What is the -bls -ltr ?

[lietava]

ls -ltr is copy/paste remnant of linux command. Correct option is -b only. Sorry, R.

[mbombara]

I can confirm, with -bonly I got o2_ctpdigits.root .

[mbombara]

And it looks fine, i.e. histograms from the tree look reasonable.

[mbombara]

I have done tests by reading values in the trees. Digits from MC looks fine (ctpdigits.root). After generating 10 events the results are (shown only Orbits and BCs, but I have also Input and Class Masks if needed):

Orbit 0 (BCs): 252, 1396, 1414, 1748, 1974
Orbit 1 (BCs): 430, 431, 466, 467, 468, 2010, 2134, 2135
Orbit 2 (BCs): 226

Clearly, if I consider adjanced BCs as one (“pile-up”) event, I got 10 events.

digits from raw data do not look fine (o2_ctpdigits.root):

Orbit 0 (BCs): 0 512 768 1024 1280 1536 1792 2048 2304 2560 2816 3072 3328 3584 3840
Orbit 1 (BCs): 0 512 768 1024 1280 1536 1792 2048 2304 2560 2816 3072 3328 3584 3840

and so on untill (few times some numbers are missing):

Orbit 255 (BCs): 0 512 768 1024 1280 1536 1792 2048 2304 2560 2816 3072 3328 3584 3840

i.e. in each orbit (HBF?) there are 15 identical BCs and this is repeated 256 times (Time Frame?). Interesting thing is that these numbers look hexadecimally as:

Orbit N (BCs): 0x0 0x200 0x300 0x400 0x500 0x600 0x700 0x800 0x900 0xa00 0xb00 0xc00 0xd00 0xe00 0xf00

therefore I guess they are not really BCs. Does something look familiar, or I have to look at the code?

[shahor02]

@mbombara well, the decoded ones are certainly not BCs as they go above the allowed range. So, something is wrong in the encoding or decoding the raw data. As for the MC digits, having 3 pairs of consequent BCs in 10 events looks somewhat suspicious. Can you check what was actually generated, by doing from the root command line:

auto dc = o2::steer::DigitizationContext::loadFromFile();
auto& irs = dc->getEventRecords();
for (auto ir : irs) std::cout << ir << "\n";

You should see the list of sampled interaction records, like:

BCid:  580 Orbit:      0 |T in BC(ns): -0.055
BCid: 1040 Orbit:      0 |T in BC(ns): -0.138
BCid: 1180 Orbit:      0 |T in BC(ns): -0.237
BCid: 1938 Orbit:      0 |T in BC(ns): 0.248
BCid: 2158 Orbit:      0 |T in BC(ns): -0.215
BCid:  214 Orbit:      1 |T in BC(ns): -0.097
...

[mbombara]

Hi Ruben,

this is what I get:

BCid:  252 Orbit:      0 |T in BC(ns): -0.054
BCid: 1396 Orbit:      0 |T in BC(ns): 0.052
BCid: 1414 Orbit:      0 |T in BC(ns): -0.002
BCid: 1748 Orbit:      0 |T in BC(ns): 0.218
BCid: 1974 Orbit:      0 |T in BC(ns): 0.099
BCid:  430 Orbit:      1 |T in BC(ns): 0.011
BCid:  466 Orbit:      1 |T in BC(ns): -0.156
BCid: 2010 Orbit:      1 |T in BC(ns): 0.087
BCid: 2134 Orbit:      1 |T in BC(ns): -0.012
BCid:  226 Orbit:      2 |T in BC(ns): 0.037

[shahor02]

@mbombara thanks, so, the "paired BCs" mean that the FT0 "triggers" on slow particles. How it was handled in Run2, @AllaMaevskaya ?

[lietava]

@shahor02 : is there tool to dump raw data like rdh + payload ?

[shahor02]

@lietava the RDHs only can be dumped by in human readable form by o2-raw-file-check -v 2 <rawfilename> or in hex by the same using -v 3.

Otherwise, you should either use hexdump or this python script (prints GBT words from right to left), you can recognize RDH by 4006

[lietava]

@shahor02 : thanks @mbombara here is link to RDH definition. CTP payload definition is at CTP presentation on today trigger meeting.

[mbombara]

@lietava link to RDH does not work.

[mbombara]

There is no link, at least I can not click on anything in the sentence..

[lietava]

https://indico.cern.ch/event/1028936/contributions/4320435/attachments/2228986/3776460/cru.pdf

[lietava]

Now is there, sorry.

[mbombara]

@lietava, this link is very useful, thank you!

[mbombara]

I have written a macro which can read the raw data dump. The dump reader output as well as other files are stored in:

https://cernbox.cern.ch/index.php/s/22QCwEWERfgSX8C

The raw data dump is produced by the python script sent by @shahor02 :

python eventDump4.py ctp.raw > evdump.txt

The dump reader output is produced by:

root.exe RawEventDumpReader.C > RawDumpReaderOutput.txt

The data can be divided into two parts according to the link to CRU. Link 1 (represented in RDH by LINK_ID = 1 or FEE_ID =1) should be related to ClassMask payload, link 0 (represented in RDH by LINK_ID=0 or FEE_ID=0) should be related to Interaction Record (InputMask). Each part represents one TF with 256 HBF (Orbits). The data are not zero supressed so there is payload for every BC. The data for each link are the same. Each starts by BCID (12 bits) follows by 12x4 = 48 bits payload. Since the trigger data are non zero suppressed there are always 5 raw data pages (one raw data page is 8KB). I have not found any non zero payload. The BC IDs go from 0 to 3563, followed by 0 payloads. I.e. the generated 10 events are not present in the raw data.

[shahor02]

Hi,

To start with: in the digits all your input masks seem to be empty, is this normal?

root [11] o2sim->Scan("CTPDigits.printStream(std::cout)")
***********************************
*    Row   * Instance * CTPDigits *
***********************************
CTP Digit:  BC 4 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *        0 *         0 *
CTP Digit:  BC 6 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *        1 *         0 *
CTP Digit:  BC 8 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *        2 *         0 *
CTP Digit:  BC 10 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *        3 *         0 *
CTP Digit:  BC 12 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *        4 *         0 *
CTP Digit:  BC 14 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *        5 *         0 *
CTP Digit:  BC 16 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *        6 *         0 *
CTP Digit:  BC 17 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *        7 *         0 *
CTP Digit:  BC 20 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *        8 *         0 *
CTP Digit:  BC 22 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000

[lietava]

The BC which has non zero inputs should have non zero masks. BC s with no inputs should not be in digits. For simulation, I thought , the digits were checked by Marek at: https://github.com/lietava/AliceO2/issues/5#issuecomment-845897249 ?

[mbombara]

Digits from MC looks fine (ctpdigits.root), as I wrote before:

root [1] o2sim->Scan("CTPDigits.printStream(std::cout)")
***********************************
*    Row   * Instance * CTPDigits *
***********************************
CTP Digit:  BC 252 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *        0 *         0 *
CTP Digit:  BC 1396 orbit 0
Input Mask: 0000000000000000000000000000000000000000000101
*        0 *        1 *         0 *
CTP Digit:  BC 1414 orbit 0
Input Mask: 0000000000000000000000000000000000000000000001
*        0 *        2 *         0 *
CTP Digit:  BC 1748 orbit 0
Input Mask: 0000000000000000000000000000000000000000000101
*        0 *        3 *         0 *
CTP Digit:  BC 1974 orbit 0
Input Mask: 0000000000000000000000000000000000000000000101
*        0 *        4 *         0 *
CTP Digit:  BC 430 orbit 1
Input Mask: 0000000000000000000000000000000000000000000101

The problematic ones are digits from raw (o2_ctpdigits.root):

root [1] o2sim->Scan("CTPDigits.printStream(std::cout)")
***********************************
*    Row   * Instance * CTPDigits *
***********************************
CTP Digit:  BC 0 orbit 0
Input Mask: 0000000101110000000000000000000000000000000000
*        0 *        0 *         0 *
CTP Digit:  BC 512 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *        1 *         0 *
CTP Digit:  BC 768 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *        2 *         0 *
CTP Digit:  BC 1024 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *        3 *         0 *
CTP Digit:  BC 1280 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *        4 *         0 *
CTP Digit:  BC 1536 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *        5 *         0 *
CTP Digit:  BC 1792 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *        6 *         0 *
CTP Digit:  BC 2048 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *        7 *         0 *
CTP Digit:  BC 2304 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *        8 *         0 *
CTP Digit:  BC 2560 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *        9 *         0 *
CTP Digit:  BC 2816 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *       10 *         0 *
CTP Digit:  BC 3072 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *       11 *         0 *
CTP Digit:  BC 3328 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *       12 *         0 *
CTP Digit:  BC 3584 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000
*        0 *       13 *         0 *
CTP Digit:  BC 3840 orbit 0
Input Mask: 0000000000000000000000000000000000000000000000

The BCids are not originals (they even go beyond the limit). I have checked the raw data as I mentioned and found empty payload for every BC for whole timeframe.

[shahor02]

@mbombara sorry, my fault: my prev. message about absence of masks was based on the digits from the full system test produced some time ago. Rerunning digitization again over the same simulation I see the mask. Not sure what was the reason, but in any case the simulated digits are ok.

[mbombara]

Hi @shahor02 , @lietava, just a recent status of raw data encoding (to keep you informed). Most recent fixes (focusing on Input Mask payload only) do propagate payload to raw data, but only for first 4 BCids (the reason is being studied). I.e. for example for this list:

BCid:  222 Orbit:      0 |T in BC(ns): 0.170
BCid:  236 Orbit:      0 |T in BC(ns): -0.014
BCid:  628 Orbit:      0 |T in BC(ns): -0.069
BCid: 1742 Orbit:      0 |T in BC(ns): -0.254
BCid: 1810 Orbit:      0 |T in BC(ns): -0.297
BCid:  410 Orbit:      1 |T in BC(ns): 0.245
BCid:  982 Orbit:      1 |T in BC(ns): 0.490
BCid: 1048 Orbit:      1 |T in BC(ns): 0.145
BCid: 1222 Orbit:      1 |T in BC(ns): -0.252
BCid: 1952 Orbit:      1 |T in BC(ns): 0.240

I could see (either by my script or by eyes from raw data dump) no empty payload for BC ids 222, 236, 628 and 1742. Rest payloads are empty.

[lietava]

1.) o2-sim -n 10 -m FT0 FV0 ITS CTP -g pythia8pp -j10 = generation of interactions 2.) o2-sim-digitizer-workflow -b = creation of CTP digits (and other digits) from interactions 3.) o2-ctp-digi2raw = creation of raw data from digits 4.) o2-raw-file-reader-workflow --input-conf CTPraw.cfg | o2-ctp-reco-workflow -b = reading raw data and creation of digits

And the brutal test is 2 = 4 o2-sim-digitizer-workflow: --interactionRate

[mbombara]

Attaching the macro for the brutal test (2=4). Just change .txt to .C (.C is apparently not supported for attaching). CTPdigitsReader.txt

[lietava]

@shahor02 How can I get LOG(DEBUG) when running code ?

[shahor02]

Try with --severity debug option, but as far as I remember, the code has to be compiled with debug flag. Usually I temporary change DEBUG to INFO

[lietava]

Hmmm, this is what I was just doing :) Thanks.

[shahor02]

Hi @lietava @mbombara sorry, I did not understand if there was a conclusion on CTP raw encoding / decoding, does it work?

[lietava]

Hi Ruben, I was still working on it as you can see from the commits. As far as I can see ctpdigits and o2_ctpdigits are now consistent. I would still like if @mbombara and @JColburn-CERN would have independent look.