New Feature: Interface for repeated register read

bdorney commented 5 years ago

Description

Adding some functionality to send an RPC call that will repeatedly read a given register and query the VFAT3 slow control error counters to see if a communication error occurred during this period.

Note a bus error (e.g. 0xdeaddead is returned) when accessing a VFAT register always causes the timeout error counter in this block of the AMC FW to increment; so this will catch these type, and other communication errors.

Requires:

https://github.com/cms-gem-daq-project/ctp7_modules/pull/115

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Motivation and Context

Following our conversation about how we would pick a "probably good" GBT phase and then perform N >> 1 slow control actions with this VFAT it was necessary to implement an RPC method for this and this serves as the interface.

How Has This Been Tested?

QC7 land...

Screenshots (if appropriate):

Checklist:

[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

lpetre-ulb commented 5 years ago

[I'm not sure if I should comment here or open a new issue, but since the issue concerns this PR I'll comment here.]

When trying to debug communication with VFAT14 on GE11-X-S-INDIA0014, this kind of output appeared in the CTP7 log file :

Apr 30 12:27:03 CTP7 local0.notice rpcsvc[20951]: rpcsvc: Client connected from 192.168.0.180
Apr 30 12:27:03 CTP7 local0.info rpcsvc[20951]: memory:  Memhub initialized a semaphore. Current semaphore value = 1
Apr 30 12:27:03 CTP7 local0.info rpcsvc[20951]: gbt.writeGBTPhase: Writing 9 to the VFAT #1 phase of OH #0.
Apr 30 12:27:04 CTP7 local0.info rpcsvc[20951]: amc.repeatedRegRead: attempting to repeatedly reading register $^? for 1000 times
Apr 30 12:27:04 CTP7 local0.err rpcsvc[20951]: amc.repeatedRegRead: Key: $^? is NOT found
Apr 30 12:27:04 CTP7 local0.info rpcsvc[20951]: amc.repeatedRegRead: attempting to repeatedly reading register $^? for 1000 times
Apr 30 12:27:04 CTP7 local0.err rpcsvc[20951]: amc.repeatedRegRead: Key: $^? is NOT found
Apr 30 12:27:04 CTP7 local0.info rpcsvc[20951]: amc.repeatedRegRead: attempting to repeatedly reading register $^? for 1000 times
Apr 30 12:27:04 CTP7 local0.err rpcsvc[20951]: amc.repeatedRegRead: Key: $^? is NOT found

So, it seems that the repeated reads for the checking the GBT RX phase for "All good phases" issue are currently not performed (and the check always succeed during the testConnectivity.py procedure).

One part of the issue could be in the RPC module which breaks without error if the register name does not exist; see src/utils.cpp:401 of this PR. However, I think the main issue lies in this PR; see the comments inline for details.

bdorney commented 5 years ago

Apr 30 12:27:04 CTP7 local0.err rpcsvc[20951]: amc.repeatedRegRead: Key: $^? is NOT found Apr 30 12:27:04 CTP7 local0.info rpcsvc[20951]: amc.repeatedRegRead: attempting to repeatedly reading register $^? for 1000 times

This issue is resolved now:

May  1 13:42:58 CTP7 local0.info rpcsvc[9538]: memory:  Memhub initialized a semaphore. Current semaphore value = 1
May  1 13:42:59 CTP7 local0.info rpcsvc[9538]: gbt.writeGBTPhase: Writing 7 to the VFAT #23 phase of OH #4.
May  1 13:42:59 CTP7 local0.info rpcsvc[9538]: amc.repeatedRegRead: attempting to repeatedly reading register GEM_AMC.OH.OH4.GEB.VFAT23.CFG_RUN for 1000 times
May  1 13:42:59 CTP7 local0.info rpcsvc[9538]: amc.repeatedRegRead: attempting to repeatedly reading register GEM_AMC.OH.OH4.GEB.VFAT23.HW_ID for 1000 times
May  1 13:42:59 CTP7 local0.info rpcsvc[9538]: amc.repeatedRegRead: attempting to repeatedly reading register GEM_AMC.OH.OH4.GEB.VFAT23.HW_ID_VER for 1000 times

bdorney commented 5 years ago

@jsturdy @mexanick rebase'd off the current HEAD of legacyPreBoost.

jsturdy commented 5 years ago

@jsturdy @mexanick rebase'd off the current HEAD of legacyPreBoost.

@bdorney, needs an update, this is next in the list, i.e., additional PRs to xhal:legacyPreBoost will be done after this

bdorney commented 5 years ago

@jsturdy @mexanick rebase'd off the current HEAD of legacyPreBoost.

@bdorney, needs an update, this is next in the list, i.e., additional PRs to xhal:legacyPreBoost will be done after this

done

cms-gem-daq-project / xhal