Distributed TRC support

[okt-kostik]

TE allows to have TRC in the tests. This is really great.

Said that if one is developing a NIC and wants to track results that are specific to the NIC. W/o merging the results into the TRC of the test suite:

• it's very bad for merges
• it can't be upstreamed given that it might be private.

I suggest you support distributed TRC. Namely:

• TRC might be located in several places
• Places have priority
• If a result expectation is specified in high priority location - it overrides low priority
• it's possible to specify:

 359     <test name="mtu_tcp" type="script">
 360       <objective>Check that TCP packets of full MTU size are sent and processed correctly.</objective>
 361       <notes/>
 362       <iter result="PASSED">
 363         <arg name="env"/>
 364         <arg name="mtu">576</arg>
 365         <arg name="tx"/>
 366         <arg name="gso_on"/>
 367         <arg name="tso_on">FALSE</arg>
 368         <arg name="pkt_size"/>
 369         <arg name="sends_num"/>

something like:

<test name="mtu_tcp" type="script" override="true">

that will result in override of the entire subtree by the content in the file

[ol-dmitryiz]

I propose the following plan. Let test suite have more than one TRC database, like trc and trc_private (in private branch). So we can have more than one --trc-db option in command line. In that case the second TRC database (and third and etc) is merged into the first one loaded in memory. Merging is simple – for every test all iteration records from the secondary database are added before or after initial iterations depending on whether they should overwrite them.

This may lead to a situation when for a given test iteration from log more than one iteration record matches. So handling of such situation in trc_db_walker_step_iter() should be changed (may be as an option) so that in such case it does not print big warning and consider only the last matching record but silently collects expected results from all matching iteration records and then chooses matching one among them.

[ol-dmitryiz]

Or, to avoid adding handling of multiple --trc-db options in various TRC tools and Tester, a simple TRC tool can be added that merges multiple TRC databases into a single temporary one which is then used during testing.

[ol-andrewr]

Yes, distributed assumes more than one TRC DB instance.

I think example with private branch is bad. In practice private part should be either located in ts-rigs or separate repository.

Merging rules is the key problem here. To start with I'd just append it to tail. Later, if we have problems with it, we'll add priorities to achieve required order (order of expectations matter).

If TRC tags match exactly, entries from subsequent DB should overwrite previous.

I'd not touch trc_db_walker_step_iter() etc for now. It will overcomplicate understanding. Right now first match wins and let's keep it for now and care about correct order on merging.

lib/trc should care about this logic. Interface for Tester and TRC tools should remain (may be except initialization). So, logical DB is only one for these tools.

[ol-dmitryiz]

Unfortunately I cannot avoid touching trc_db_walker_step_iter() because it has a problem with more than one matching iteration entry. We hardly want one or many messages like

                  "TEST='%s || %s'\n"
                  "Hash: %s\n"
                  "Duplicated iteration in the database! "
                  "May be caused by wrong wildcards. "
                  "Will match the last entry for compatibility, but FIX "
                  "THE DATABASE!!!"

in console during normal run.

And multiple matching iteration records cannot be avoided during merging into single logical TRC DB because of wildcard iteration entries for which it is not known whether sets of matching actual iterations corresponding to them can intersect.

[ol-andrewr]

Can we fix it on merging by split wildcard iterations to be merged with non-wildcard in another DB?

[ol-dmitryiz]

I do not understand what you propose.

[ol-andrewr]

Let's consider the following case:

DB-A:
    <test "a">
       <iter result="PASSED">
          <arg name+"p"/>
          <notes/>
          <results tags+"tagA">
             <result value+"FAILED">
                <verdict>message A</verdict>
             </result>
          </results>
       </iter>
    </test>

DB-B:
    <test "a">
       <iter result="PASSED">
          <arg name+"p">
             <value>v1</value>
          </arg>
          <notes/>
          <results tags+"tagB">
             <result value+"FAILED">
                <verdict>message B1</verdict>
             </result>
          </results>
       </iter>
       <iter result="PASSED">
          <arg name+"p">
             <value>v2</value>
          </arg>
          <notes/>
          <results tags+"tagB">
             <result value+"FAILED">
                <verdict>message B2</verdict>
             </result>
          </results>
       </iter>
    </test>

DB-RESULT:
    <test "a">
       <iter result="PASSED">
          <arg name+"p">
             <value>v1</value>
          </arg>
          <notes/>
          <results tags+"tagA">
             <result value+"FAILED">
                <verdict>message A</verdict>
             </result>
          </results>
          <results tags+"tagB">
             <result value+"FAILED">
                <verdict>message B1</verdict>
             </result>
          </results>
       </iter>
       <iter result="PASSED">
          <arg name+"p">
             <value>v2</value>
          </arg>
          <notes/>
          <results tags+"tagA">
             <result value+"FAILED">
                <verdict>message A</verdict>
             </result>
          </results>
          <results tags+"tagB">
             <result value+"FAILED">
                <verdict>message B2</verdict>
             </result>
          </results>
       </iter>
    </test>

May be I underestimate merging complexity here, but I'd like to note that there is no necessity to optimize number of iterations. Just expand argument values if required. If argument is wildcard in DB-A and have values in DB-B, just add the same values to DB-RESULT and duplicate DB-A expectations (results entries). Note that entries form A must go by default before entries from B.

[ol-dmitryiz]

"Just expand argument values if required" - what do you mean by that? TRC has no knowledge about package.xml and so cannot replace wildcards with full records. And given that we actually saw test suite having a billion iterations it may be costly.

TRC_A: param1=a param2= tags1: result1

TRC_B param1= param2=b tags2: result2

How to merge this so that we can be sure there is no iteration which can match more than one iteration record?

[ol-dmitryiz]

Perhaps I can merge this like

TRC_RESULT param1=a param2= tags1: result1

param1= param2=b tags2: result2

param1=a param2=b tags1: result1 tags2: result2

to minimize changes in trc_db_walker_step_iter() (instead of the last matching record, it will just have to select the last record with most non-wildcard matches).

[ol-andrewr]

No, it should be just:

TRC_RESULT param1=a param2=b tags1: result1 tags2: result2

Your merge is bad since it will be unclear which expectation to use for iteration a+b (3 iters match in your merge result)

[ol-dmitryiz]

As I explained, the most specific (with least wildcards) match will be used.

Your merge is bad because what about iteration

param1=a param2=c

or

param1=x param2=b

With your merge they no longer match anything.

[ol-andrewr]

OK, I get you point. If so, we have no any better option than first-match-wins and result must be:

param1=a param2=b tags1: result1 tags2: result2

param1=a param2= tags1: result1

param1= param2=b tags2: result2