search

vanadium / issues

Vanadium issue tracker
1 stars 1 forks source link

Don't include "app id" in test case name #578

Open jinjingforever opened 9 years ago

jinjingforever commented 9 years ago

I noticed that some syncbase js tests have "app id" in their test case names. For example:

https://dashboard.staging.v.io/?arch=amd64&class=&job=vanadium-js-syncbase-browser&n=8532&os=linux&part=0&suite=javascript.browser.%5Btest%2Fintegration%2Ftest-batch.js%5D.readonly+batches&test=%2312+Error%3A+NaClMain%3A%3Cwspr%3E%22test%2Fsyncbased%2Fapp_1435618766746%22.Create%3A+No+usable+servers+found%3A+test%2Fsyncbased%2Fapp_1435618766746+NaClMain%3A%3Crpc.Client%3E%22test%2Fsyncbased%2Fapp_1435618766746%22.__Signature%3A+No+usable+servers+found%3A+%5B%2F%405%40wsh%4023.251.152.245%3A57650%40c8962382ffeee44cbe89bf902aeee889%40s%40test%40%40%2Fapp_1435618766746%3Atest%2Fsyncbased%2Fapp_1435618766746.__Signature%3A+%3A+failed%5D&type=presubmit#

Notice the "app_1435618766746" part. I guess this part will be different every time this test runs. This will cause problems in presubmit test logic because it relies on the test case name to tell whether a failed test case is "new" or "known".

Thanks!

nlacasse commented 9 years ago

@jinjingforever The app_id isn't in the test name, it's in the error message.

The test name is [test/integration/test-batch.js].readonly batches which should be fine.

The full console output for this test is below:

# [test/integration/test-batch.js].readonly batches
not ok 12 Error: NaClMain:<wspr>"test/syncbased/app_1435618766746".Create: No usable servers found: test/syncbased/app_1435618766746 NaClMain:<rpc.Client>"test/syncbased/app_1435618766746".__Signature: No usable servers found: [/@5@wsh@23.251.152.245:57650@c8962382ffeee44cbe89bf902aeee889@s@test@@/app_1435618766746:test/syncbased/app_1435618766746.__Signature: : failed]
  ---
    operator: error
    expected:
      undefined
    actual:
      { _langId: 'en-US', id: 'v.io/v23/verror.NoServers', msg: 'NaClMain:<wspr>"test/syncbased/app_1435618766746".Create: No usable servers found: test/syncbased/app_1435618766746 NaClMain:<rpc.Client>"test/syncbased/app_1435618766746".__Signature: No usable servers found: [/@5@wsh@23.251.152.245:57650@c8962382ffeee44cbe89bf902aeee889@s@test@@/app_1435618766746:test/syncbased/app_1435618766746.__Signature: : failed]', paramList: [ { val: 'NaClMain' }, { val: '<wspr>"test/syncbased/app_1435618766746".Create' }, { val: 'test/syncbased/app_1435618766746' }, { _langId: 'en-US', id: 'v.io/v23/verror.NoServers', msg: 'NaClMain:<rpc.Client>"test/syncbased/app_1435618766746".__Signature: No usable servers found: [/@5@wsh@23.251.152.245:57650@c8962382ffeee44cbe89bf902aeee889@s@test@@/app_1435618766746:test/syncbased/app_1435618766746.__Signature: : failed]', paramList: [ { val: 'NaClMain' }, { val: '<rpc.Client>"test/syncbased/app_1435618766746".__Signature' }, { val: [ ... ] } ], retryCode: 'RetryRefetch' } ], retryCode: 'RetryRefetch' }
    at: Test.assert (http://0.0.0.0:7559/assets/run.js:8112:17)
    stack:
      Error: NaClMain:<wspr>"test/syncbased/app_1435618766746".Create: No usable servers found: test/syncbased/app_1435618766746 NaClMain:<rpc.Client>"test/syncbased/app_1435618766746".__Signature: No usable servers found: [/@5@wsh@23.251.152.245:57650@c8962382ffeee44cbe89bf902aeee889@s@test@@/app_1435618766746:test/syncbased/app_1435618766746.__Signature: : failed]
        at new Errors.(anonymous function) (http://0.0.0.0:7559/assets/run.js:33121:19)
        at fromNativeValue (http://0.0.0.0:7559/assets/run.js:30535:11)
        at Object.fromNativeValue (http://0.0.0.0:7559/assets/run.js:31178:12)
        at canonicalize (http://0.0.0.0:7559/assets/run.js:29152:34)
        at canonicalizeExternal (http://0.0.0.0:7559/assets/run.js:29115:10)
        at Object.canonicalizeReduce [as reduce] (http://0.0.0.0:7559/assets/run.js:29080:10)
        at http://0.0.0.0:7559/assets/run.js:34013:27
  ...
jinjingforever commented 9 years ago

You are right about the test name. But I think the app id is in the name of the test case :)

https://paste.googleplex.com/4528344041259008

nlacasse commented 9 years ago

Strange. Something must be wrong with the TAP parsing if it thinks the error is part of the test case.

@aghassemi Do you have any ideas?

aghassemi commented 9 years ago

This is kind of WAI. So, from TAP to XML, the rules are that the "Test" name becomes the <testsuite> and each "assertion" in TAP output for that test becomes a <testcase> so for

#My Awesome Test
ok 1 Name is  good
not ok 2 Id is bad

we end up with two XML test cases as "Name is good" and "Id is bad" in a test suite named "My Awesome Test"

Now in this example, the TAP "assertion" name is actually the error.message itself ( Because TAPE does that for t.error() by default. Providing a manual msg to the t.error() should override this behaviour. ) so for

# [test/integration/test-batch.js].readonly batches
not ok 12 Error: NaClMain:<wspr>"test/syncbased/app_1435618766746".Create: No usable servers found: test/syncbased/app_1435618766746 NaClMain:<rpc.Client>"test/syncbased/app_1435618766746".__Signature: No usable servers found: [/@5@wsh@23.251.152.245:57650@c8962382ffeee44cbe89bf902aeee889@s@test@@/app_1435618766746:test/syncbased/app_1435618766746.__Signature: : failed]
  ---
    operator: error
    expected:
      undefined
    actual:
      { _langId: 'en-US', id: 'v.io/v23/verror.NoServers', msg: 'NaClMain:<wspr>"test/syncbased/app_1435618766746".Create: No usable servers found: test/syncbased/app_1435618766746 NaClMain:<rpc.Client>"test/syncbased/app_1435618766746".__Signature: No usable servers found: [/@5@wsh@23.251.152.245:57650@c8962382ffeee44cbe89bf902aeee889@s@test@@/app_1435618766746:test/syncbased/app_1435618766746.__Signature: : failed]', paramList: [ { val: 'NaClMain' }, { val: '<wspr>"test/syncbased/app_1435618766746".Create' }, { val: 'test/syncbased/app_1435618766746' }, { _langId: 'en-US', id: 'v.io/v23/verror.NoServers', msg: 'NaClMain:<rpc.Client>"test/syncbased/app_1435618766746".__Signature: No usable servers found: [/@5@wsh@23.251.152.245:57650@c8962382ffeee44cbe89bf902aeee889@s@test@@/app_1435618766746:test/syncbased/app_1435618766746.__Signature: : failed]', paramList: [ { val: 'NaClMain' }, { val: '<rpc.Client>"test/syncbased/app_1435618766746".__Signature' }, { val: [ ... ] } ], retryCode: 'RetryRefetch' } ], retryCode: 'RetryRefetch' }
    at: Test.assert (http://0.0.0.0:7559/assets/run.js:8112:17)
    stack:
      Error: NaClMain:<wspr>"test/syncbased/app_1435618766746".Create: No usable servers found: test/syncbased/app_1435618766746 NaClMain:<rpc.Client>"test/syncbased/app_1435618766746".__Signature: No usable servers found: [/@5@wsh@23.251.152.245:57650@c8962382ffeee44cbe89bf902aeee889@s@test@@/app_1435618766746:test/syncbased/app_1435618766746.__Signature: : failed]
        at new Errors.(anonymous function) (http://0.0.0.0:7559/assets/run.js:33121:19)
        at fromNativeValue (http://0.0.0.0:7559/assets/run.js:30535:11)
        at Object.fromNativeValue (http://0.0.0.0:7559/assets/run.js:31178:12)
        at canonicalize (http://0.0.0.0:7559/assets/run.js:29152:34)
        at canonicalizeExternal (http://0.0.0.0:7559/assets/run.js:29115:10)
        at Object.canonicalizeReduce [as reduce] (http://0.0.0.0:7559/assets/run.js:29080:10)
        at http://0.0.0.0:7559/assets/run.js:34013:27
  ...
ok 13 rt.close should not error.
not ok 14 Timeout, test did not finish in 20000ms

In the above TAP output, Error: NaClMain:<wspr>"test/syncbased/app_1435618766746".Create: No usable servers found: test/syncbased/app_1435618766746 NaClMain:<rpc.Client>"test/syncbased/app_1435618766746".__Signature: No usable servers found: [/@5@wsh@23.251.152.245:57650@c8962382ffeee44cbe89bf902aeee889@s@test@@/app_1435618766746:test/syncbased/app_1435618766746.__Signature: : failed] is the name of the 12th asersion, same way rt.close should not error is the name for assertion 13.

Two possible solutions

aghassemi commented 9 years ago

I guess another issue is that the reporter here https://dashboard.staging.v.io/index.html?type=presubmit&n=8532 does not group test cases by their test suite so the list looks confusing as every test has the same name. Is it possible to configure the reporter to group test cases by their suite in the UI ( if not, TAP->XUNIT can prepend the suite name to the test case name ).

jinjingforever commented 9 years ago

Thanks for the explanations Ali! Either solution looks good to me. This is not urgent. Please take your time!

For the UI issue, I can prepend the test suite name to the test case name from the dashboard UI side, because other test results also have this issue. Thanks!