Open jinjingforever opened 9 years ago
@jinjingforever The app_id isn't in the test name, it's in the error message.
The test name is [test/integration/test-batch.js].readonly batches
which should be fine.
The full console output for this test is below:
# [test/integration/test-batch.js].readonly batches
not ok 12 Error: NaClMain:<wspr>"test/syncbased/app_1435618766746".Create: No usable servers found: test/syncbased/app_1435618766746 NaClMain:<rpc.Client>"test/syncbased/app_1435618766746".__Signature: No usable servers found: [/@5@wsh@23.251.152.245:57650@c8962382ffeee44cbe89bf902aeee889@s@test@@/app_1435618766746:test/syncbased/app_1435618766746.__Signature: : failed]
---
operator: error
expected:
undefined
actual:
{ _langId: 'en-US', id: 'v.io/v23/verror.NoServers', msg: 'NaClMain:<wspr>"test/syncbased/app_1435618766746".Create: No usable servers found: test/syncbased/app_1435618766746 NaClMain:<rpc.Client>"test/syncbased/app_1435618766746".__Signature: No usable servers found: [/@5@wsh@23.251.152.245:57650@c8962382ffeee44cbe89bf902aeee889@s@test@@/app_1435618766746:test/syncbased/app_1435618766746.__Signature: : failed]', paramList: [ { val: 'NaClMain' }, { val: '<wspr>"test/syncbased/app_1435618766746".Create' }, { val: 'test/syncbased/app_1435618766746' }, { _langId: 'en-US', id: 'v.io/v23/verror.NoServers', msg: 'NaClMain:<rpc.Client>"test/syncbased/app_1435618766746".__Signature: No usable servers found: [/@5@wsh@23.251.152.245:57650@c8962382ffeee44cbe89bf902aeee889@s@test@@/app_1435618766746:test/syncbased/app_1435618766746.__Signature: : failed]', paramList: [ { val: 'NaClMain' }, { val: '<rpc.Client>"test/syncbased/app_1435618766746".__Signature' }, { val: [ ... ] } ], retryCode: 'RetryRefetch' } ], retryCode: 'RetryRefetch' }
at: Test.assert (http://0.0.0.0:7559/assets/run.js:8112:17)
stack:
Error: NaClMain:<wspr>"test/syncbased/app_1435618766746".Create: No usable servers found: test/syncbased/app_1435618766746 NaClMain:<rpc.Client>"test/syncbased/app_1435618766746".__Signature: No usable servers found: [/@5@wsh@23.251.152.245:57650@c8962382ffeee44cbe89bf902aeee889@s@test@@/app_1435618766746:test/syncbased/app_1435618766746.__Signature: : failed]
at new Errors.(anonymous function) (http://0.0.0.0:7559/assets/run.js:33121:19)
at fromNativeValue (http://0.0.0.0:7559/assets/run.js:30535:11)
at Object.fromNativeValue (http://0.0.0.0:7559/assets/run.js:31178:12)
at canonicalize (http://0.0.0.0:7559/assets/run.js:29152:34)
at canonicalizeExternal (http://0.0.0.0:7559/assets/run.js:29115:10)
at Object.canonicalizeReduce [as reduce] (http://0.0.0.0:7559/assets/run.js:29080:10)
at http://0.0.0.0:7559/assets/run.js:34013:27
...
You are right about the test name. But I think the app id is in the name of the test case :)
Strange. Something must be wrong with the TAP parsing if it thinks the error is part of the test case.
@aghassemi Do you have any ideas?
This is kind of WAI. So, from TAP to XML, the rules are that the "Test" name becomes the <testsuite>
and each "assertion" in TAP output for that test becomes a <testcase>
so for
#My Awesome Test
ok 1 Name is good
not ok 2 Id is bad
we end up with two XML test cases as "Name is good" and "Id is bad" in a test suite named "My Awesome Test"
Now in this example, the TAP "assertion" name is actually the error.message
itself ( Because TAPE does that for t.error()
by default. Providing a manual msg to the t.error()
should override this behaviour. )
so for
# [test/integration/test-batch.js].readonly batches
not ok 12 Error: NaClMain:<wspr>"test/syncbased/app_1435618766746".Create: No usable servers found: test/syncbased/app_1435618766746 NaClMain:<rpc.Client>"test/syncbased/app_1435618766746".__Signature: No usable servers found: [/@5@wsh@23.251.152.245:57650@c8962382ffeee44cbe89bf902aeee889@s@test@@/app_1435618766746:test/syncbased/app_1435618766746.__Signature: : failed]
---
operator: error
expected:
undefined
actual:
{ _langId: 'en-US', id: 'v.io/v23/verror.NoServers', msg: 'NaClMain:<wspr>"test/syncbased/app_1435618766746".Create: No usable servers found: test/syncbased/app_1435618766746 NaClMain:<rpc.Client>"test/syncbased/app_1435618766746".__Signature: No usable servers found: [/@5@wsh@23.251.152.245:57650@c8962382ffeee44cbe89bf902aeee889@s@test@@/app_1435618766746:test/syncbased/app_1435618766746.__Signature: : failed]', paramList: [ { val: 'NaClMain' }, { val: '<wspr>"test/syncbased/app_1435618766746".Create' }, { val: 'test/syncbased/app_1435618766746' }, { _langId: 'en-US', id: 'v.io/v23/verror.NoServers', msg: 'NaClMain:<rpc.Client>"test/syncbased/app_1435618766746".__Signature: No usable servers found: [/@5@wsh@23.251.152.245:57650@c8962382ffeee44cbe89bf902aeee889@s@test@@/app_1435618766746:test/syncbased/app_1435618766746.__Signature: : failed]', paramList: [ { val: 'NaClMain' }, { val: '<rpc.Client>"test/syncbased/app_1435618766746".__Signature' }, { val: [ ... ] } ], retryCode: 'RetryRefetch' } ], retryCode: 'RetryRefetch' }
at: Test.assert (http://0.0.0.0:7559/assets/run.js:8112:17)
stack:
Error: NaClMain:<wspr>"test/syncbased/app_1435618766746".Create: No usable servers found: test/syncbased/app_1435618766746 NaClMain:<rpc.Client>"test/syncbased/app_1435618766746".__Signature: No usable servers found: [/@5@wsh@23.251.152.245:57650@c8962382ffeee44cbe89bf902aeee889@s@test@@/app_1435618766746:test/syncbased/app_1435618766746.__Signature: : failed]
at new Errors.(anonymous function) (http://0.0.0.0:7559/assets/run.js:33121:19)
at fromNativeValue (http://0.0.0.0:7559/assets/run.js:30535:11)
at Object.fromNativeValue (http://0.0.0.0:7559/assets/run.js:31178:12)
at canonicalize (http://0.0.0.0:7559/assets/run.js:29152:34)
at canonicalizeExternal (http://0.0.0.0:7559/assets/run.js:29115:10)
at Object.canonicalizeReduce [as reduce] (http://0.0.0.0:7559/assets/run.js:29080:10)
at http://0.0.0.0:7559/assets/run.js:34013:27
...
ok 13 rt.close should not error.
not ok 14 Timeout, test did not finish in 20000ms
In the above TAP output, Error: NaClMain:<wspr>"test/syncbased/app_1435618766746".Create: No usable servers found: test/syncbased/app_1435618766746 NaClMain:<rpc.Client>"test/syncbased/app_1435618766746".__Signature: No usable servers found: [/@5@wsh@23.251.152.245:57650@c8962382ffeee44cbe89bf902aeee889@s@test@@/app_1435618766746:test/syncbased/app_1435618766746.__Signature: : failed]
is the name of the 12th asersion, same way rt.close should not error
is the name for assertion 13.
Two possible solutions
error.message
as the message for the assertion by default.t.error(err, 'Should not error')
)I guess another issue is that the reporter here https://dashboard.staging.v.io/index.html?type=presubmit&n=8532 does not group test cases by their test suite so the list looks confusing as every test has the same name. Is it possible to configure the reporter to group test cases by their suite in the UI ( if not, TAP->XUNIT can prepend the suite name to the test case name ).
Thanks for the explanations Ali! Either solution looks good to me. This is not urgent. Please take your time!
For the UI issue, I can prepend the test suite name to the test case name from the dashboard UI side, because other test results also have this issue. Thanks!
I noticed that some syncbase js tests have "app id" in their test case names. For example:
https://dashboard.staging.v.io/?arch=amd64&class=&job=vanadium-js-syncbase-browser&n=8532&os=linux&part=0&suite=javascript.browser.%5Btest%2Fintegration%2Ftest-batch.js%5D.readonly+batches&test=%2312+Error%3A+NaClMain%3A%3Cwspr%3E%22test%2Fsyncbased%2Fapp_1435618766746%22.Create%3A+No+usable+servers+found%3A+test%2Fsyncbased%2Fapp_1435618766746+NaClMain%3A%3Crpc.Client%3E%22test%2Fsyncbased%2Fapp_1435618766746%22.__Signature%3A+No+usable+servers+found%3A+%5B%2F%405%40wsh%4023.251.152.245%3A57650%40c8962382ffeee44cbe89bf902aeee889%40s%40test%40%40%2Fapp_1435618766746%3Atest%2Fsyncbased%2Fapp_1435618766746.__Signature%3A+%3A+failed%5D&type=presubmit#
Notice the "app_1435618766746" part. I guess this part will be different every time this test runs. This will cause problems in presubmit test logic because it relies on the test case name to tell whether a failed test case is "new" or "known".
Thanks!