Timed out waiting for devmgr to appear in mounttable

[hpucha]

https://veyron.corp.google.com/jenkins/job/vanadium-create-instance-test/905/console

[caprita]

I looked into this, and it's unclear why it's failing. One hypothesis is that we're not waiting long enough for the devmgr entry to show up in the local mounttable (a sign that the device is back post-claim). I've just checked in a cl (https://vanadium-review.googlesource.com/#/c/14419/) to add more information when the script fails.

[hpucha]

Here's another occurrence: https://veyron.corp.google.com/jenkins/job/vanadium-create-instance-test/927/console

[hpucha]

This has a bit more info

https://veyron.corp.google.com/jenkins/job/vanadium-create-instance-test/991/console

... we now hang for about 30 seconds ... [See v.io/i/306] ERROR: SetPermissions(devmgr/device) failed: device:"devmgr/device".SetPermissions: No usable servers found: [/@5@wsh@10.240.71.135:8150@ae46e4b6925475f3a742213b06616d88@s@dev.v.io/role/vprod/device/gce/vanadium-cell-master@@/device:devmgr/device.SetPermissions: : dial tcp 10.240.71.135:8150: connection timed out]

Cleaning up test instances

Deleted [https://www.googleapis.com/compute/v1/projects/google.com:veyron/zones/us-central1-c/instances/create-instance-test-20150814022232]. exit status 1: exit status 1

FAILED

[caprita]

This seems to be failing in a slightly different place now; I've added more logging on Wednesday (v.io/c/14489), but it doesn't help here :-) I'll make another change to dump out the device manager log in the new failure place as well. Ultimately, I strongly suspect it's a change in the rpc system that has been causing this recent flakiness, but it may take a while to track down since it's not easily reproducible. Thanks for your patience :-)

[hpucha]

Perhaps this is correlated although it gets further along before failing. https://veyron.corp.google.com/jenkins/job/vanadium-create-instance-test/1024/console

[ghost]

Could this also be related? https://veyron.corp.google.com/jenkins/job/vanadium-create-instance-test/1679/console 09:29:04 ##### Checking test instance ##### 09:29:05 mounttable ... ok 09:29:05 application repository ... ok 09:39:05 Waiting for command to exit: "/usr/local/google/home/veyron/workspace/vanadium-create-instance-test/root/release/go/bin/vrpc" 09:39:06 binary repository ... failed 09:39:06 09:39:07 macaroon service ... ok 09:39:07 google identity service ... ok 09:39:07 binary discharger ... ok 09:39:07 proxy service ... ok 09:39:07 some checks failed: 09:39:07 some checks failed 09:39:07 09:39:07 ##### FAILED #####