make sure the rust example in testground examples works

[laurentsenta]

At the moment, trying to run the example, I get a sidecar error (on local machine, ubuntu server, stagin ipdx)

Apr 29 12:27:29.523437  INFO   0.7505s      OTHER << sidecar      >> Apr 29 12:27:29.552671   INFO   resolved route to host  {"host": "testground-redis", "ip": "192.18.0.3"}
Apr 29 12:27:29.524434  INFO   started containers  {"runner": "local:docker", "run_id": "c9ltj8lvqc7ik10ovcm0", "count": 2}
Apr 29 12:27:29.524517  INFO   0.7515s      OTHER << sidecar      >> Apr 29 12:27:29.553279   INFO   resolved route to host  {"host": "testground-sync-service", "ip": "192.18.0.4"}
Apr 29 12:27:29.525696  INFO   0.7527s      OTHER << sidecar      >> Apr 29 12:27:29.553979   INFO   resolved route to host  {"host": "testground-influxdb", "ip": "192.18.0.5"}
Apr 29 12:27:29.526080  INFO   0.7814s    MESSAGE << sidecar      >> InfluxDB unavailable; no metrics will be dispatched: no InfluxDB URL in $INFLUXDB_URL env var
Apr 29 12:27:29.530401  INFO   0.7574s      OTHER << sidecar      >> Apr 29 12:27:29.557732   INFO   successfully resolved route to host {"container_id": "4e323101ef3fd0775d85219145752028df36e4bc2f5fbd2cbdf214a54cd0ef7d"}
Apr 29 12:27:29.530480  INFO   0.7575s      OTHER << sidecar      >> Apr 29 12:27:29.559004   INFO   external routing disabled
Apr 29 12:27:29.530517  INFO   0.7576s      OTHER << sidecar      >> Apr 29 12:27:29.559072   INFO   waiting for all networks to be ready    {"sidecar": true, "run_id": "c9ltj8lvqc7ik10ovcm0"}
Apr 29 12:27:29.530620  INFO   0.7866s    MESSAGE << sidecar      >> InfluxDB unavailable; no metrics will be dispatched: no InfluxDB URL in $INFLUXDB_URL env var
Apr 29 12:27:29.533352  INFO   0.7604s      OTHER << sidecar      >> Apr 29 12:27:29.562208   ERROR  sidecar worker failed: failed to initialise the container: failed to lookup the net namespace: no such file or directory    {"host": "unix:///var/run/docker.sock", "container": "39e959e903e1636d6c6ed0befc7bc67bfc1dea0762138e170c03df08f3352d44"}
Apr 29 12:27:29.576200  INFO   0.8032s      OTHER << sidecar      >> Apr 29 12:27:29.605182   ERROR  sidecar worker failed: container worker failed: failed to signal network ready: channel closed before getting response  {"host": "unix:///var/run/docker.sock", "container": "4e323101ef3fd0775d85219145752028df36e4bc2f5fbd2cbdf214a54cd0ef7d"}
Apr 29 12:27:29.578846  INFO   0.8059s INCOMPLETE << single[001] (4e3231) >> 
Apr 29 12:27:29.578871  INFO   0.8059s INCOMPLETE << single[000] (39e959) >> 
Apr 29 12:27:29.579152  INFO   deleting containers {"runner": "local:docker", "run_id": "c9ltj8lvqc7ik10ovcm0", "ids": ["39e959e903e1636d6c6ed0befc7bc67bfc1dea0762138e170c03df08f3352d44", "4e323101ef3fd0775d85219145752028df36e4bc2f5fbd2cbdf214a54cd0ef7d"]}
Apr 29 12:27:29.579228  INFO   deleting container  {"runner": "local:docker", "run_id": "c9ltj8lvqc7ik10ovcm0", "id": "4e323101ef3fd0775d85219145752028df36e4bc2f5fbd2cbdf214a54cd0ef7d"}
Apr 29 12:27:29.579237  INFO   deleting container  {"runner": "local:docker", "run_id": "c9ltj8lvqc7ik10ovcm0", "id": "39e959e903e1636d6c6ed0befc7bc67bfc1dea0762138e170c03df08f3352d44"}
Apr 29 12:27:29.890251  WARN   run finished in error   {"run_id": "c9ltj8lvqc7ik10ovcm0", "plan": "example-rust", "case": "tcp-connect", "runner": "local:docker", "instances": 2, "error": "2 nodes failed"}

Something is broken with the sidecar, using regular example and this branch: https://github.com/testground/testground/tree/devx/improve-rust-tests (current commit: https://github.com/testground/testground/commit/cd1f83a6c8796ebd75d9040257a3ba3a711262c1)

Taking some time to figure out how to run debug tools with the sidecar (goland -> docker -> sidecar ran by testground)

[laurentsenta]

Debugging:

• change the Dockerfile.sidecar to start with dlv (inspiration from https://blog.jetbrains.com/go/2020/05/06/debugging-a-go-application-inside-a-docker-container/)
• change the local_docker.go runner to pass additional parameters (app armor, sys_ptrace), port binding, remove the entry point parameter (keep dlv),

see the commit: https://github.com/testground/testground/commit/51cfc8bbc125903974defc0aecaf737f7e84c314

• build again, make docker-sidecar && make doinstall && testground daemon

Run healthcheck to restart container:

testground healthcheck --runner local:docker --fix

The port mapping doesn't always work, but you can always docker inspect testground-sidecar, find the port and connect to it from the IDE.

[laurentsenta]

debug containers built & run with local docker:

--run-cfg keep_containers=true