Open balopat opened 7 years ago
@aslo @nkubala FYI, let me know what you guys think.
Haha, good timing on this one: I'm in the middle of a major redesign of the structure tests as we speak, which will pull the test execution outside of a container. Check out https://github.com/GoogleCloudPlatform/runtimes-common/pull/332 for progress
awesome @nkubala, let's see how that shakes out, it looks promising! Also it seems that based on our conversation the container-builder-local might have support in the future to define workspace as a directory vs the sandbox volume only - that would help as well in our case (b/65417396)
Great work on this analysis! I @nkubala 's change looks promising.
I think we might need a bit of a redesign here, couldn't solve it quickly, so I'll document the problems I see.
Currently in our build we have the following participants:
structure_test.sh
scriptstructure_test.sh
downloadsext_run.sh
- which was seemingly designed to run from a docker-enabled host directly, not a container - and runs itext_run.sh
spins up a "sibling-container" (as it shares now the daemon with the "step-container") from the built openjdk imageVolumes and directories:
GCCB:
/workspace
contains the project directory (passed in)/workspace
is a volume mounted from the/workspace
directory from host (NOTE: in the future this might change to a separate, isolated volume, that is the clone of the /workspace directory so that the steps don't mutate the host filesystem!!)/cfg
- that is mounted with-v /workspace/openjdk8/target/classes/.cfg:/cfg
in the case of openjdk8/workspace
- that is coming from the--workspace
argument ofext_run.sh
:--workspace /workspace/openjdk8/target/test-classes/openjdk-test-common-out/workspace
for exampleThe problematic part is that both sibling-container volumes work currently because of the coincidental design overlap - host
\workspace
is directly mounted onto\workspace
in the step-container. As soon as this symmetry breaks we have issues. The reason is that the "sibling-container" is talking to the docker daemon on the host and as such resolves directory based docker volumes from the host.To see where this reliance already causes problem, let's have a look at the same directories in the container-builder-local case:
/home/balopat/dev/proj/openjdk-runtime
contains the project directory (passed in)/workspace
is a cloned volume mounted from the/home/balopat/dev/proj/openjdk-runtime
directory (NOTE: this is intentionally designed like this and it might be the future on GCE too!)/cfg
- that is mounted with-v /workspace/openjdk8/target/classes/.cfg:/cfg
in the case of openjdk8 ==> ERROR, the directory is empty/workspace
- that is coming from the--workspace
argument ofext_run.sh
:--workspace /workspace/openjdk8/target/test-classes/openjdk-test-common-out/workspace
for example ==> ERROR, the directory is emptyStructure tests fail with this setup.
The problem:
Not exactly sure about the best way to go. Couple of options:
ext_run.sh
to prepare it for running from within a container and pass in explicitly the right volumes