rework iscsi tests architecture

[dustymabe]

Right now we have two iscsi tests (and will soon have a 3rd) that all:

1. start a VM
2. pull down/start a targetd container
3. Run coreos-installer to do an install with iscsi kargs set
4. pull down coreos-assembler container to:
5. spin up a nested VM using iPXE with sanboot options set

This makes for an extremely heavyweight test. It downloads two containers (cosa being a really large one) and relies on nested virtualization, which apparently doesn't work on multiarch.

I suggest that we re-architect this such that we

• start one VM to run targetd
  • needs to be a VM and not just a process started by COSA because targetd requires a kernel module.
• start other VMs (not nested) to run the individual tests
  • they will talk to the initially started VM
    • of course, qemu user mode networking presents a challenge here, but I think we can handle it

The idea here is that if a single one of the tests are called they will spin up the initial VM and run the test, but if multiple of the tests are called then the initial VM will only need to be setup once.

Benefits of this approach:

• pulls targetd container once per test run (versus 3 times)
• does not pull coreos-assembler container at all
  • this also was problematic anyway because the cosa container we were pulling could be a different version than the COSA we are initiating this whole test from
• does not rely on nested virtualization

One potential model we could follow for this is what is currently done for the luks/tang tests in:

https://github.com/coreos/coreos-assembler/blob/79b15c89d57a4b97334c70881b036aee3462f1e4/mantle/kola/tests/ignition/luks.go#L124

[jbtrystram]

Note : we'd need some code to cleanup the iscsi volume between test to avoid coreos-installer trying to install on a non-empty disk. it may be a non-issue, i'm not sure, it's worth checking.

[dustymabe]

Installing to a non-empty disk should be a non-issue (coreos-installer will just happily overwrite what is there), but I'd be worried about false positives where the install from the previous test somehow was used for the next test, so I agree that it would be nice to start from a fresh disk each time.