Closed vjdhama closed 5 years ago
On further investigation, was able to reproduce the issue using virsh.
For below XML
<network>
<name>default</name>
<forward mode="bridge"/>
<bridge name='virbr0'/>
</network>
virsh net-define net.xml
Network default defined from net.xml
root@svr02:~# virsh net-list --all
Name State Autostart Persistent
----------------------------------------------------------
default inactive no yes
virsh net-create test-net.xml
error: Failed to create network from test-net.xml
error: operation failed: network 'default' already exists with uuid fcfcfe24-9884-4bec-9a98-005a0027964f
The flow IMO, should be
virsh net-define net.xml
and then virsh net-start default
, instead on virsh net-define net.xml
and virsh net-create net.xml
.
Relevent code : https://github.com/dmacvicar/terraform-provider-libvirt/blob/e7b65f1425580d6dfa542c884cdbf1863706ea27/libvirt/resource_libvirt_network.go#L296
thx @vjdhama for issue, i will tag it a need_investigation atm i"m on other issues. :+1:
@vjdhama i will look at it once i have free cycles. I think i know the problem thx for you issue
@MalloZup Thanks for looking into this.
@vjdhama just some updates on your issue https://github.com/dmacvicar/terraform-provider-libvirt/issues/389.
Shortly is because for network_resources
we don't update the resources.
So if you have your bridge which is already existing you will have always that error.
As workaround you should create/destroy bridges.
For fixing this we need to look at https://godoc.org/github.com/libvirt/libvirt-go#Network.Update
and particulary here we shold implement the update call for the bridge part of networking, and update the resouce correctly.
Is there a workaround to have a VM directly connected to a bridge (and thus to the LAN)? Or can someone elaborate this:
As workaround you should create/destroy bridges.
further? Which bridge, the libvirt or my br0
on the host?
I want my VMs to have IPs in my home network, to access them directly.
Thanks
@tommyknows as workaround you can cut the bridge
generation via terraform.
So basically like this example here: https://github.com/dmacvicar/terraform-provider-libvirt/blob/master/examples/ubuntu/ubuntu-example.tf#L41
We don't have any network/bridge creation in terraform, you just attach the domain to existing ones and the creation should not be part of the TF file.
The comment above means:
if you create a br
via terraform-libvirt currently you can create it only 1 time.
At moment the codebase is so that if you do an apply 2 times, you will have problem as posted in this issue.
So the best solution is to don't create network/bridge via terraform-libvirt.
You can still specify the domain external to tf network here: https://github.com/dmacvicar/terraform-provider-libvirt/blob/master/website/docs/r/domain.html.markdown#handling-network-interfaces
hope it helps :+1: :white_flower:
Thanks for the reply. I'm just getting started with TF (and KVM kinda too). I want to set up a CoreOS VM on my ubuntu system. Currently, I have the following tf config:
provider "libvirt" {
uri = "qemu:///system"
}
resource "libvirt_ignition" "k8s-ignition" {
name = "kubernetes.ign"
content = "/home/ramon/terraform/definitions/k8s.ign"
}
resource "libvirt_domain" "kubernetes" {
name = "kubernetes-terraform"
memory = "1024"
vcpu = 2
coreos_ignition = "${libvirt_ignition.k8s-ignition.id}"
network_interface {
bridge = "br0"
addresses = ["192.168.1.187"]
}
boot_device {
dev = [ "hd", "network"]
}
console {
type = "pty"
target_port = "0"
target_type = "serial"
}
console {
type = "pty"
target_type = "virtio"
target_port = "1"
}
disk {
volume_id = "${libvirt_volume.coreos.id}"
}
graphics {
type = "spice"
listen_type = "address"
autoport = true
}
}
resource "libvirt_volume" "coreos" {
name = "coreos"
pool = "default"
source = "/home/ramon/terraform/images/coreos_production_qemu_image.img"
format = "qcow2"
}
However, I cannot access my host on .187
.
When using a NATed network, it works fine.
Now, if I cannot set an IP Address for the host (because qemu-guest-agent
is not installed on coreOS?), how can I find the IP Address?
ip a
doesn't show a new IP Address either.
Setting the wait_for_lease
just waits forever.
Thanks for your help.
yop, in bridge mode you need the qemu-guest-agent
installed on the Domain. Afaik this is the only solution and yop wait_for_lease
wait forever because we cannot get IP without the qemu-agent
So there's no way to have CoreOS hosts in bridged mode then? :/
@tommyknows you could use the cloud_init
for installing the qemu-guest-agent
pkg for the CoreOS and have it in bridge
mode
https://github.com/dmacvicar/terraform-provider-libvirt/tree/master/examples/ubuntu
but there's no networking, right? It's not that I just can't inspect it with KVM, the guest does not have any kind of connection (?). -> Means I'd need to copy a file onto the host by sharing a volume.
(And "installing" in CoreOS would mean running a docker container, although I'm going to try this.
agh yes i think then that coreos doesn\t have cloudinit in that case :smile: but i think you could be in a good directoin in that way :)
@tommyknows , i have the same issue: I want to install qemu-guest-agent on CoreOS. Do you have found a solution?
I started working with RancherOS, as they provide a qemu-guest-agent docker image. I guess you should be able to get this working on CoreOS too. Basically, what I did:
1) Download the docker image on your host:
docker pull docker.io/rancher/os-qemuguestagent:v1.4.0-rc1
2) Export the docker image to a .tar
file:
docker export rancher/os-qemuguestagent:v1.4.0-rc1 -o qemu-guest-agent.tar
3) Add the folder as a mount to your Terraform VMs (inside the libvirt_domain
resource):
filesystem {
source = "/media/terraform/images"
target = "qemu_docker_image"
readonly = true
}
And mount the folder on the host via cloudinit, ignition or just run it as a command. This here is displayed as a mount resource in cloudinit:
- - qemu_docker_image
- /media/images
- 9p
- trans=virtio,version=9p2000.L,rw
4) Import the docker image on the host:
docker load -i /media/images/qemu-guest-agent.tar
5) Now that you've got the image on the host, you need to start a container with it. I am using the RancherOS Service template, so I did not have to figure out the options for the container. The RancherOS Service template looks like this:
qemu-guest-agent:
image: rancher/os-qemuguestagent:v1.4.0-rc1
command: ["/usr/bin/qemu-ga"]
privileged: true
restart: always
labels:
io.rancher.os.scope: system
pid: host
ipc: host
net: host
uts: host
volumes_from:
- command-volumes
volumes:
- /dev:/host/dev
If needed, I could provide the output of a docker inspect
on the running container.
Awesome hint! Thanks for this. Really interesting workflow!
Hey ... here is a part of my CoreOS setup like what @tommyknows was done:
data "ignition_systemd_unit" "dockerimages-mount" {
name = "images.mount"
content = "${file("${path.module}/../ignition/mount-images.cli")}"
}
data "ignition_systemd_unit" "qemuagent" {
name = "qemuagent.service"
content = "${file("${path.module}/../ignition/qemuagent.cli")}"
}
data "ignition_config" "master" {
systemd = [
"${data.ignition_systemd_unit.dockerimages-mount.id}",
"${data.ignition_systemd_unit.qemuagent.id}",
]
}
mount-images.cli
[Unit]
Before=local-fs.target
[Mount]
What=qemu_docker_image
Where=/images
Options=ro,trans=virtio,version=9p2000.L
Type=9p
[Install]
WantedBy=local-fs.target
The mount works, but not the run of the qemu-agent container. Try to solve it ...
UPDATE: I tried with this:
[Unit]
Description=QEMU Agent
After=docker.service
[Service]
ExecStartPre=/usr/bin/docker load -i /images/qemu-guest-agent.tar
ExecStart=/usr/bin/docker run \
--privileged=true \
--cap-add=ALL \
--net=host \
-e container=1 \
-e HOST=/host \
-e TERM=xterm \
-v /dev/virtio-ports:/dev/virtio-ports \
-v /etc/os-release:/etc/os-release:ro \
-v /dev:/dev \
-v /proc:/hostproc \
-v /run/systemd:/run/systemd \
-v /var/log/qemu-ga:/var/log/qemu-ga:rw \
-v /var/run/docker.sock:/var/run/docker.sock:ro \
rancher/os-qemuguestagent:v1.4.0-rc1
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.target
But this doesn't run correct.
/usr/bin/ros: no such file or directory": unknown.
. So, this container runs only on RangerOS ?! :)
Mh I'm not sure - i think it could just be the entrypoint of the container. What happens if you change the entrypoint (the command that the container is started with) to /usr/bin/qemu-ga
?
Or am I missing something?
This issue got mixed. @tommyknows feel free to open an issue, I will fix the bridge net problem reported here. 1st post
this was fixed by https://github.com/dmacvicar/terraform-provider-libvirt/pull/531 which will merged on this week on master.
Thx for all infos and comments. :sun_behind_large_cloud:
Version Reports:
Distro version of host:
Ubuntu 18.04
Terraform Version Report
Libvirt version
terraform-provider-libvirt plugin version (git-hash)
Description of Issue/Question
By default virsh allows creation on a libvirt network on existing host bridge. https://libvirt.org/formatnetwork.html#examplesBridge
That works if you create libvirt network with virsh.
Using terraform it throws
But the new network is created anyway.
Also rerunning
terraform apply
fails withsince the original run fails to collect metadata, id of network.
Setup
(Please provide the full main.tf file for reproducing the issue (Be sure to remove sensitive info)
Steps to Reproduce Issue
(Include debug logs if possible and relevant.)
terraform init terraform plan
Additional Infos:
Do you have SELinux or Apparmor/Firewall enabled? Some special configuration? Have you tried to reproduce the issue without them enabled?