Hi expert,
Kernel based vRouter support live migration, while DPDK based vRouter doesn't support live migration.
I'm using Redhat OSP10+Contrail 4.1.1
while use DPDK vRouter, the VM's live migration operation always get failure result.
The failure log is displayed on destionation host's message:
Jan 11 03:53:05 overcloud-contraildpdk-1 libvirtd: 2019-01-11 03:53:05.691+0000: 59541: warning : qemuDomainObjBeginJobInternal:4709 : Cannot start job (query, none) for domain instance-00000004; current job is (none, migration in) owned by (0 , 0 remoteDispatchDomainMigratePrepare3Params) for (0s, 89s)
Jan 11 03:53:05 overcloud-contraildpdk-1 libvirtd: 2019-01-11 03:53:05.691+0000: 59541: error : qemuDomainObjBeginJobInternal:4721 : Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainMigratePrepare3Params)
below is qemu debug information
Thread 8 (Thread 0x7f0d50ab2700 (LWP 84881)):
0 0x00007f0d79177bfd in recvmsg () from /lib64/libpthread.so.0
1 0x000055c1399cf18c in qio_channel_socket_readv ()
2 0x000055c1399c05e2 in tcp_chr_recv ()
3 0x000055c1399c1da1 in tcp_chr_sync_read ()
4 0x000055c1399bc761 in qemu_chr_fe_read_all ()
5 0x000055c13976a15b in vhost_user_read.isra.6 ()
6 0x000055c13976a776 in vhost_user_set_log_base ()
7 0x000055c139766eed in vhost_log_global_start ()
8 0x000055c13971726e in memory_global_dirty_log_start ()
9 0x000055c13971de76 in ram_save_setup ()
10 0x000055c1398e4f95 in qemu_savevm_state_setup ()
11 0x000055c1398e0d8f in migration_thread ()
12 0x00007f0d79170dd5 in start_thread () from /lib64/libpthread.so.0
13 0x00007f0d78e9ab3d in clone () from /lib64/libc.so.6
Redhat guy said "qemu call recvmsg in qio_channel_socket_readv and keep waiting for vrouter's response, but vrouter doesn't give any response"
So please help to identify this issue, thank you very much!
Hi expert, Kernel based vRouter support live migration, while DPDK based vRouter doesn't support live migration. I'm using Redhat OSP10+Contrail 4.1.1 while use DPDK vRouter, the VM's live migration operation always get failure result.
The failure log is displayed on destionation host's message: Jan 11 03:53:05 overcloud-contraildpdk-1 libvirtd: 2019-01-11 03:53:05.691+0000: 59541: warning : qemuDomainObjBeginJobInternal:4709 : Cannot start job (query, none) for domain instance-00000004; current job is (none, migration in) owned by (0, 0 remoteDispatchDomainMigratePrepare3Params) for (0s, 89s)
Jan 11 03:53:05 overcloud-contraildpdk-1 libvirtd: 2019-01-11 03:53:05.691+0000: 59541: error : qemuDomainObjBeginJobInternal:4721 : Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainMigratePrepare3Params)
below is qemu debug information Thread 8 (Thread 0x7f0d50ab2700 (LWP 84881)):
0 0x00007f0d79177bfd in recvmsg () from /lib64/libpthread.so.0
1 0x000055c1399cf18c in qio_channel_socket_readv ()
2 0x000055c1399c05e2 in tcp_chr_recv ()
3 0x000055c1399c1da1 in tcp_chr_sync_read ()
4 0x000055c1399bc761 in qemu_chr_fe_read_all ()
5 0x000055c13976a15b in vhost_user_read.isra.6 ()
6 0x000055c13976a776 in vhost_user_set_log_base ()
7 0x000055c139766eed in vhost_log_global_start ()
8 0x000055c13971726e in memory_global_dirty_log_start ()
9 0x000055c13971de76 in ram_save_setup ()
10 0x000055c1398e4f95 in qemu_savevm_state_setup ()
11 0x000055c1398e0d8f in migration_thread ()
12 0x00007f0d79170dd5 in start_thread () from /lib64/libpthread.so.0
13 0x00007f0d78e9ab3d in clone () from /lib64/libc.so.6
Redhat guy said "qemu call recvmsg in qio_channel_socket_readv and keep waiting for vrouter's response, but vrouter doesn't give any response"
So please help to identify this issue, thank you very much!