Closed phol closed 5 months ago
Can you try your copy again but with --mode=relay
?
When you run lxc copy av j:av
, it will have your local Incus generate a token which is then sent by the CLI to the j
host, instructing that host to directly connect to your local system to retrieve the instance.
The fact that the target connects to the source is why you often will have no problem interacting with both servers yourself but a copy may fail due to network issues going the other way.
--mode=relay
has the CLI tool itself connect to both source and target and relay the data, so that should usually work fine if your CLI was already able to interact with both servers.
You also have the option of using --mode=push
which instead has the source server push directly to the target server.
Hi @stgraber and thanks for replying so quickly.
With
lxc copy av j:av --mode=relay
I'm getting
Error: Error transferring instance data: Failed migration on target: Error reading migration control source: websocket: close 1006 (abnormal closure): unexpected EOF
During the copy, I do observe a btrfs receive
command in htop
. However, after about 1m30s, I get the error. I tried this multiple times. During the copy, I can also observe a av
instance on j
when running lxc ls
.
with
lxc copy av j:av --mode=push
I'm getting this message:
Transferring instance: av: 1.32GB (18.71MB/s)
The "transferring instance" is not shown when I specify --mode=relay
or nothing at all.
However, it unfortunately also errors out. With all three modes, this happens after around 1m30s.
time lxc copy av j:av --mode=push
Error: Failed instance migration: Failed migration on source: migration dump failed
(00.011761) Error (criu/namespaces.c:460): Can't dump nested uts namespace for 4738
(00.011764) Error (criu/namespaces.c:721): Can't make utsns id
(00.018309) Error (criu/util.c:642): exited, status=1
(00.021460) Error (criu/util.c:642): exited, status=1
(00.022562) Error (criu/cr-dump.c:2098): Dumping FAILED.
incus copy av j:av --mode=push 0.09s user 0.02s system 0% cpu 1:24.65 total
Ah, please stop the container first or pass --stateless as live migrations of containers are very unlikely to succeed.
Alright, I'm running this now. Will update you when it completes.
time lxc copy av j:av --mode=push --stateless
I tried the commands.
lxc copy av j:av --mode=push --stateless
Works.
lxc copy av j:av --mode=relay --stateless
lxc copy av j:av --stateless
Work as well but don't show progress. Do you have any idea why this might happen?
Also, about the --stateless
flag: Would it perhaps be possible to force the --stateless
flag when copying across containers between hosts with differing processor architectures?
Hmm, I've been unable to reproduce the migration error issue here:
root@v1:~# incus copy u1 v2:u1
Error: Failed instance creation: Error transferring instance data: Failed migration on target: Error from migration control source: Failed migration on source: migration dump failed
(00.003977) Error (criu/namespaces.c:460): Can't dump nested uts namespace for 3850
(00.003979) Error (criu/namespaces.c:721): Can't make utsns id
(00.007038) Error (criu/util.c:642): exited, status=1
(00.007842) Error (criu/util.c:642): exited, status=1
(00.008333) Error (criu/cr-dump.c:2098): Dumping FAILED.
root@v1:~# incus copy u1 v2:u1 --mode=relay
Error: Error transferring instance data: Failed migration on target: Error from migration control source: Failed migration on source: migration dump failed
(00.003417) Error (criu/namespaces.c:460): Can't dump nested uts namespace for 3850
(00.003418) Error (criu/namespaces.c:721): Can't make utsns id
(00.005060) Error (criu/util.c:642): exited, status=1
(00.005759) Error (criu/util.c:642): exited, status=1
(00.006247) Error (criu/cr-dump.c:2098): Dumping FAILED.
root@v1:~# incus copy u1 v2:u1 --mode=push
Error: Failed instance migration: Failed migration on source: migration dump failed
(00.003274) Error (criu/namespaces.c:460): Can't dump nested uts namespace for 3850
(00.003276) Error (criu/namespaces.c:721): Can't make utsns id
(00.005401) Error (criu/util.c:642): exited, status=1
(00.006276) Error (criu/util.c:642): exited, status=1
(00.006805) Error (criu/cr-dump.c:2098): Dumping FAILED.
root@v1:~#
Then with stateless:
root@v1:~# incus copy u1 v2:u1 --stateless && incus delete -f v2:u1
root@v1:~# incus copy u1 v2:u1 --stateless --mode=relay && incus delete -f v2:u1
root@v1:~# incus copy u1 v2:u1 --stateless --mode=push && incus delete -f v2:u1
All of those showed transfer progress information as expected.
In my case, this was with Incus 0.7 on source and target server, CLI is Incus 0.7 too and storage on both source and target is basic dir
backend.
Moving the issue back to Incomplete and un-milestone until I can reproduce an issue with either the error handling or transfer progress.
I was running both machines with incus 0.7, but with the btrfs storage backend on both.
Can I be of any help by e.g. retrying with debugging flags or something like that?
I'll retry with btrfs see if that makes it behave differently here.
Errors still propagate correctly but the progress information is indeed missing, so that's a btrfs driver issue then.
Alright, great. I'm happy to hear I was able to report a bug which helps to improve the project and I wasn't wasting your time. Also, wow, I don't think I've ever seen a bug to be resolved as quickly as this one!
Thank you for all your efforts and good night :)
I don't think it's specific to the btrfs driver only. This affects also the zfs driver: using --mode pull --refresh
does not show any progress when running on the target host.
I also get the error mentioned in the issue description on some containers when copying over WAN network thru a tunnel:
Error: Failed instance creation: Error transferring instance data: Failed migration on target: Error reading migration control source: local error: tls: bad record MAC
rerunning the command later may be successfull.
full command used:
incus copy $host:$container $container --mode pull --refresh --project $project --target-project=$project
on source target server runs: incus 6.0.0
Required information
Incus info of local machine
Incus info of remote
Issue description
(I'm using
alias lxc = incus
)Adding a remote and then executing
lxc list
orlxc exec
from local to the remote works properly.This also does seem to execute, but at the end returns
Steps to reproduce
Information to attach
I think the error might either be related to the differing processor architectures, or are related to the fact that the machines, which are run on Oracle cloud free tier, are doing something strange with the network interface MAC.
On both machines,
br0
provides networking for the containers.Network config local machine
enp0s6
is the primary network interface with DHCP IP10.0.0.136
Network config remote
ens3
is the primary network interface with DHCP IP10.0.0.45