XenServer Tools is unable to report CoreOS' VM IP address to parent XenServer

[sanmigit]

Issue Report

XenServer Tools is unable to report CoreOS' VM IP address to the XenServer host: Permission Denied.

Bug

CoreOS VM's XenServer Tools agent fails to register itself with host XenServer. This means there is no IP address shown in Networking tab, resulting in inability to enable Docker management (along with any other functionality, when a connection from XenServer to VM is required), due to XenServer being unable to connect to host VM.

Container Linux Version

NAME="Container Linux by CoreOS" ID=coreos VERSION=2079.6.0 VERSION_ID=2079.6.0 BUILD_ID=2019-06-11-0821 PRETTY_NAME="Container Linux by CoreOS 2079.6.0 (Rhyolite)" ANSI_COLOR="38;5;75" HOME_URL="https://coreos.com/" BUG_REPORT_URL="https://issues.coreos.com" COREOS_BOARD="amd64-usr"

Environment

Xenserver 7.2 Updates to E017 Container Management 1.0

Expected Behavior

CoreOS boots XenServer Tools agent starts - no errors XenCenter shows 'Virtualization state: Optimized (version 7.2 installed)' in VM's General tab XenCenter shows IP address(es) related to VM in Networking tab XenCenter is able to turn on Docker management via an SSH connection

Actual Behavior

CoreOS boots XenServer Tools agent starts - no errors XenCenter shows 'Virtualization state: Optimized (version 7.1 installed)' in VM's General tab XenCenter missing IP address(es) related to VM in Networking tab XenCenter is not able to turn on Docker management via an SSH connection. Failing due to missing IP address.

Reproduction Steps

1.Boot CoreOS from ISO image with a default cloud-config 2.Open XenCenter, navigate to VM's Networking tab

Other Information

cloud-config hostname: %VMNAMETOHOSTNAME% ssh_authorized_keys:

• ssh-rsa AAAAB3NzaC1yc2EAAAADAQblahblahblah

  The following entry will automatically be replaced with a public key

  generated by container management plugin. The key-entry must exist,

  in order to enable container management for this VM.

• ssh-rsa %CONTAINERRSAPUB% coreos: units:
• name: etcd-member.service command: start %HINEXISTS%

  Avoid routing via host internal networks

• name: 00-eth%HIN%.network runtime: true content: | [Match] Name=eth%HIN% [Network] DHCP=yes [DHCP] UseRoutes=false%ENDHINEXISTS%

  Hypervisor Linux Guest Agent

• name: xe-linux-distribution.service command: start content: | [Unit] Description=Hypervisor Linux Guest Agent After=docker.service [Service] ExecStartPre=/media/configdrive/agent/xe-linux-distribution /var/cache/xe-linux-distribution ExecStart=/media/configdrive/agent/xe-daemon etcd: name: %VMNAMETOHOSTNAME%

  generate a new token for each unique cluster from https://discovery.etcd.io/new?size=3

  specify the initial cluster size using ?size=X

  discovery: "https://discovery.etcd.io/"

  write_files:

  • path: /etc/sysctl.d/10-enable-arp-notify.conf permissions: 0644 owner: root content: | net.ipv4.conf.all.arp_notify = 1

    Log

    Container Linux by CoreOS stable (2079.6.0) core@coreos ~ $ strace -tt -T -s 1024 -f xenstore read name 14:08:36.504072 execve("/usr/bin/xenstore", ["xenstore", "read", "name"], [/ 27 vars /]) = 0 <0.000299> 14:08:36.504712 brk(0) = 0x564a659ea000 <0.000129> 14:08:36.505041 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) <0.000114> 14:08:36.505361 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 <0.000102> 14:08:36.505656 fstat(3, {st_mode=S_IFREG|0644, st_size=31463, ...}) = 0 <0.000110> 14:08:36.505969 mmap(NULL, 31463, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f6deb34d000 <0.000105> 14:08:36.506258 close(3) = 0 <0.000129> 14:08:36.506553 openat(AT_FDCWD, "/lib64/libxenstore.so.3.0", O_RDONLY|OCLOEXEC) = 3 <0.000154> 14:08:36.506886 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220 \0\0\0\0\0\0@\0\0\0\0\0\0\0\370p\0\0\0\0\0\0\0\0\0\0@\0008\0\6\0@\0\31\0\30\0\1\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\310^\0\0\0\0\0\0\310^\0\0\0\0\0\0\0\0 \0\0\0\0\0\1\0\0\0\6\0\0\0\340j\0\0\0\0\0\0\340j \0\0\0\0\0\340j \0\0\0\0\0(\5\0\0\0\0\0\0h5\0\0\0\0\0\0\0\0 \0\0\0\0\0\2\0\0\0\6\0\0\0\360k\0\0\0\0\0\0\360k \0\0\0\0\0\360k \0\0\0\0\0\300\1\0\0\0\0\0\0\300\1\0\0\0\0\0\0\10\0\0\0\0\0\0\0P\345td\4\0\0\0000R\0\0\0\0\0\0000R\0\0\0\0\0\0000R\0\0\0\0\0\0\344\1\0\0\0\0\0\0\344\1\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\0\340j\0\0\0\0\0\0\340j \0\0\0\0\0\340j \0\0\0\0\0 \5\0\0\0\0\0\0 \5\0\0\0\0\0\0\1\0\0\0\0\0\0\0%\0\0\0006\0\0\0\10\0\0\0\t\0\0\0\303\24\204\20 \2J\1\2\fP\0\0\22\200\23\0\0 \0\0\0\0\1\322\0\1\200-\30\10\0\210\fLL\2 \10\200K\0\204\0\4B \0D\1\0\22@ \3\211 8@\1\21\6\22\t6\0\0\0\0\0\0\0007\0\0\0009\0\0\0;\0\0\0<\0\0\0>\0\0\0\0\0\0\0\0\0\0\0?\0\0\0\0\0\0\0@\0\0\0A\0\0\0C\0\0\0\0\0\0\0F\0\0\0H\0\0\0\0\0\0\0I\0\0\0K\0\0\0M\0\0\0\0\0\0\0N\0\0\0O\0\0\0P\0\0\0S\0\0\0\0\0\0\0U\0\0\0V\0\0\0X\0\0\0Z\0\0\0\\0\0\0\0\0\0\0\0\0\0\0\0\0c\0\0\0f\0\0\0\r\342w4\314A\254\26\33'<\265\300\16${\305X-US\323\376<F\0013\265\353\323\357\16\23\200:\364\3\217\307\3433\322\207\327 =\373ZK\330\332\354|\224\367\332\300\300|\226\307D\203\264@\224\353\2077\256\224YAko\5\n\177S\33\305IF~\342\346\6AwX\234\207\271\252k5\7\230-\354\2353F\306\26\25\333b\304\f\205\351\331qX\34\0008C,\273\343\222|\v\360\201\273$\f\317\263\353Xw[\344X\32\362\365\231\3479BE\325\354ir\214R\202\0210?\320>\230&\255\341\267\7\313\255D,\360wc0\276M\213\"U\2509^x\254\377u\22\347\3\267\271\215\361\16\32[\233\220o/\273\20\0\0\0\0", 832) = 832 <0.000129> 14:08:36.507453 fstat(3, {st_mode=S_IFREG|0755, st_size=30520, ...}) = 0 <0.000087> 14:08:36.507751 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6deb34b000 <0.000099> 14:08:36.508019 mmap(NULL, 2138184, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6deaf24000 <0.000097> 14:08:36.508266 mprotect(0x7f6deaf2a000, 2097152, PROT_NONE) = 0 <0.000167> 14:08:36.508594 mmap(0x7f6deb12a000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f6deb12a000 <0.000137> 14:08:36.508921 mmap(0x7f6deb12c000, 8264, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f6deb12c000 <0.000128> 14:08:36.509212 close(3) = 0 <0.000098> 14:08:36.509471 openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 <0.000105> 14:08:36.509741 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\34\2\0\0\0\0\0@\0\0\0\0\0\0\0\240\244\34\0\0\0\0\0\0\0\0\0@\0008\0\n\0@\0D\0C\0\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0000\2\0\0\0\0\0\0000\2\0\0\0\0\0\0\10\0\0\0\0\0\0\0\3\0\0\0\4\0\0\0\340\307\31\0\0\0\0\0\340\307\31\0\0\0\0\0\340\307\31\0\0\0\0\0\34\0\0\0\0\0\0\0\34\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0\1\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0359\34\0\0\0\0\0\0359\34\0\0\0\0\0\0\0 \0\0\0\0\0\1\0\0\0\6\0\0\0\230D\34\0\0\0\0\0\230D<\0\0\0\0\0\230D<\0\0\0\0\0(R\0\0\0\0\0\0\250\224\0\0\0\0\0\0\0\0 \0\0\0\0\0\2\0\0\0\6\0\0\0\0z\34\0\0\0\0\0\0z<\0\0\0\0\0\0z<\0\0\0\0\0\340\1\0\0\0\0\0\0\340\1\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0p\2\0\0\0\0\0\0p\2\0\0\0\0\0\0p\2\0\0\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\7\0\0\0\4\0\0\0\230D\34\0\0\0\0\0\230D<\0\0\0\0\0\230D<\0\0\0\0\0\20\0\0\0\0\0\0\0\220\0\0\0\0\0\0\0\10\0\0\0\0\0\0\0P\345td\4\0\0\0\374\307\31\0\0\0\0\0\374\307\31\0\0\0\0\0\374\307\31\0\0\0\0\0\354^\0\0\0\0\0\0\354^\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\0\230D\34\0\0\0\0\0\230D<\0\0\0\0\0\230D<\0\0\0\0\0h;\0\0\0\0\0\0h;\0\0\0\0\0\0\1\0\0\0\0\0\0\0\4\0\0\0\20\0\0\0\1\0\0\0GNU\0\0\0\0\0\3\0\0\0\2\0\0\0\0\0\0\0\363\3\0\0\n\0\0\0\0\1\0\0\16\0\0\0\0000\20D\240 \2\1\210\3\346\220\305E\214\0\304\0X\0\7\204\0p\302\200\0\r\212\fA\4\20\0\210@2\10*@\210T<- \0162H&\204\300\214\4\10\0\2\2\16\241\254\32\6f\310\0\3002\0\300\4P\t \201\10\204\v ($\0\4 Z\0\20X\200\312DB(\0\6\200\0208C\0 @\200\0IP\0Q\212@\22\0\0\0\0\10\0\0\21\20@\210@\200\265\0!\310B\301\203\240\333\244X\2\10\0040P\7\20\4\2\220@\"\0 \32\20\2\31\4F&\0\1\240\252", 832) = 832 <0.000105> 14:08:36.510270 fstat(3, {st_mode=S_IFREG|0755, st_size=1881504, ...}) = 0 <0.000124> 14:08:36.510615 mmap(NULL, 3987776, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6deab56000 <0.000130> 14:08:36.510910 mprotect(0x7f6dead1a000, 2097152, PROT_NONE) = 0 <0.000157> 14:08:36.511230 mmap(0x7f6deaf1a000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c4000) = 0x7f6deaf1a000 <0.000211> 14:08:36.511648 mmap(0x7f6deaf20000, 14656, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f6deaf20000 <0.000141> 14:08:36.511979 close(3) = 0 <0.000115> 14:08:36.512284 openat(AT_FDCWD, "/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3 <0.000185> 14:08:36.512705 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20c\0\0\0\0\0\0@\0\0\0\0\0\0\0\10\263\1\0\0\0\0\0\0\0\0\0@\0008\0\t\0@\0\"\0!\0\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0\370\1\0\0\0\0\0\0\370\1\0\0\0\0\0\0\10\0\0\0\0\0\0\0\3\0\0\0\4\0\0\0\0m\1\0\0\0\0\0\0m\1\0\0\0\0\0\0m\1\0\0\0\0\0\34\0\0\0\0\0\0\0\34\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0\1\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0(\245\1\0\0\0\0\0(\245\1\0\0\0\0\0\0\0 \0\0\0\0\0\1\0\0\0\6\0\0\0000\251\1\0\0\0\0\0000\251!\0\0\0\0\0000\251!\0\0\0\0\0 \7\0\0\0\0\0\0\260H\0\0\0\0\0\0\0\0 \0\0\0\0\0\2\0\0\0\6\0\0\0\10\253\1\0\0\0\0\0\10\253!\0\0\0\0\0\10\253!\0\0\0\0\0\0\2\0\0\0\0\0\0\0\2\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0008\2\0\0\0\0\0\0008\2\0\0\0\0\0\0008\2\0\0\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0\0\4\0\0\0\0\0\0\0P\345td\4\0\0\0\34m\1\0\0\0\0\0\34m\1\0\0\0\0\0\34m\1\0\0\0\0\0$\t\0\0\0\0\0\0$\t\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\0000\251\1\0\0\0\0\0000\251!\0\0\0\0\0000\251!\0\0\0\0\0\320\6\0\0\0\0\0\0\320\6\0\0\0\0\0\0\1\0\0\0\0\0\0\0\4\0\0\0\20\0\0\0\1\0\0\0GNU\0\0\0\0\0\3\0\0\0\2\0\0\0\0\0\0\0\345\1\0\0[\0\0\0 \0\0\0\v\0\0\0\31#\2\261\1\10\20\2@@a\370\3\10\10\25\200 \0\0\0\0\200\300\321Q\0\0\0\22\353\3020D\0\10\20A\0\2\0\2\f\1\200\v\221\1\330\240\r\240@\230 \244\200\21\n\202-l@g\214V\24\0\224 \200$H\200P(\1\22\f\311B\240\220\22\10\f \2ZdA\245c\4@\n\n\n\0\2009\1(\314D\204\201\300\22\10(\fD\0\0\0\200Q\10\200\35\4B\320\2608A\0\1\0\0\265\0300\0\200\2\20\"\0\tA\20\1\5\0P(\251\22G(\0\0\202\4\230@\4\0\20\340T\0\2@\2\2\20\3010f\26\200\0\0\0$\4\24\2\0\34\200\243\220\6\0\30\0\10\20 \1\200\0(\6D%\210\10 \0\20`\220\200\260\0\0\0\1\0", 832) = 832 <0.000137> 14:08:36.513266 fstat(3, {st_mode=S_IFREG|0755, st_size=113544, ...}) = 0 <0.000171> 14:08:36.513694 mmap(NULL, 2224608, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6dea936000 <0.000143> 14:08:36.514034 mprotect(0x7f6dea951000, 2093056, PROT_NONE) = 0 <0.000166> 14:08:36.514418 mmap(0x7f6deab50000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a000) = 0x7f6deab50000 <0.000135> 14:08:36.514781 mmap(0x7f6deab52000, 12768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f6deab52000 <0.000144> 14:08:36.515116 close(3) = 0 <0.000118> 14:08:36.515494 mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6deb348000 <0.000099> 14:08:36.515797 arch_prctl(ARCH_SET_FS, 0x7f6deb348740) = 0 <0.000151> 14:08:36.516194 mprotect(0x7f6deaf1a000, 16384, PROT_READ) = 0 <0.000172> 14:08:36.516585 mprotect(0x7f6deab50000, 4096, PROT_READ) = 0 <0.000131> 14:08:36.516904 mprotect(0x7f6deb12a000, 4096, PROT_READ) = 0 <0.000147> 14:08:36.517234 mprotect(0x564a64ad3000, 4096, PROT_READ) = 0 <0.000208> 14:08:36.517656 mprotect(0x7f6deb355000, 4096, PROT_READ) = 0 <0.000156> 14:08:36.518042 munmap(0x7f6deb34d000, 31463) = 0 <0.000156> 14:08:36.518436 set_tid_address(0x7f6deb348a10) = 5089 <0.000137> 14:08:36.518778 set_robust_list(0x7f6deb348a20, 24) = 0 <0.000145> 14:08:36.519163 rt_sigaction(SIGRTMIN, {0x7f6dea93bcd0, [], SA_RESTORER|SA_SIGINFO, 0x7f6dea94a0f0}, NULL, 8) = 0 <0.000199> 14:08:36.519604 rt_sigaction(SIGRT_1, {0x7f6dea93bd80, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7f6dea94a0f0}, NULL, 8) = 0 <0.000155> 14:08:36.519971 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0 <0.000146> 14:08:36.520325 prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=81921024, rlim_max=RLIM64_INFINITY}) = 0 <0.000067> 14:08:36.520801 stat("/var/run/xenstored/socket", 0x7ffc206510b0) = -1 ENOENT (No such file or directory) <0.000107> 14:08:36.521153 stat("/proc/xen/xenbus", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 <0.000217> 14:08:36.521624 openat(AT_FDCWD, "/proc/xen/xenbus", O_RDWR) = -1 EACCES (Permission denied) <0.000114> 14:08:36.521965 write(2, "xenstore: ", 10xenstore: ) = 10 <0.000160> 14:08:36.522342 write(2, "xs_open", 7xs_open) = 7 <0.000156> 14:08:36.522733 write(2, ": ", 2: ) = 2 <0.000122> 14:08:36.523052 write(2, "Permission denied\n", 18Permission denied ) = 18 <0.000143> 14:08:36.523479 exit_group(1) = ? 14:08:36.523728 +++ exited with 1 +++

[ajeddeloh]

Is this new with that release or does it also affect older versions?

[sanmigit]

No idea... I just stumbled on it on a new installation. For what I could google, seems related to #1563 and #2158 which where closed without positive confirmation of the proposed solution...

[longdoan]

I know for sure that it works on CoreOS stable (1465.6.0) (Ladybug), so it must have been re-broken sometime after that. It has the same error in the logs though, so I'm going to suggest that it's a different bug:

[root@xcp01 ~]# xe vm-list name-label=core01 params=name-label,networks name-label ( RW) : core01 networks (MRO): 0/ip: 172.16.1.131

core@core01 ~ $ uname -a Linux core01 4.12.7-coreos #1 SMP Tue Aug 15 23:54:56 UTC 2017 x86_64 Intel(R) Xeon(R) CPU E5645 @ 2.40GHz GenuineIntel GNU/Linux core@core01 ~ $ strace -tt -T -s 1024 -f xenstore read name 01:58:00.387352 execve("/usr/bin/xenstore", ["xenstore", "read", "name"], [/ 24 vars /]) = 0 <0.000197> 01:58:00.387798 brk(0) = 0x82d2d83000 <0.000022> 01:58:00.387916 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f31397a6000 <0.000028> 01:58:00.388023 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) <0.000026> 01:58:00.388121 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 <0.000029> 01:58:00.388220 fstat(3, {st_mode=S_IFREG|0644, st_size=30332, ...}) = 0 <0.000023> 01:58:00.388368 mmap(NULL, 30332, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f313979e000 <0.000027> 01:58:00.388449 close(3) = 0 <0.000022> 01:58:00.388548 open("/lib64/libxenstore.so.3.0", O_RDONLY|O_CLOEXEC) = 3 <0.000033> ... ... 01:58:00.393278 getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0 <0.000022> 01:58:00.393404 stat("/var/run/xenstored/socket", 0x7ffdca6d69a0) = -1 ENOENT (No such file or directory) <0.000033> 01:58:00.393515 stat("/proc/xen/xenbus", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 <0.000029> 01:58:00.393639 open("/proc/xen/xenbus", O_RDWR) = -1 EACCES (Permission denied) <0.000027> 01:58:00.393745 write(2, "xenstore: ", 10xenstore: ) = 10 <0.000031> 01:58:00.393843 write(2, "xs_open", 7xs_open) = 7 <0.000031> 01:58:00.393937 write(2, ": ", 2: ) = 2 <0.000031> 01:58:00.394041 write(2, "Permission denied\n", 18Permission denied ) = 18 <0.000022> 01:58:00.394139 exit_group(1) = ? 01:58:00.394288 +++ exited with 1 +++

[longdoan]

Here are the difference I found, between core01 (works) and core03 (doesn't work):

core@core01 ~ $ cat /etc/lsb-release DISTRIB_ID="Container Linux by CoreOS" DISTRIB_RELEASE=1465.6.0 DISTRIB_CODENAME="Ladybug" DISTRIB_DESCRIPTION="Container Linux by CoreOS 1465.6.0 (Ladybug)" core@core01 ~ $ sudo xenstore list attr eth0 PVAddons core@core01 ~ $

---

core@core03 ~ $ cat /etc/lsb-release DISTRIB_ID="Container Linux by CoreOS" DISTRIB_RELEASE=2191.5.0 DISTRIB_CODENAME="Rhyolite" DISTRIB_DESCRIPTION="Container Linux by CoreOS 2191.5.0 (Rhyolite)" core@core03 ~ $ sudo xenstore list attr PVAddons core@core03 ~ $

---

Notice how eth0 is missing from core03's listing.

[longdoan]

A few more diffs... It looks like the newer versions of CoreOS have reveretd back to an older version of xentools.

---

core@core01 ~ $ grep DESCRIPTION /etc/lsb-release DISTRIB_DESCRIPTION="Container Linux by CoreOS 1465.6.0 (Ladybug)" core@core01 ~ $ sudo -s core01 core # xenstore list attr eth0 PVAddons core01 core # xenstore read attr/eth0/ip 172.16.1.131 core01 core # xenstore list attr/PVAddons MajorVersion MinorVersion MicroVersion BuildVersion Installed core01 core # xenstore read attr/PVAddons/MajorVersion 6 core01 core # xenstore read attr/PVAddons/MinorVersion 5 core01 core # exit

---

core@core03 ~ $ grep DESCRIPTION /etc/lsb-release DISTRIB_DESCRIPTION="Container Linux by CoreOS 2191.5.0 (Rhyolite)" core@core03 ~ $ sudo -s core03 core # xenstore list attr PVAddons core03 core # xenstore list attr/PVAddons MajorVersion MinorVersion MicroVersion core03 core # xenstore read attr/PVAddons/MajorVersion 6 core03 core # xenstore read attr/PVAddons/MinorVersion 2 core03 core # exit

[longdoan]

Got 2191.5.0 to work, the xe-linux-distribution.service was missing the XE_UPDATE_GUEST_ATTRS environment (see below)

---

- name: xe-linux-distribution.service
  command: start
  content: |
    [Unit]
    Description=Hypervisor Linux Guest Agent
    After=docker.service

    [Service]
    ExecStartPre=/media/configdrive/agent/xe-linux-distribution /var/cache/xe-linux-distribution
    Environment="XE_UPDATE_GUEST_ATTRS=/media/configdrive/agent/xe-update-guest-attrs"
    ExecStart=/media/configdrive/agent/xe-daemon