Closed askfongjojo closed 1 year ago
The top of the crucible-agent log file (the rest of it are retries of the same thing)
[ Jun 24 06:03:45 Enabled. ]
[ Jun 24 06:03:45 Rereading configuration. ]
[ Jun 24 06:03:46 Rereading configuration. ]
[ Jun 24 06:03:46 Executing start method ("/opt/oxide/lib/svc/manifest/crucible/agent.sh"). ]
+ . /lib/svc/share/smf_include.sh
++ SMF_EXIT_OK=0
++ SMF_EXIT_NODAEMON=94
++ SMF_EXIT_ERR_FATAL=95
++ SMF_EXIT_ERR_CONFIG=96
++ SMF_EXIT_MON_DEGRADE=97
++ SMF_EXIT_MON_OFFLINE=98
++ SMF_EXIT_ERR_NOSMF=99
++ SMF_EXIT_ERR_PERM=100
++ svcprop -c -p config/datalink svc:/oxide/crucible/agent:default
+ DATALINK=oxControlService5
++ svcprop -c -p config/gateway svc:/oxide/crucible/agent:default
+ GATEWAY=fd00:1122:3344:108::1
++ svcprop -c -p config/dataset svc:/oxide/crucible/agent:default
+ DATASET=oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9/crucible
++ svcprop -c -p config/listen_addr svc:/oxide/crucible/agent:default
+ LISTEN_ADDR=fd00:1122:3344:108::7
++ svcprop -c -p config/listen_port svc:/oxide/crucible/agent:default
+ LISTEN_PORT=32345
++ svcprop -c -p config/portbase svc:/oxide/crucible/agent:default
+ PORTBASE=19000
++ svcprop -c -p config/downstairs_prefix svc:/oxide/crucible/agent:default
+ DOWNSTAIRS_PREFIX=downstairs
++ svcprop -c -p config/snapshot_prefix svc:/oxide/crucible/agent:default
+ SNAPSHOT_PREFIX=snapshot
+ [[ oxControlService5 == unknown ]]
+ [[ fd00:1122:3344:108::1 == unknown ]]
+ ipadm show-addr oxControlService5/ll
ipadm: Address object not found
+ ipadm create-addr -t -T addrconf oxControlService5/ll
ipadm: Could not create address: Addrconf already in progress
[ Jun 24 06:03:46 Stopping because service exited with an error. ]
Accessing that zone, here's some other info I found:
ipadm
:root@oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9:~# ipadm
ADDROBJ TYPE STATE ADDR
lo0/v4 static ok 127.0.0.1/8
lo0/v6 static ok ::1/128
root@oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9:~# ipadm create-addr -t -T addrconf oxControlService5/ll
ipadm: Could not create address: Addrconf already in progress
This is also a zone with an exclusive IP stack, and here's the "addrobj" config file:
root@oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9:~# cat /etc/svc/volatile/ipadm/aobjmap.conf
_aobjname=lo0/v4;_ifname=lo0;_lifnum=0;_family=2;_flags=4;_atype=1;_addr=;
_aobjname=lo0/v6;_ifname=lo0;_lifnum=0;_family=26;_flags=4;_atype=1;_addr=;
This matches the ipadm
output, but not the error message
It may be worth following up on two questions here:
./usr/src/cmd/cmd-inet/lib/ipmgmtd
in illumos-gate. I'd like to extract state from this system before restarting the service, but that would also be the next step -- I don't see any external indications that an "Addrconf is already in progress"Within illumos-gate, I see only two spots where the error "Addrconf is already in progress" could be returned.
IPADM_ADDRCONF_EXISTS
i_ipadm_create_linklocal
, if any prior operation failed and returned EEXIST
):i_ipadm_send_ndpd_cmd
, under similar conditions):There have been a few issues similar to this, where the user land and kernel parts of the IP stack disagree. The one that seems most relevant is https://www.illumos.org/issues/13316, though I’d have expected that to be fixed. Is it possible the OS on this host doesn’t include the fix for this?
This error is coming from ndpd
, here is the log.
root@oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9:~# cat $(svcs -L ndp) [ Jun 24 06:03:42 Disabled. ] [ Jun 24 06:03:46 Enabled. ] [ Jun 24 06:03:46 Executing start method ("/lib/svc/method/svc-ndp"). ] [ Jun 24 06:03:46 Method "start" exited with status 0. ] root@oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9:~# dmesg Sun Jun 25 02:19:40 UTC 2023 Jun 25 02:18:56 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:18:56 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:18:56 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:18:56 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:18:57 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:18:57 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:18:57 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:18:57 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:18:58 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:18:58 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:18:58 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:18:58 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:18:59 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:18:59 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:18:59 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:18:59 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:00 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:00 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:00 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:00 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:00 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:00 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:00 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:00 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:02 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:02 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:02 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:02 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:03 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:03 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:03 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:03 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:04 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:04 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:04 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:04 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:05 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:05 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:05 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:05 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:05 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:05 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:05 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:05 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:06 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:06 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:06 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:06 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:07 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:07 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:07 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:07 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:08 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:08 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:08 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:08 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:09 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:09 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:09 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:09 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:09 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:09 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:09 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:09 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:10 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:10 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:10 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:10 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:11 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:11 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:11 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:11 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:13 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:13 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:13 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:13 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:14 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:14 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:14 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:14 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:14 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:14 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:14 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:14 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:15 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:15 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:15 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:15 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:16 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:16 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:16 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:16 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:17 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:17 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:17 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:17 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:18 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:18 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:18 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:18 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:18 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:18 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:18 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:18 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:19 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:19 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:19 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:19 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:20 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:20 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:20 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:20 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:21 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:21 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:21 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:21 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:23 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:23 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:23 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:23 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:23 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:23 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:23 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:23 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:24 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:24 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:24 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:24 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:25 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:25 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:25 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:25 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:26 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:26 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:26 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:26 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:27 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:27 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:27 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:27 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:27 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:27 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:27 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:27 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:28 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:28 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:28 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:28 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:29 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:29 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:29 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:29 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:30 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:30 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:30 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:30 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:31 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:31 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:31 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:31 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:32 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:32 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:32 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:32 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:33 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:33 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:33 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:33 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:34 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:34 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:34 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:34 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:35 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:35 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:35 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:35 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:36 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:36 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:36 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:36 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:36 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:36 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:36 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:36 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:37 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:37 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:37 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:37 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:38 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:38 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:38 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:38 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:39 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:39 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:39 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:39 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1 Jun 25 02:19:40 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress Jun 25 02:19:40 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists Jun 25 02:19:40 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it Jun 25 02:19:40 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
The point at which the error is produced is here
note this was going on approximately twice a second for several hours, until around 4:11 on the system clock.
There is still a plumbed interface for oxControlService5
root@oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9:~# ipadm show-if
IFNAME CLASS STATE CURRENT PERSISTENT
lo0 VIRTUAL ok -m-v------46 ---
oxControlService5 IP down bm--------46 ---
Since the log shows a bunch of these entries
Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
We're entering this function, presumably because the entity trying to establish this address is doing so in a loop that takes out the interface on retries.
In looking through that code. We see this at the tail end of the function
/*
* Clear state so that should the phyint reappear we will
* start with initial advertisements or solicitations.
*/
phyint_cleanup(pi);
However, one of the things that phyint_cleanup
does not clean up is phyint`pi_autoconf
, which is the variable that is checked to see if "autoconfiguration is already in progress".
I'll work on reproducing this and will see if clearing pi->pi_autoconf
in phyint_cleanup
solves the issue.
This log
Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
shows up when phyint`pi_kernel_state
does not have the PI_PRESENT
flag set. This is actively happening on the system, and I'm looking at the interface in the kernel, and it definitely is present in the kernel (and in ipmgmtd for that matter). So ndpd
is confused on multiple fronts here.
Notably, this field is cleared via phyint_cleanup
.
moving discussion to
Note that running the following makes the service spring to life.
ipadm delete-if oxControlService5
@smklein we may want to add
ipadm delete-if $DATALINK || true
just before
ipadm show-addr "$DATALINK/ll" || ipadm create-addr -t -T addrconf "$DATALINK/ll"
as a workaround until the underlying issue is pegged as I'm having some difficulty reproducing this.
Thank you all for looking into the issue! Feel free to move this to omicron if it is where the fix should be.
Because this is proving difficult to reproduce, we should consider sticking some DTrace instrumentation into the OS image we're building that is always collecting evidence we would need so that when it happens again we can fish that out.
Reading this and looking at stlouis#435, I concur with Ry's analysis: this is a bug in in.ndpd
that is manifesting itself as an ipadm
failure. Upstream 13316 is definitely a red-herring and unrelated.
On rack2, disk create failed repeated for me and they were all on the same disk.
This is gc18. So one of the crucible agents actually isn't listening for requests: