crucible agent SMF service stuck in "ipadm: Could not create address: Addrconf already in progress" error

askfongjojo commented 1 year ago

On rack2, disk create failed repeated for me and they were all on the same disk.

root@oxz_nexus:~# grep 'Communication Error' /var/svc/log/oxide-nexus\:default.log | looker -o bare error_message_internal | sort | uniq -c | sort -n
   6 saga ACTION error at node "regions_ensure": Communication Error: error sending request for url (http://[fd00:1122:3344:108::7]:32345/crucible/0/regions): operation timed out
   8 -

This is gc18. So one of the crucible agents actually isn't listening for requests:

BRM42220026 # zlogin oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9
[Connected to zone 'oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9' pts/2]
Last login: Sat Jun 24 17:49:02 on pts/2
The illumos Project     stlouis-0-g4999fc97d0   June 2023
root@oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9:~# svcs -xv
root@oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9:~# svcs *agent*
STATE          STIME    FMRI
online         17:52:27 svc:/oxide/crucible/agent:default
root@oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9:~# netstat -an | grep 1122:3344

root@oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9:~# tail /var/svc/log/oxide-crucible-agent\:default.log 
++ svcprop -c -p config/snapshot_prefix svc:/oxide/crucible/agent:default
+ SNAPSHOT_PREFIX=snapshot
+ [[ oxControlService5 == unknown ]]
+ [[ fd00:1122:3344:108::1 == unknown ]]
+ ipadm show-addr oxControlService5/ll
ipadm: Address object not found
+ ipadm create-addr -t -T addrconf oxControlService5/ll
ipadm: Could not create address: Addrconf already in progress
[ Jun 24 17:52:19 Stopping because service exited with an error. ]
[ Jun 24 17:52:19 Failing too quickly, throttling. ]

askfongjojo commented 1 year ago

The top of the crucible-agent log file (the rest of it are retries of the same thing)

[ Jun 24 06:03:45 Enabled. ]
[ Jun 24 06:03:45 Rereading configuration. ]
[ Jun 24 06:03:46 Rereading configuration. ]
[ Jun 24 06:03:46 Executing start method ("/opt/oxide/lib/svc/manifest/crucible/agent.sh"). ]
+ . /lib/svc/share/smf_include.sh
++ SMF_EXIT_OK=0
++ SMF_EXIT_NODAEMON=94
++ SMF_EXIT_ERR_FATAL=95
++ SMF_EXIT_ERR_CONFIG=96
++ SMF_EXIT_MON_DEGRADE=97
++ SMF_EXIT_MON_OFFLINE=98
++ SMF_EXIT_ERR_NOSMF=99
++ SMF_EXIT_ERR_PERM=100
++ svcprop -c -p config/datalink svc:/oxide/crucible/agent:default
+ DATALINK=oxControlService5
++ svcprop -c -p config/gateway svc:/oxide/crucible/agent:default
+ GATEWAY=fd00:1122:3344:108::1
++ svcprop -c -p config/dataset svc:/oxide/crucible/agent:default
+ DATASET=oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9/crucible
++ svcprop -c -p config/listen_addr svc:/oxide/crucible/agent:default
+ LISTEN_ADDR=fd00:1122:3344:108::7
++ svcprop -c -p config/listen_port svc:/oxide/crucible/agent:default
+ LISTEN_PORT=32345
++ svcprop -c -p config/portbase svc:/oxide/crucible/agent:default
+ PORTBASE=19000
++ svcprop -c -p config/downstairs_prefix svc:/oxide/crucible/agent:default
+ DOWNSTAIRS_PREFIX=downstairs
++ svcprop -c -p config/snapshot_prefix svc:/oxide/crucible/agent:default
+ SNAPSHOT_PREFIX=snapshot
+ [[ oxControlService5 == unknown ]]
+ [[ fd00:1122:3344:108::1 == unknown ]]
+ ipadm show-addr oxControlService5/ll
ipadm: Address object not found
+ ipadm create-addr -t -T addrconf oxControlService5/ll
ipadm: Could not create address: Addrconf already in progress
[ Jun 24 06:03:46 Stopping because service exited with an error. ]

smklein commented 1 year ago

Accessing that zone, here's some other info I found:

This error repeats itself when the SMF manifest script re-runs, as it tries to re-start the service. This makes sense, as the addrconf address stays in the same state.
The address is not visible through ipadm:

root@oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9:~# ipadm
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
lo0/v6            static   ok           ::1/128

The error is reproducible manually:

root@oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9:~# ipadm create-addr -t -T addrconf oxControlService5/ll
ipadm: Could not create address: Addrconf already in progress

smklein commented 1 year ago

This is also a zone with an exclusive IP stack, and here's the "addrobj" config file:

root@oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9:~# cat /etc/svc/volatile/ipadm/aobjmap.conf
_aobjname=lo0/v4;_ifname=lo0;_lifnum=0;_family=2;_flags=4;_atype=1;_addr=;
_aobjname=lo0/v6;_ifname=lo0;_lifnum=0;_family=26;_flags=4;_atype=1;_addr=;

This matches the ipadm output, but not the error message

smklein commented 1 year ago

It may be worth following up on two questions here:

What's the state of the IP management in this zone? I think that would involve prodding at ./usr/src/cmd/cmd-inet/lib/ipmgmtd in illumos-gate. I'd like to extract state from this system before restarting the service, but that would also be the next step -- I don't see any external indications that an "Addrconf is already in progress"
Why might an addrconf already be in-progress? I tried reading the very start of the crucible agent log, and it looks like it's following the expected steps in https://github.com/oxidecomputer/crucible/blob/main/agent/agent_method_script.sh , failing here specifically: https://github.com/oxidecomputer/crucible/blob/ae135d3d9920c1a8f5d7b2ecb9437c43e1176664/agent/agent_method_script.sh#L23 . But it's mysterious that the creation of this addrconf object would "already be in progress" when it looks like it's running for the first time?

smklein commented 1 year ago

Within illumos-gate, I see only two spots where the error "Addrconf is already in progress" could be returned.

That text string looks like it translates to the error IPADM_ADDRCONF_EXISTS
That error is only returned from here (within i_ipadm_create_linklocal, if any prior operation failed and returned EEXIST):

https://github.com/oxidecomputer/illumos-gate/blob/4999fc97d071dcdb5a22a8c60238d237b5f9827f/usr/src/lib/libipadm/common/ipadm_ndpd.c#L209

or here (within i_ipadm_send_ndpd_cmd, under similar conditions):

https://github.com/oxidecomputer/illumos-gate/blob/4999fc97d071dcdb5a22a8c60238d237b5f9827f/usr/src/lib/libipadm/common/ipadm_ndpd.c#L302

bnaecker commented 1 year ago

There have been a few issues similar to this, where the user land and kernel parts of the IP stack disagree. The one that seems most relevant is https://www.illumos.org/issues/13316, though I’d have expected that to be fixed. Is it possible the OS on this host doesn’t include the fix for this?

rcgoodfellow commented 1 year ago

This error is coming from ndpd, here is the log.

root@oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9:~# cat $(svcs -L ndp)
[ Jun 24 06:03:42 Disabled. ]
[ Jun 24 06:03:46 Enabled. ]
[ Jun 24 06:03:46 Executing start method ("/lib/svc/method/svc-ndp"). ]
[ Jun 24 06:03:46 Method "start" exited with status 0. ]
root@oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9:~# dmesg

Sun Jun 25 02:19:40 UTC 2023
Jun 25 02:18:56 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:18:56 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:18:56 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:18:56 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:18:57 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:18:57 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:18:57 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:18:57 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:18:58 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:18:58 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:18:58 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:18:58 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:18:59 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:18:59 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:18:59 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:18:59 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:00 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:00 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:00 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:00 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:00 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:00 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:00 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:00 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:02 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:02 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:02 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:02 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:03 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:03 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:03 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:03 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:04 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:04 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:04 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:04 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:05 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:05 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:05 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:05 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:05 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:05 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:05 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:05 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:06 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:06 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:06 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:06 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:07 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:07 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:07 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:07 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:08 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:08 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:08 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:08 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:09 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:09 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:09 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:09 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:09 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:09 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:09 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:09 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:10 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:10 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:10 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:10 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:11 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:11 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:11 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:11 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:13 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:13 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:13 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:13 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:14 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:14 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:14 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:14 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:14 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:14 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:14 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:14 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:15 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:15 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:15 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:15 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:16 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:16 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:16 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:16 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:17 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:17 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:17 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:17 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:18 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:18 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:18 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:18 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:18 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:18 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:18 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:18 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:19 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:19 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:19 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:19 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:20 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:20 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:20 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:20 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:21 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:21 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:21 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:21 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:23 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:23 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:23 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:23 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:23 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:23 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:23 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:23 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:24 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:24 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:24 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:24 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:25 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:25 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:25 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:25 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:26 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:26 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:26 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:26 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:27 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:27 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:27 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:27 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:27 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:27 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:27 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:27 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:28 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:28 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:28 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:28 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:29 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:29 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:29 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:29 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:30 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:30 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:30 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:30 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:31 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:31 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:31 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:31 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:32 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:32 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:32 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:32 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:33 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:33 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:33 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:33 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:34 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:34 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:34 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:34 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:35 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:35 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:35 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:35 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:36 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:36 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:36 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:36 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:36 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:36 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:36 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:36 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:37 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:37 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:37 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:37 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:38 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:38 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:38 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:38 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:39 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:39 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:39 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:39 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1
Jun 25 02:19:40 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 949453 daemon.error] autoconfiguration already in progress
Jun 25 02:19:40 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 581749 daemon.error] ndpd command on interface oxControlService5 failed with error File exists
Jun 25 02:19:40 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 in.ndpd[13181]: [ID 169330 daemon.error] Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it
Jun 25 02:19:40 oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9 svc.startd[12572]: [ID 694882 daemon.notice] instance svc:/oxide/crucible/agent:default exited with status 1

The point at which the error is produced is here

note this was going on approximately twice a second for several hours, until around 4:11 on the system clock.

rcgoodfellow commented 1 year ago

There is still a plumbed interface for oxControlService5

root@oxz_crucible_oxp_3552987c-8da9-4a9b-bd9b-535e2d4bffe9:~# ipadm show-if
IFNAME     CLASS     STATE    CURRENT      PERSISTENT
lo0        VIRTUAL   ok       -m-v------46 ---
oxControlService5 IP down     bm--------46 ---

Since the log shows a bunch of these entries

Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it

We're entering this function, presumably because the entity trying to establish this address is doing so in a loop that takes out the interface on retries.

In looking through that code. We see this at the tail end of the function

        /*
         * Clear state so that should the phyint reappear we will
         * start with initial advertisements or solicitations.
         */
        phyint_cleanup(pi);

However, one of the things that phyint_cleanup does not clean up is phyint`pi_autoconf, which is the variable that is checked to see if "autoconfiguration is already in progress".

I'll work on reproducing this and will see if clearing pi->pi_autoconf in phyint_cleanup solves the issue.

rcgoodfellow commented 1 year ago

This log

Interface oxControlService5 has been removed from kernel. in.ndpd will no longer use it

shows up when phyint`pi_kernel_state does not have the PI_PRESENT flag set. This is actively happening on the system, and I'm looking at the interface in the kernel, and it definitely is present in the kernel (and in ipmgmtd for that matter). So ndpd is confused on multiple fronts here.

Notably, this field is cleared via phyint_cleanup.

rcgoodfellow commented 1 year ago

moving discussion to

oxidecomputer/stlouis#435

rcgoodfellow commented 1 year ago

Note that running the following makes the service spring to life.

ipadm delete-if oxControlService5

@smklein we may want to add

ipadm delete-if $DATALINK || true

just before

ipadm show-addr "$DATALINK/ll" || ipadm create-addr -t -T addrconf "$DATALINK/ll"

as a workaround until the underlying issue is pegged as I'm having some difficulty reproducing this.

askfongjojo commented 1 year ago

Thank you all for looking into the issue! Feel free to move this to omicron if it is where the fix should be.

jclulow commented 1 year ago

Because this is proving difficult to reproduce, we should consider sticking some DTrace instrumentation into the OS image we're building that is always collecting evidence we would need so that when it happens again we can fish that out.

dancrossnyc commented 1 year ago

Reading this and looking at stlouis#435, I concur with Ry's analysis: this is a bug in in.ndpd that is manifesting itself as an ipadm failure. Upstream 13316 is definitely a red-herring and unrelated.

rcgoodfellow commented 1 year ago

Fixed in

https://github.com/oxidecomputer/stlouis/issues/435

oxidecomputer / crucible

crucible agent SMF service stuck in "ipadm: Could not create address: Addrconf already in progress" error #818