bnaecker
commented
1 month ago After a lot of debugging with @rcgoodfellow and @ahl, we believe this is caused by https://www.illumos.org/issues/15782.
The crux of the issue is that:
zone-setupattempts to create an addrconf address over its data link:ipadm create-addr -t -T addrconf oxControlService0/ll- that makes a call into
libipadmto create an interface and address object - the last step is "duplicate address detection", in which
ipadmasksin.ndpd, then NDP daemon, to check that this address is available on the local network (no other directly-connected links have the same address) - through an as-yet-unexplained synchronization issue,
in.ndpdbelieves it is already doing duplicate address detection on this address, which causes theipadm create-addrcall to fail in the way we see.
We've put a core from in.ndpd which proves the invalid state at /staff/core/omicron-6947/core.7593. From there, we can see that the phyint.pi_autoconf flag is set to true on an interface with the right name:
> phyints::print [0] ! grep -E '_name|_autoconf'
[0].pi_name = [ "oxControlService10" ]
[0].pi_autoconf = 0x1 (B_TRUE)That causes this code in ndpd_create_addrs() to find that there already exists an interface and that autoconfiguration is already in progress for it. That's not correct. At this point, we had deleted the IP interface object, but in.ndpd still believes it is there:
root@oxz_crucible_26fb3830:~# ipadm show-if
IFNAME CLASS STATE CURRENT PERSISTENT
lo0 VIRTUAL ok -m-v------46 ---
oxControlService10 IP down bm--------46 ---
root@oxz_crucible_26fb3830:~# ipadm delete-if oxControlService10
root@oxz_crucible_26fb3830:~# gcore 7593
gcore: core.7593 dumped
root@oxz_crucible_26fb3830:~# mdb core.7593
Loading modules: [ libc.so.1 ld.so.1 ]
> num_of_phyint::print
mdb: failed to dereference symbol: unknown symbol name
> num_of_phyints::print
0x1
> phyints::print [0]
[0] = {
[0].pi_next = 0
[0].pi_prev = 0
[0].pi_prefix_list = 0
[0].pi_router_list = 0
[0].pi_adv_prefix_list = 0
[0].pi_index = 0x2
[0].pi_name = [ "oxControlService10" ]
[0].pi_sock = 0xffffffff
[0].pi_ifaddr = ::
[0].pi_flags = 0x21002000840
[0].pi_mtu = 0
[0].pi_token = ::
[0].pi_token_length = 0
[0].pi_stateless = 0x1 (B_TRUE)
[0].pi_stateful = 0x1 (B_TRUE)
[0].pi_tmp_token = ::
[0].pi_dst_token = ::
[0].pi_state = 0
[0].pi_kernel_state = 0
[0].pi_num_k_routers = 0
[0].pi_reach_time_since_random = 0
[0].pi_adv_time_left = 0
[0].pi_adv_time_since_sent = 0
[0].pi_adv_state = 0 (NO_ADV)
[0].pi_adv_count = 0
[0].pi_sol_time_left = 0
[0].pi_sol_state = 0 (0)
[0].pi_sol_count = 0
[0].pi_config = [
{
cf_value = 0x1
cf_notdefault = 0 (0)
},
{
cf_value = 0
cf_notdefault = 0 (0)
},
...Most of the state here is cleared, but the pi_autoconf flag is not. That seems to be reused in error when we try to recreate the address object.
Again, we're not sure how the system got into this state, but we are pretty confident that there is some race that prevents in.ndpd from completely resetting its state, in particular the pi_autoconf field.
rcgoodfellow
I had a provision hang: dogfood instance
c49820df-25ea-4c0f-aeb2-5e16980b6e84, saga311f4acf-798d-435e-9f1c-d8ef75977aac, which is apparently stuck inregions_ensure(based on it being node 173 that has started but not finished and looking in the saga_dag). From the Nexus log, this went bad around here:That zone did not come up properly:
That log file is rotated out and there are subsequent empty ones but here's the last non-empty one:
It seems like there are a few issues here:
ipadmerror -- I'm not sure what this wasI'm not sure if these are issues: