oxidecomputer / omicron

Omicron: Oxide control plane
Mozilla Public License 2.0
252 stars 40 forks source link

preflight check reports `missing metadata` #7108

Open leftwo opened 2 days ago

leftwo commented 2 days ago

During a RSS setup preflight check, we report a bunch of missing metadata.

On a successful run, I see a number of these lines printed:

alan@jeeves $ ssh wicket@fe80::aa40:25ff:fe05:2e02%dublin_sw1tp0 preflight uplink                                                                       
✔ Configuring switch (11.951972ms)                                                                                                                      
    configured Qsfp(Qsfp("qsfp0"))/0: ips [                                                                                                             
    UplinkAddressConfig {                                                                                                                               
        address: V4(                                                                                                                                    
            Ipv4Net {                                                                                                                                   
                addr: 172.20.15.62,                                                                                                                     
                width: 29,                                                                                                                              
            },                                                                                                                                          
        ),                                                                                                                                              
        vlan_id: None,                                                                                                                                  
    },                                                                                                                                                  
], routes [                                                                                                                                             
    RouteConfig {                                                                                                                                       
        destination: V4(                                                                                                                                
            Ipv4Net {                                                                                                                                   
                addr: 0.0.0.0,                                                                                                                          
                width: 0,                                                                                                                               
            },                                                                                                                                          
        ),                                                                                                                                              
        nexthop: 172.20.15.57,                                                                                                                          
        vlan_id: None,                                                                                                                                  
        rib_priority: None,                                                                                                                             
    },                                                                                                                                                  
]                                                                                                                                                       
✔ Waiting for L1 link up (8.523888951s)                                                                                                                 
    missing metadata                                                                                                                                    
✔ Configuring IP address in host OS (631.281056ms)                                                                                                      
    configured uplinks/qsfp0_0                                                                                                                          
✔ Configuring routing in host OS (12.510337ms)                                                                                                          
    added routes [                                                                                                                                      
    RouteConfig {                                                                                                                                       
        destination: V4(                                                                                                                                
            Ipv4Net {                                                                                                                                   
                addr: 0.0.0.0,                                                                                                                          
                width: 0,                                                                                                                               
            },                                                                                                                                          
        ),                                                                                                                                              
        nexthop: 172.20.15.57,                                                                                                                          
        vlan_id: None,                                                                                                                                  
        rib_priority: None,                                                                                                                             
    },                                                                                                                                                  
]                                                                                                                                                       
✔ Checking for external DNS connectivity (5.208714335s)                                                                                                 
    DNS server 1.1.1.1 A query attempt 1: resolved ntp.eng.oxide.computer to 172.20.0.5                                                                 
    DNS server 9.9.9.9 A query attempt 1: resolved ntp.eng.oxide.computer to 172.20.0.5                                                                 
✔ Checking for external NTP connectivity (4.261500524s)                                                                                                 
    successfully contacted NTP server 172.20.0.5                                                                                                        
✔ Cleaning up host OS routing configuration (14.820905ms)                                                                                               
    missing metadata                                                                                                                                    
✔ Cleaning up host OS IP address configuration (37.567512ms)                                                                                            
    missing metadata                                                                                                                                    
✔ Cleaning up switch configuration (14.26761ms)                                                                                                         
    missing metadata 

It's a little confusing as I'm not sure if there is a problem. When there is an actual failure, It's more misleading as it appears that the problem is 'missing metadata':

alan@jeeves $ ssh wicket@fe80::aa40:25ff:fe05:2202%dublin_sw0tp0 preflight uplink                                                                       
✔ Configuring switch (9.71393ms)                                                                                                                        
    configured Qsfp(Qsfp("qsfp0"))/0: ips [                                                                                                             
    UplinkAddressConfig {                                                                                                                               
        address: V4(                                                                                                                                    
            Ipv4Net {                                                                                                                                   
                addr: 172.20.15.61,                                                                                                                     
                width: 29,                                                                                                                              
            },                                                                                                                                          
        ),                                                                                                                                              
        vlan_id: None,                                                                                                                                  
    },                                                                                                                                                  
], routes [                                                                                                                                             
    RouteConfig {                                                                                                                                       
        destination: V4(                                                                                                                                
            Ipv4Net {                                                                                                                                   
                addr: 0.0.0.0,                                                                                                                          
                width: 0,                                                                                                                               
            },                                                                                                                                          
        ),                                                                                                                                              
        nexthop: 172.20.15.57,                                                                                                                          
        vlan_id: None,                                                                                                                                  
        rib_priority: None,                                                                                                                             
    },                                                                                                                                                  
]                                                                                                                                                       
] Waiting for L1 link up (30.086649834s)                                                                                                                
    missing metadata                                                                                                                                    
    link state: Unknown                                                                                                                                 
* Configuring IP address in host OS (6.653µs)                                                                                                           
    missing metadata                                                                                                                                    
    link not up due to earlier failure                                                                                                                  
* Configuring routing in host OS (4.528µs)                                                                                                              
    missing metadata                                                                                                                                    
    addresses not configured due to earlier failure                                                                                                     
* Checking for external DNS connectivity (4.84µs)                                                                                                       
    missing metadata                                                                                                                                    
    skipped due to previous failure                                                                                                                     
* Checking for external NTP connectivity (3.847µs)                                                                                                      
    missing metadata                                                                                                                                    
    skipped due to previous failure                                                                                                                     
* Cleaning up host OS routing configuration (3.887µs)                                                                                                   
    missing metadata                                                                                                                                    
    routing was not configured due to earlier failure                                                                                                   
* Cleaning up host OS IP address configuration (3.907µs)                                                                                                
    missing metadata                                                                                                                                    
    addresses were not configured due to earlier failure                                                                                                
✔ Cleaning up switch configuration (13.50209ms)                                                                                                         
    missing metadata
jgallagher commented 2 days ago

It looks like this was added in #6078; prior to that, if a step didn't have any metadata attached, we would print nothing at all. We can probably remove this branch to go back to that: https://github.com/oxidecomputer/omicron/blob/ac3fb34a7621afcc8a2ad89a99821adf704b5dc0/wicket/src/cli/preflight.rs#L264-L266

Cc @sunshowers in case I'm missing something - maybe we expected all the steps to have metadata with other changes, and this is indicating a different problem?