canonical / checkbox

Checkbox
https://checkbox.readthedocs.io
GNU General Public License v3.0
30 stars 44 forks source link

Wireless tests with netplan passed even though the given APs were not exist #1308

Open weizhenwu opened 1 week ago

weizhenwu commented 1 week ago

Bug Description

While running wireless-automated test plan, the tests using netplan were passed with non-exist APs

Please refer to the problematic submission You can find the log in the passed tests. For example:

## Generate a test netplan configuration
# This is the network config written by checkbox
network:
  version: 2
  wifis:
    wlan0:
      access-points:
        ubuntu-cert-ac-open: {}
      addresses: []
      dhcp4: True
      nameservers: {}

## Write the test netplan config file to disk

## Apply the test configuration
+ netplan --debug apply

## Wait for interface to be routable
 routable (configured)
Reached routable state

## Display address
+ ip address show dev wlan0
2: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 3000
    link/ether 00:03:7f:12:d6:d6 brd ff:ff:ff:ff:ff:ff
    inet 10.102.137.157/22 metric 600 brd 10.102.139.255 scope global dynamic wlan0
       valid_lft 555sec preferred_lft 555sec
    inet6 fe80::c7d7:24fa:5a6e:571d/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

## Display route table
+ ip route
default via 10.102.136.1 dev wlan0 proto dhcp src 10.102.137.157 metric 600 
10.102.136.0/22 dev wlan0 proto kernel scope link src 10.102.137.157 metric 600 
10.102.136.1 dev wlan0 proto dhcp scope link src 10.102.137.157 metric 600 
91.189.91.157 via 10.102.136.1 dev wlan0 proto dhcp src 10.102.137.157 metric 600 
185.125.190.56 via 10.102.136.1 dev wlan0 proto dhcp src 10.102.137.157 metric 600 
185.125.190.57 via 10.102.136.1 dev wlan0 proto dhcp src 10.102.137.157 metric 600 
185.125.190.58 via 10.102.136.1 dev wlan0 proto dhcp src 10.102.137.157 metric 600 

Given AP ubuntu-cert-ac-open was not available, but the test still passed. The IP here was from the AP Canonical I connected before the tests started. Seems like the tests didn't check which one was connected but only see if it's pingable.

To Reproduce

  1. Connect to AP Canonical

  2. Install checkbox

    sudo snap install checkbox22 --beta
    sudo snap install checkbox-carmel-classic --classic --edge
  3. checkbox-carmel-classic.checkbox-cli run com.canonical.certification::wireless-automated

Environment

OS: Ubuntu server 22.04 Checkbox: Snap 4.0.0-dev270 DUT: 202403-33869

Relevant log output

pasted above.

Additional context

No response

syncronize-issues-to-jira[bot] commented 1 week ago

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CHECKBOX-1472.

This message was autogenerated