aweiteka / UATFramework

A lightweight User Acceptance Testing framework
GNU General Public License v2.0
3 stars 12 forks source link

add atomic scanner testing #89

Closed chuanchang closed 8 years ago

chuanchang commented 8 years ago

The latest testing is on RHAH 7.2.5 (9bfe1fb650), the test result looks good .

1 feature passed, 0 failed, 0 skipped 17 scenarios passed, 0 failed, 0 skipped 55 steps passed, 0 failed, 0 skipped, 0 undefined Took 3m26.646s

the details as follows

$ behave features/rhelah/atomic/atomic_scanner.feature
Feature: Atomic scanner test # features/rhelah/atomic/atomic_scanner.feature:2
  Describes the basic 'atomic scan' command test
  Background: Atomic hosts are discovered  # features/rhelah/atomic/atomic_scanner.feature:5

  @list_default_scanners
  Scenario: 1. List default scanners             # features/rhelah/atomic/atomic_scanner.feature:9
    Given "all" hosts can be pinged              # steps/common.py:201 1.390s
    When List default scanners                   # steps/rhelah.py:714 0.904s
    Then find specified scanner "rhel7/openscap" # steps/rhelah.py:722 0.860s

  @pull_atomic_scan_openscap_image
  Scenario: 2. Pull latest openscap image from repository                             # features/rhelah/atomic/atomic_scanner.feature:14
    Given "all" hosts can be pinged                                                   # steps/common.py:201 0.210s
    When atomic update latest "docker.io/fedora/atomic_scan_openscap" from repository # steps/rhelah.py:534 8.144s
    Then Check whether "atomic_scan_openscap" is installed                            # steps/rhelah.py:543 0.846s

  @atomic_install_openscap_image
  Scenario: 3. Execute openscap image install method                    # features/rhelah/atomic/atomic_scanner.feature:19
    Given "all" hosts can be pinged                                     # steps/common.py:201 0.210s
    When Execute "docker.io/fedora/atomic_scan_openscap" install method # steps/rhelah.py:667 3.875s

  @list_current_scanners
  Scenario: 4. List available scanners                # features/rhelah/atomic/atomic_scanner.feature:23
    Given "all" hosts can be pinged                   # steps/common.py:201 0.250s
    When List available scanners                      # steps/rhelah.py:714 0.871s
    Then find specified scanner "rhel7/openscap"      # steps/rhelah.py:722 0.862s
    And find specified scanner "atomic_scan_openscap" # steps/rhelah.py:722 0.894s

  @pull_centos_image
  Scenario: 5. Pull latest centos image from repository  # features/rhelah/atomic/atomic_scanner.feature:29
    Given "all" hosts can be pinged                      # steps/common.py:201 0.253s
    When atomic update latest "centos" from repository   # steps/rhelah.py:534 11.344s
    Then Check whether "centos" is installed             # steps/rhelah.py:543 0.867s

  @scan_centos_image
  Scenario: 6. Scanning centos image                                     # features/rhelah/atomic/atomic_scanner.feature:34
    Given "all" hosts can be pinged                                      # steps/common.py:201 0.209s
    When Scan "centos" with "atomic_scan_openscap" and "cve" "--verbose" # steps/rhelah.py:729 8.374s
    Then Check "not supported" in scanner report                         # steps/rhelah.py:743 0.000s
    And check whether "/run/atomic" does not exist                       # steps/rhelah.py:527 0.224s

  @pull_rhel7_image
  Scenario: 7. Pull latest rhel7 image from repository                           # features/rhelah/atomic/atomic_scanner.feature:40
    Given "all" hosts can be pinged                                              # steps/common.py:201 0.206s
    When atomic update latest "registry.access.redhat.com/rhel7" from repository # steps/rhelah.py:534 73.535s
    Then Check whether "rhel7" is installed                                      # steps/rhelah.py:543 0.884s

  @scan_rhel7_image
  Scenario: 8. Scanning rhel7 image                                                                # features/rhelah/atomic/atomic_scanner.feature:45
    Given "all" hosts can be pinged                                                                # steps/common.py:201 0.209s
    When Scan "registry.access.redhat.com/rhel7" with "atomic_scan_openscap" and "cve" "--verbose" # steps/rhelah.py:729 9.695s
    Then Check "issues were found" in scanner report                                               # steps/rhelah.py:743 0.000s
    And check whether "/run/atomic" does not exist                                                 # steps/rhelah.py:527 0.352s

  @scan_all_images
  Scenario: 9. Scanning all of images                                      # features/rhelah/atomic/atomic_scanner.feature:51
    Given "all" hosts can be pinged                                        # steps/common.py:201 0.214s
    When Scan "--images" with "atomic_scan_openscap" and "cve" "--verbose" # steps/rhelah.py:729 15.553s
    Then Check "issues were found" in scanner report                       # steps/rhelah.py:743 0.000s
    And Check "not supported" in scanner report                            # steps/rhelah.py:743 0.000s
    And check whether "/run/atomic" does not exist                         # steps/rhelah.py:527 0.232s

  @run_container1_in_bg
  Scenario: 10. docker run container1 with detach mode         # features/rhelah/atomic/atomic_scanner.feature:58
    Given "all" hosts can be pinged                            # steps/common.py:201 0.210s
    When docker run "centos" in detach mode with "C1" "top -b" # steps/docker.py:71 3.162s
    Then find latest created container by name "C1"            # steps/docker.py:84 0.281s

  @run_container2_in_bg
  Scenario: 11. docker run container2 with detach mode        # features/rhelah/atomic/atomic_scanner.feature:63
    Given "all" hosts can be pinged                           # steps/common.py:201 0.218s
    When docker run "rhel7" in detach mode with "C2" "top -b" # steps/docker.py:71 2.201s
    Then find latest created container by name "C2"           # steps/docker.py:84 0.308s

  @scan_containers
  Scenario: 12. scan containers                                                # features/rhelah/atomic/atomic_scanner.feature:68
    Given "all" hosts can be pinged                                            # steps/common.py:201 0.217s
    When Scan "--containers" with "atomic_scan_openscap" and "cve" "--verbose" # steps/rhelah.py:729 20.227s
    Then Check "issues were found" in scanner report                           # steps/rhelah.py:743 0.000s
    And Check "not supported" in scanner report                                # steps/rhelah.py:743 0.000s
    And check whether "/run/atomic" does not exist                             # steps/rhelah.py:527 0.225s

  @scan_images_and_containers
  Scenario: 13. scan all of images and containers                       # features/rhelah/atomic/atomic_scanner.feature:75
    Given "all" hosts can be pinged                                     # steps/common.py:201 0.216s
    When Scan "--all" with "atomic_scan_openscap" and "cve" "--verbose" # steps/rhelah.py:729 30.227s
    Then Check "issues were found" in scanner report                    # steps/rhelah.py:743 0.000s
    And Check "not supported" in scanner report                         # steps/rhelah.py:743 0.000s
    And check whether "/run/atomic" does not exist                      # steps/rhelah.py:527 0.247s

  @stop_container1
  Scenario: 14. atomic stop previous running container1  # features/rhelah/atomic/atomic_scanner.feature:82
    Given "all" hosts can be pinged                      # steps/common.py:201 0.208s
    When atomic stop container "C1"                      # steps/rhelah.py:472 2.609s

  @stop_container2
  Scenario: 15. atomic stop previous running container2  # features/rhelah/atomic/atomic_scanner.feature:86
    Given "all" hosts can be pinged                      # steps/common.py:201 0.208s
    When atomic stop container "C2"                      # steps/rhelah.py:472 2.916s

  @remove_all_containers
  Scenario: 16. Remove all of containers  # features/rhelah/atomic/atomic_scanner.feature:90
    Given "all" hosts can be pinged       # steps/common.py:201 0.210s
    When docker remove all of containers  # steps/docker.py:118 0.682s

  @remove_all_images
  Scenario: 17. Remove all of containers  # features/rhelah/atomic/atomic_scanner.feature:94
    Given "all" hosts can be pinged       # steps/common.py:201 0.211s
    When docker remove all of images      # steps/docker.py:112 0.397s

1 feature passed, 0 failed, 0 skipped
17 scenarios passed, 0 failed, 0 skipped
55 steps passed, 0 failed, 0 skipped, 0 undefined
Took 3m26.646s

Signed-off-by: Alex Jia ajia@redhat.com

miabbott commented 8 years ago

Nice work!