Acceptance tests failing in nightly CI

[phil-davis]

Various acceptance tests have been failing in nightly CI since https://drone.owncloud.com/owncloud/search_elastic/1641 2021-08-07 AM nightly run.

Needs investigation.

I will post a list of the failures...

[phil-davis]

https://drone.owncloud.com/owncloud/search_elastic/1660/23/12

runsh: Total unexpected failed scenarios throughout the test run:
apiSearchElastic/searchContent.feature:158
apiSearchElastic/searchContent.feature:166
apiSearchElastic/searchContent.feature:306

  Scenario Outline: search for files by pattern (edge case)                                   # /var/www/owncloud/testrunner/apps/search_elastic/tests/acceptance/features/apiSearchElastic/searchContent.feature:148
    Given using <dav_version> DAV path                                                        # FeatureContext::usingOldOrNewDavPath()
    And user "Alice" has uploaded file with content <file-content> to "/upload-edge-case.txt" # FeatureContext::userHasUploadedAFileWithContentTo()
    And the search index has been updated                                                     # SearchElasticContext::updateIndex()
    When user "Alice" searches for <search> using the WebDAV API                              # SearchContext::userSearchesUsingWebDavAPI()
    Then the HTTP status code should be "207"                                                 # FeatureContext::thenTheHTTPStatusCodeShouldBe()
    And the search result of user "Alice" should contain these files:                         # FeatureContext::thePropfindResultShouldContainEntries()
      | /upload-edge-case.txt |

    Examples:
      | dav_version | file-content | search  |
      | old         | "000"        | "000"   |
        response does not contain the entry '/upload-edge-case.txt'
        Failed asserting that a boolean is not empty.
      | new         | "000"        | "000"   |
      | old         | "000"        | "  0"   |
      | new         | "000"        | "  0"   |
      | old         | "text -1 t"  | " -1"   |
      | new         | "text -1 t"  | " -1"   |
      | old         | "false"      | "false" |
      | new         | "false"      | "false" |
      | old         | "null"       | "null"  |
        response does not contain the entry '/upload-edge-case.txt'
        Failed asserting that a boolean is not empty.
      | new         | "null"       | "null"  |

  Scenario Outline: user should not be able to search in files of other users              # /var/www/owncloud/testrunner/apps/search_elastic/tests/acceptance/features/apiSearchElastic/searchContent.feature:289
    Given using <dav_version> DAV path                                                     # FeatureContext::usingOldOrNewDavPath()
    And user "Brian" has been created with default attributes and without skeleton files   # FeatureContext::userHasBeenCreatedWithDefaultAttributesAndWithoutSkeletonFiles()
    And user "Brian" has uploaded file with content "files content" to "/Brian-upload.txt" # FeatureContext::userHasUploadedAFileWithContentTo()
    And the search index has been updated                                                  # SearchElasticContext::updateIndex()
    When user "Brian" searches for "content" using the WebDAV API                          # SearchContext::userSearchesUsingWebDavAPI()
    Then the HTTP status code should be "207"                                              # FeatureContext::thenTheHTTPStatusCodeShouldBe()
    And the search result of user "Brian" should contain these files:                      # FeatureContext::thePropfindResultShouldContainEntries()
      | /Brian-upload.txt |
    But the search result of user "Brian" should not contain these files:                  # FeatureContext::thePropfindResultShouldContainEntries()
      | /upload.txt                   |
      | /just-a-folder/upload.txt     |
      | /just-a-folder/uploadÜठिF.txt |
      | /फन्नि näme/upload.txt        |

    Examples:
      | dav_version |
      | old         |
      | new         |
        response does not contain the entry '/Brian-upload.txt'
        Failed asserting that a boolean is not empty.

[phil-davis]

But the previous few nights everything passed: https://drone.owncloud.com/owncloud/search_elastic/1659 https://drone.owncloud.com/owncloud/search_elastic/1658

So this is an intermittent issue.

The previous run that failed was https://drone.owncloud.com/owncloud/search_elastic/1655/24/12 5 days ago.

runsh: Total unexpected failed scenarios throughout the test run:
apiSearchElastic/searchContent.feature:201

  Scenario Outline: search for files by pattern - pattern matches filename of one file and content of others # /var/www/owncloud/testrunner/apps/search_elastic/tests/acceptance/features/apiSearchElastic/searchContent.feature:185
    Given using <dav_version> DAV path                                                                       # FeatureContext::usingOldOrNewDavPath()
    And user "Alice" has uploaded file with content "files content" to "/ownCloud.txt"                       # FeatureContext::userHasUploadedAFileWithContentTo()
    And the search index has been updated                                                                    # SearchElasticContext::updateIndex()
    When user "Alice" searches for "ownCloud" using the WebDAV API                                           # SearchContext::userSearchesUsingWebDavAPI()
    Then the HTTP status code should be "207"                                                                # FeatureContext::thenTheHTTPStatusCodeShouldBe()
    And the search result of user "Alice" should contain these files:                                        # FeatureContext::thePropfindResultShouldContainEntries()
      | /ownCloud.txt  |
      | /textfile0.txt |
      | /textfile1.txt |
      | /textfile2.txt |
      | /textfile3.txt |
      | /textfile4.txt |

    Examples:
      | dav_version |
      | old         |
      | new         |
        response does not contain the entry '/ownCloud.txt'
        Failed asserting that a boolean is not empty.

My thought is that there is some timing issue with "the search index has been updated"

[phil-davis]

PR #226 passed first time. When that is merged I will make another test PR and see if I can get any fails.

[phil-davis]

Fixed by #227 - we will see over coming nightly CI if the tests are reliable.