search

status-im / status-mobile

a free (libre) open source, mobile OS for Ethereum
https://status.app
Mozilla Public License 2.0
3.92k stars 984 forks source link

E2E tests fail with Python key error #8426

Closed jakubgs closed 5 years ago

jakubgs commented 5 years ago

Problem

The status-app-nightly job started failing with a python error on the 14th(#926:

+ python3 -m pytest -m testrail_id --testrail_report=True --rerun_count=2 --network=ropsten --apk=StatusIm-190614-010400-9b18f1-e2e.apk -n24
INTERNALERROR> Traceback (most recent call last):
...
INTERNALERROR>   File "/home/jenkins/workspace/end-to-end-tests/status-app-nightly/test/appium/support/testrail_report.py", line 61, in actual_milestone_id
INTERNALERROR>     return self.get_milestones()[-1]['id']
INTERNALERROR> KeyError: -1
jakubgs commented 5 years ago

At first I thought this was caused by wrong apk parameter, which usually looks like:

--apk=StatusIm-190614-010400-9b18f1-e2e.apk

And I saw sometimes people running it manually with:

--apk=https://status-im.ams3.digitaloceanspaces.com/StatusIm-190613-135222-742347-e2e.apk

Which I wasn't sure if it's valid format, but it didn't fail the same way in this job: https://ci.status.im/job/end-to-end-tests/job/status-app-nightly/925/console

jakubgs commented 5 years ago

First I added some exception catching and got back:

requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

Then I made a slight modification to test/appium/support/testrail_report.py:

    def get(self, method):
        rval = requests.get(self.api_url + method, headers=self.headers).json()
        if 'error' in rval:
            raise Exception('Failed request: %s' % rval['error'])
        return rval

And I got back this:

Exception: Failed request: Authentication failed: invalid or missing user/password or session cookie.

Which at least makes some sense, our credentials for TestRail are wrong.

jakubgs commented 5 years ago

But then i checked all the credentials we use in our CI and found out this:

So we have two issues:

  1. We need to get another user for CI use of SauceLabs than anton-status.
  2. Where the hell did TestRail credentials disappear to?
jakubgs commented 5 years ago

But the weird thing is, our credentials.xml file hasn't been modified since 2019-04-27, so it can't be that the TestRail credentials have been removed recently.

jakubgs commented 5 years ago

It would be best if we created new CI dedicated accounts using the auto@status.im email address for both SauceLabs and TestRail, then I can add those to Jenkins credentials properly(in case of SauceLabs an API token would be necessary).

Not sure who can actually do that: @annadanchenko @asemiankevich @Serhy @churik

I've also created a proper Jenkinsfile for these tests: https://github.com/status-im/status-react/blob/fix-e2e-build-run/ci/Jenkinsfile.nightly-end-to-end

Serhy commented 5 years ago

@jakubgs It was fixed and e2e tests are working now. What happened was that TestRail account which is used for api access has been disabled. We've created a new one, configured new api key and updated Credentials for TESTRAIL_USER and TESTRAIL_PASS (api_key here) in Jenkins.

SauceLabs account (email, user first and last name) has been updated as well from Anton's user data. However anton-status kept as it is because it's a username of the account we have (we may use email address for logins too).

jakubgs commented 5 years ago

Wait, i still don't get where you are getting TESTRAIL_USER and TESTRAIL_PASS from. I don't see it in the credentials we have available in Jenkins: https://ci.status.im/credentials/

Where exactly are they set?

jakubgs commented 5 years ago

Also I've made this new Jenkinsfile for the tests: #8451

Serhy commented 5 years ago

Wait, i still don't get where you are getting TESTRAIL_USER and TESTRAIL_PASS from. I don't see it in the credentials we have available in Jenkins: https://ci.status.im/credentials/

Where exactly are they set?

@jakubgs https://ci.status.im/job/end-to-end-tests/credentials/store/folder/domain/_/ - is where I found TESTRAIL_USER and TESTRAIL_PASS credentials stored.

jakubgs commented 5 years ago

Thanks for the info, I've merged #8451 and added #8511 with a tiny fix for triggering the new E2E builds.

jakubgs commented 5 years ago

Looks like we are still having trouble with some SauceLabs errors:

java.lang.Exception: no credentials provided
    at com.saucelabs.jenkins.pipeline.SauceStep$Execution.start(SauceStep.java:62)
    at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:268)
    at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:176)
    at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
    at sun.reflect.GeneratedMethodAccessor8458.invoke(Unknown Source)
...
jakubgs commented 5 years ago

The error seems to be originating from here: https://github.com/jenkinsci/sauce-ondemand-plugin/blob/2b82df305a378687abdb9ccc1a970f24adc85817/src/main/java/com/saucelabs/jenkins/pipeline/SauceStep.java#L60-L63

SauceCredentials credentials = SauceCredentials.getCredentialsById(job, step.getCredentialsId());
if (credentials == null) {
    throw new Exception("no credentials provided");
}

I think this refers to the sauce() call we have hardcoded with some UUID in our pipelines:

node ('linux1'){sauce('aaaaaaaa-bbbb-cccc-dddd-sssssssssssss'){
jakubgs commented 5 years ago

The documentation doesn't say much about this call: https://jenkins.io/doc/pipeline/steps/sauce-ondemand/

sauce: Sauce credentialsId Type: String

Not very useful.

jakubgs commented 5 years ago

Neither does the plugin page help much: https://wiki.jenkins.io/display/JENKINS/Sauce+OnDemand+Plugin

jakubgs commented 5 years ago

I can see examples of other people hardcoding this UUID in their Jenkinsfiles as well: https://github.com/billmeyer/testng-multi-dataprovider/blob/95d4cbe389cae867c3ddeb9a4402d5190c26867c/Jenkinsfile#L25

jakubgs commented 5 years ago

I found this: https://wiki.saucelabs.com/display/DOCS/Using+the+Sauce+Labs+Jenkins+Plugin+with+Jenkins+Pipeline Which has an example:

node('mac') {
  sauce('36987f5a-62da-40ac-bbc0-583806f9df4d') {
    sauceconnect(useGeneratedTunnelIdentifier: true, verboseLogging: true) {
      sh 'env \| sort'
    }
  }
}

And describes this as:

The hashed key in the code block below is generated when you create a Sauce Labs credential in Jenkins from the Credentials dialog So that's something.

jakubgs commented 5 years ago

Okay, I've created a new global credential in our Jenkins, we'll see if it works.

jakubgs commented 5 years ago

Included a fix for this in #8511.

jakubgs commented 5 years ago

I merged #8511 but I still see weird failures in PR tests:

  File "/home/jenkins/workspace/end-to-end-tests/status-app-end-to-end-tests/test/appium/support/github_report.py", line 29, in build_html_report
    passed_tests_html = self.build_tests_table_html(passed_tests, run_id, failed_tests=False)
  File "/home/jenkins/workspace/end-to-end-tests/status-app-end-to-end-tests/test/appium/support/github_report.py", line 49, in build_tests_table_html
    html += self.build_test_row_html(i, test, run_id)
  File "/home/jenkins/workspace/end-to-end-tests/status-app-end-to-end-tests/test/appium/support/github_report.py", line 56, in build_test_row_html
    test_rail_link = TestrailReport().get_test_result_link(run_id, test.testrail_case_id)
  File "/home/jenkins/workspace/end-to-end-tests/status-app-end-to-end-tests/test/appium/support/testrail_report.py", line 120, in get_test_result_link
    test_id = self.get('get_results_for_case/%s/%s' % (test_run_id, test_case_id))[0]['test_id']
  File "/home/jenkins/workspace/end-to-end-tests/status-app-end-to-end-tests/test/appium/support/testrail_report.py", line 36, in get
    raise Exception('Failed request: %s' % rval['error'])
Exception: Failed request: No (active) test found for the run/case combination.

https://ci.status.im/job/end-to-end-tests/job/status-app-end-to-end-tests/3095/console

Based on this issue I found this might be due to passing wrong run_id:

It has "run_id":1454. Does run 1454 exist in your testrail?

https://github.com/mvndaai/testrail-promise/issues/13#issuecomment-331603266

jakubgs commented 5 years ago

The docs for the API call itself are here: http://docs.gurock.com/testrail-api2/reference-results#get_results_for_case

And they state that:

GET index.php?/api/v2/get_results_for_case/:run_id/:case_id

Returns a list of test results for a test run and case combination.

In TestRail, tests are part of a test run and the test cases are part of the related test suite. So, when you create a new test run, TestRail creates a test for each test case found in the test suite of the run. You can therefore think of a test as an “instance” of a test case which can have test results, comments and a test status.

jakubgs commented 5 years ago

This must be related to my change in: 37dc51a93d542135e67fe4f26767ebd0c2d114ae https://github.com/status-im/status-react/blob/37dc51a93d542135e67fe4f26767ebd0c2d114ae/test/appium/support/testrail_report.py#L33-L37

I changed the get() method to check for error in return value of the API call.

jakubgs commented 5 years ago

Created a fix which logs the error instead of raising an exception: #8522

jakubgs commented 5 years ago

I've merged #8522 and the test pass now(sometimes...), so I'm going to close this.