lczub / TestLink-API-Python-client

A Python client to use the TestLink API
105 stars 63 forks source link

Which TestLink-API-Python-client API to use to bulk upload test results in XML? #59

Closed kennethwork99 closed 8 years ago

kennethwork99 commented 9 years ago

This is not an issue but a question on how to use TestLink-API-Python-client to bulk upload test results in XML into TestLink. Generated a results.xml file and manually able to upload to testlink via "Import XML Results" in Test Execution page. It shows correct Status, Date and Exec (min) values so the results.xml should be in correct format.

In TestLink-API-Python-client-0.6.1, which API should be used to upload results.xml? At first I thought uploadExecutionAttachment might work but what value to use for executionid? I tried using some bogus integer and string value but failed each time.

testlink.testlinkerrors.TLResponseError: 6004: (uploadExecutionAttachment) - Invalid Foreign Key ID[1] Table[executions]

Then I tried reportTCResult and that worked but cannot get Date to be the supplied timestamp string and Exec (min) is always blank. I tried using customfields and passed in a dictionary:

customfields = dict(timestamp=starttime2, execution_type="2", execution_duration=str(minutes)) No error was reported but those fields did not show up in the Test Execution page correctly.

I prefer to upload the results.xml file if such an API is available. If not, any help to get the Exec (min) and Date as test execution timestamp via reportTCResult is appreciated. Thanks.

Server informations Version : 1.9.10 Testlink API Version: 1.0

lczub commented 9 years ago

Hello kennethwork99,

thanks for interesting questing. I'm sorry, but currently I could not help you to import the result.xml file via the API, but maybe I have some hints for your customfields.

  1. Unfortunately, TL itself does not provide an api method to import result.xml files. (see TL Githup xmlrpc.class.php Line 6990).
  2. reportTCResult does not provide attributes to change directly the timestamp or exec duration (see TL Githup xmlrpc.class.php Line 2100)
  3. Intention of uploadExecutionAttachment is, that you add log files or screenshot to your execution result. You could add your result.xml file with this, but it will not be evaluated. Examples how to add such attachment see TestLinkExample.py Line 483

So now hopefully the good news. Here is an example, how you could import timesstamps as values for customfields via reportTCResults

import datetime
newResult = myTestLink.reportTCResult(None, aTPlanID, aBuildName, 'f', 'result one', 
                                      testcaseexternalid=aTCaseFullExID,
                                      platformname='Small Birds',
                                     customfields={'cf_tc_ex_string' : 'a custom exec value set via api',
                                                   'CF_EXEC_TIME' : 2.2,
                                                   'cf_tc_ex_numeric' : 111,
                                                   # 2014-04-28
                                                   'cf_tc_ex_date' : datetime.datetime(2015,4,28).timestamp(),
                                                   # 2014-04-28 21:10:51
                                                   'cf_tc_ex_datetime' : datetime.datetime(2014,4,28,21,10,51).timestamp()} )
print ("reportTCResult", newResult )

this will return

reportTCResult [{'id': '1644', 'operation': 'reportTCResult', 'customfieldstatus': True, 'status': True, 'overwrite': False, 'message': 'Success!'}]

and this id is the execution id, you need to upload execution attachments

Hope this helps a little bit.

Regards Luiko

kennethwork99 commented 9 years ago

Hi Luiko,

Thank you very much for your reply. This is a very nice library in Python to work with TestLink.

I want to report the test duration in "Exec (min)" via reportTCResult but I still cannot get it to work.

Can you show me what changes I need to make? Note there were no errors reported by reportTCResult and the test result were updated but "Exec (min)" is still blank. Note I had also tried using the full_external_id but no difference in the result.

report = self.tls.reportTCResult(testcase['tc_id'], testplanid, build, status_code[status['@status']], "test note", user="pandas", customfields=customfields, devKey=self.devkey)

2015-07-22 17:23:21,319 DEBUG pz.testlink:258 ProcessTestResult tc_id 20835 testplanid 21007 build 6.0.0.5.8132 status p notes test note user pandas customfields {'CF_EXEC_TIME': 4.0}

2015-07-22 17:23:21,659 DEBUG pz.testlink:272 ProcessTestResult 111 report [{'customfieldstatus': True,

'id': '112069',

'message': 'Success!',

'operation': 'reportTCResult',

'overwrite': False,

'status': True}]

kenneth

On Wed, Jul 22, 2015 at 2:30 PM, Luiko Czub notifications@github.com wrote:

Hello kennethwork99,

thanks for interesting questing. I'm sorry, but currently I could not help you to import the result.xml file via the API, but maybe I have some hints for your customfields.

1.

Unfortunately, TL itself does not provide an api method to import result.xml files. (see TL Githup xmlrpc.class.php Line 6990 https://github.com/TestLinkOpenSourceTRMS/testlink-code/blob/testlink_1_9/lib/api/xmlrpc/v1/xmlrpc.class.php#L6990 ). 2.

reportTCResult does not provide attributes to change directly the timestamp or exec duration (see TL Githup xmlrpc.class.php Line 2100 https://github.com/TestLinkOpenSourceTRMS/testlink-code/blob/testlink_1_9/lib/api/xmlrpc/v1/xmlrpc.class.php#L2100 ) 3.

Intention of uploadExecutionAttachment is, that you add log files or screenshot to your execution result. You could add your result.xml file with this, but it will not be evaluated. Examples how to add such attachment see TestLinkExample.py Line 483 https://github.com/lczub/TestLink-API-Python-client/blob/master/example/TestLinkExample.py#L483

So now hopefully the good news. Here is an example, how you could import timesstamps as values for customfields via reportTCResults

import datetime newResult = myTestLink.reportTCResult(None, aTPlanID, aBuildName, 'f', 'result one', testcaseexternalid=aTCaseFullExID, platformname='Small Birds', customfields={'cf_tc_ex_string' : 'a custom exec value set via api', 'CF_EXEC_TIME' : 2.2, 'cf_tc_ex_numeric' : 111,

2014-04-28

                                               'cf_tc_ex_date' : datetime.datetime(2015,4,28).timestamp(),
                                               # 2014-04-28 21:10:51
                                               'cf_tc_ex_datetime' : datetime.datetime(2014,4,28,21,10,51).timestamp()} )

print ("reportTCResult", newResult )

this will return

reportTCResult [{'id': '1644', 'operation': 'reportTCResult', 'customfieldstatus': True, 'status': True, 'overwrite': False, 'message': 'Success!'}]

and this id is the execution id, you need to upload execution attachments

Hope this helps a little bit.

Regards Luiko

— Reply to this email directly or view it on GitHub https://github.com/lczub/TestLink-API-Python-client/issues/59#issuecomment-123872772 .

lczub commented 9 years ago

Hello Kenneth,

it seams that CF_EXEC_TIME is no longer supported by TL since 1.9.7 (see TL GitHub README - 10. Changes regarding - 1.9.7

I played a little bit with my TL installation and find a way to extend reportTCResult, so that I could update the execution_duration. If you have write access to your TL installation, you could try out following changes in

case a - insert TC results with execution_duration

  protected function _insertResultToDB($user_id=null)
  {
  ...
// start mod LC - INSERT execution_duration
/* orig 1.9.14 dev

    $query = "INSERT INTO {$this->tables['executions']} " .
             " (build_id, tester_id, execution_ts, status, testplan_id, tcversion_id, " .
             " platform_id, tcversion_number," .
             " execution_type {$notes_field} ) " .
             " VALUES({$build_id},{$tester_id},{$db_now},'{$status}',{$testplan_id}," .
             " {$tcversion_id},{$platform_id}, {$version_number},{$execution_type} {$notes_value})";
*/
    $duration_field="";
    $duration_value="";  

      if( isset($this->args["execution_duration"]) )
      {
    $duration_value = ",{$this->args['execution_duration']}"; 
        $duration_field = ",execution_duration";   
      }
    $query = "INSERT INTO {$this->tables['executions']} " .
             " (build_id, tester_id, execution_ts, status, testplan_id, tcversion_id, " .
             " platform_id, tcversion_number," .
             " execution_type {$notes_field} {$duration_field}) " .
             " VALUES({$build_id},{$tester_id},{$db_now},'{$status}',{$testplan_id}," .
             " {$tcversion_id},{$platform_id}, {$version_number},{$execution_type} {$notes_value} {$duration_value})";
// end mod LC - INSERT execution_duration

    $this->dbObj->exec_query($query);
    return $this->dbObj->insert_id($this->tables['executions']);    
  }

With this chance following call should insert an execution result with an duration:

newResult = myTestLink.reportTCResult(None, aTPlanID, aBuildName, 'f', 'result one', 
                                      testcaseexternalid=aTCaseFullExID, overwrite=False,
                                      platformname='Small Birds', execution_duration=4.1)

case b - update TC results with execution_duration

  protected function _updateResult()
  {
    ...
// start mod LC - UPDATE execution_duration
/* orig 1.9.14 dev

      $sql = " UPDATE {$this->tables['executions']} " .
             " SET tester_id={$tester_id}, execution_ts={$db_now}," . 
             " status='{$status}', execution_type= {$execution_type} " . 
             " {$notes_update}  WHERE id = {$exec_id}";
*/      
      $duration_update = '';
      if( isset($this->args["execution_duration"]) )
      {
    $duration = $this->args["execution_duration"]; 
        $duration_update = ",execution_duration={$duration}";   
      }

tlog("duration_update {$duration_update}",'INFO');
      $sql = " UPDATE {$this->tables['executions']} " .
             " SET tester_id={$tester_id}, execution_ts={$db_now}," . 
             " status='{$status}', execution_type= {$execution_type} " . 
           " {$notes_update} {$duration_update} WHERE id = {$exec_id}";
// end mod LC - UPDATE execution_duration

            $this->dbObj->exec_query($sql);
      }
    return $exec_id;
  }  

With this chance following call should update the last execution result with an duration:

newResult = myTestLink.reportTCResult(None, aTPlanID, aBuildName, 'f', 'result one', 
                                      testcaseexternalid=aTCaseFullExID, overwrite=True,
                                      platformname='Small Birds', execution_duration=4.1)

If this helps, you could use it to report a changes request for TestLink itself via http://mantis.testlink.org/my_view_page.php

Hope this help, Regars Luiko

lczub commented 9 years ago

Hello Kenneth, fyi, with the next TestLink Release 1.9.14, reportTCResult excepts a new optional parameter execduration to set the execution duration.

Details see http://mantis.testlink.org/view.php?id=7209

Regards Luiko

kennethwork99 commented 9 years ago

Hi Luiko,

Thank you very much. This is great. One other request might be useful to add to TestLink-API-Python-client is to report when the test started. Note this information is available in our test log and our QA process is to upload the result after all the test completed. The current implementation is adding the Date when the result is uploaded. This request, if accepted, will help users who perform post processing of result instead of uploading results as the test complete. I hope you will consider making this enhancement if it is not already available.

kenneth

On Sat, Aug 1, 2015 at 11:31 AM, Luiko Czub notifications@github.com wrote:

Hello Kenneth, fyi, with the next TestLink Release 1.9.14, reportTCResult excepts a new optional parameter execduration to set the execution duration.

Details see http://mantis.testlink.org/view.php?id=7209

Regards Luiko

— Reply to this email directly or view it on GitHub https://github.com/lczub/TestLink-API-Python-client/issues/59#issuecomment-126943816 .

lczub commented 9 years ago

Hello Kenneth, your second requirement also needs as a base an extention on the testlink server site. So please try to describe your requirement as a change request against testlink itself. Maybe the requirement should be, that the changes via xml import should also possible via xmlrpc api and the only missing part is now the start time. When the server has implement it, it will be available on the client site. Luiko

lczub commented 9 years ago

Hello Kenneth,

for your timestamp requirement, I created following pull request against TestLink itself. https://github.com/TestLinkOpenSourceTRMS/testlink-code/pull/11

Luiko

kennethwork99 commented 9 years ago

Hi Luiko,

Thank you for the follow up and making the TestLink-API-Python-client more and more useful.

kenneth

On Mon, Aug 24, 2015 at 11:09 AM, Luiko Czub notifications@github.com wrote:

Hello Kenneth,

for your timestamp requirement, I created following pull request against TestLink itself. TestLinkOpenSourceTRMS/testlink-code#11 https://github.com/TestLinkOpenSourceTRMS/testlink-code/pull/11

Luiko

— Reply to this email directly or view it on GitHub https://github.com/lczub/TestLink-API-Python-client/issues/59#issuecomment-134321445 .

lczub commented 8 years ago

fixed with commit 4b6abbc in combination with TL 1.9.14 release, published on 2015/09/19