We are still seeing issues with results not being updated in testrails

billylam / wdio-wdiov5testrail-reporter

https://www.npmjs.com/package/wdio-wdiov5testrail-reporter

MIT License

6 stars 8 forks source link

We are still seeing issues with results not being updated in testrails #7

Closed hdesai-dave closed 3 years ago

hdesai-dave commented 3 years ago

When we run smoke tests in parallel, the results are being updated just fine since it is a smaller subset. 8 spec files, 20 some test cases
When we run full regression suite in parallel results are not updated in testrails: 200+ specs, 400+ test cases.

Any help appreciated debugging it locally or otherwise

billylam commented 3 years ago

TestRail just changed their API for certain calls. Are you using the newest version (1.0.29)?

hdesai-dave commented 3 years ago

Yes @billylam and our previous reporter also broke after new testrail update hence trying yours.

billylam commented 3 years ago

Hmm, ok. How many actual cases are marked in the testrail report? And if you make a call to yourdomain.testrail.com/index.php?/api/v2/get_cases/[yourprojectid]&suite_id=[yoursuiteid], are the cases that are marked the ones present in that api call and the ones that are not marked the ones that are not present?

Also, are you setting a value for the strictCaseMatching config option? If you are, try deleting that key or setting it to true.

hdesai-dave commented 3 years ago

I get 0 results updated to testrails. I do not use strictCaseMatching.

billylam commented 3 years ago

Do you see any error logging thrown by the reporter? Something like

"Server responded to yourdomain.testrail.com/index.php?/api/v2/add_results_for_cases/with status code 400:"

Also try strictCaseMatching: false. If that works, the issue is that you have a test case mismatch.

hdesai-dave commented 3 years ago

No errors in logs!

billylam commented 3 years ago

Please try the other options listed above. They will show if you have an issue with test cases in webdriverio not matching those in TestRail, which will cause this type of error.

Also try strictCaseMatching: false. If that works, the issue is that you have a test case mismatch.

Hmm, ok. How many actual cases are marked in the testrail report? And if you make a call to yourdomain.testrail.com/index.php?/api/v2/get_cases/[yourprojectid]&suite_id=[yoursuiteid], are the cases that are marked the ones present in that api call and the ones that are not marked the ones that are not present?

hdesai-dave commented 3 years ago

Yeah strictCaseMatching: false worked. How do i find the mismatch now 🤦‍♂️

hdesai-dave commented 3 years ago

Thanks for your prompt response!