search

katalon-studio / katalon-testops

Manage automation results, like you test it manually
https://www.katalon.com/testops/
12 stars 16 forks source link

[Bug] Upload from Katalon Agent to TestOps fails to display results #144

Open Dwgthomson opened 3 years ago

Dwgthomson commented 3 years ago

Steps to Reproduce:

  1. Manually or automatically via schedule, run any test via a Plan (pulled down via github test project)
  2. test will run as expected
  3. Agent claims to have uploaded results to the S3 bucket
  4. Testops do not show results.

Expected behaviour: Testops should display the detailed error logs, once the test has been completed and logs uploaded.

Screenshots: image

image image

sonhuynhpham commented 3 years ago

Hi @Dwgthomson,

There was a recent problem with our system around the time of your execution. Please try to reimport those executions and let me know whether your problem is resolved or not.

Hope it helps!

Dwgthomson commented 3 years ago

image I now have a whole bunch of tests that are all in an importing status, with even less information than while it was running.

sonhuynhpham commented 3 years ago

Hi @Dwgthomson,

Thank you for the bug report. It seems like Katalon Studio Engine was unable to upload the execution report to TestOps for an unknown reason.

For further investigation:

Dwgthomson commented 3 years ago

@sonhuynhpham what is the best way to get you those files as I would prefer to not make those files available to the public, as they contain my entire test repository?

sonhuynhpham commented 3 years ago

Hi @Dwgthomson,

You can send me the attachments to this email address: son.pham@katalon.com or a private message on Katalon Forum to my profile.

Thank you!

ghost commented 3 years ago

Hi, I've been attempting to get an update on this issue and came across 4 similar issues. All the other issues don't have a recent update either. At the risk of spamming each bug with these details, I'm going to mention the similar bugs in hopes of getting an update on this issue and maybe merging this issue into one. Lastly, the common thread in this problem is that it doesn't always fail. Issue 137 states there is an issue with the API upload using AWS and the response was error code 500.

My goal is to get an update please. Ty!

https://github.com/katalon-studio/katalon-testops/issues/80 https://github.com/katalon-studio/katalon-testops/issues/108 https://github.com/katalon-studio/katalon-testops/issues/137 https://github.com/katalon-studio/katalon-testops/issues/144

Dwgthomson commented 3 years ago

@jimsears, From our own testing, it seems that there is a file size limit on the report that gets sent back to Katalon Analytics, if your report is large, it will display no results. Our current idea/workaround is chopping the test case up into smaller chunks as we had all tests running to determine if there were any faults. It is a pain to do this, as we have no idea how much we need to do this by to make it work. As when there are faults, the error log is larger, which is making the report larger, so the upload fails. It would be nice if there was an official confirmation on this size restriction, or potentially chunking the upload to avoid the issue, but oh well...

But yes there has been silence on this issue for us.

ghost commented 3 years ago

@Dwgthomson , thanks for the info. I can also confirm the test-runs that fail to upload are the larger executions so that does help me. I can see if the smaller test-runs will work. Cheers.

ghost commented 3 years ago

FYI to those interested or following this issue

Dwgthomson commented 3 years ago

We were paying for Katalon but have since re-platformed due to multiple blockers like this one, and the painful paywalls on features. sorry I couldn't help more.

ghost commented 3 years ago

Hi Daniel,

It’s all good. If you don’t mind me asking, what tool/platform did you move too?

Sincerely, Jim Sears Sr Automation Engineer

From: Daniel @.> Sent: Thursday, June 10, 2021 7:00 PM To: katalon-studio/katalon-testops @.> Cc: Jim Sears @.>; Mention @.> Subject: Re: [katalon-studio/katalon-testops] [Bug] Upload from Katalon Agent to TestOps fails to display results (#144)

We were paying for Katalon but have since re-platformed due to multiple blockers like this one, and the painful paywalls on features. sorry I couldn't help more.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/katalon-studio/katalon-testops/issues/144#issuecomment-859163005, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APNJU35P6YRSEAZVULHIKXLTSFGY7ANCNFSM4U2H72UA.

ghost commented 3 years ago

Hi all,

KS 8.0.5 has fixed two issues with report upload. 1) Special characters causing null pointer exception 2) Loading reports into TestOps. It seems to work for me but I would be interested to know if you still have issues.

~Jim