Open jwadolowski opened 7 years ago
tomekmwagner
commented
7 years ago Hi Kuba,
Thank you very much for your plenteous contribution and time to write down these painpoints.
We will analyze these things very carefully and most probably expand our logging system to give more meaningful messages.
Thanks, Tomek
jwadolowski
commented
7 years ago Here's yet another mysterious error
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building xxxxx 1.0.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- aet-maven-plugin:2.0.1:run (default-cli) @ aet ---
[INFO] Successfully connected to tcp://10.x.y.z:61616
[INFO] HV000001: Hibernate Validator 4.3.2.Final
[INFO] HV000002: Ignoring XML configuration.
[INFO] CorrelationID: xxxxx-xxxxx-uat-suite-1489587812901
[INFO] ********************************************************************************
[INFO] ********************** Job Setup finished at 14:23:34.265.**********************
[INFO] *** Suite is now processed by the system, progress will be available below. ****
[INFO] ********************************************************************************
[INFO] [14:23:34.341]: COLLECTED: [success: 0, total: 60] ::: COMPARED: [success: 0, total: 0]
[WARNING] Report will be generated after timeout - some results might be missing!
[INFO] [14:25:35.524]: COLLECTED: [success: 0, total: 60] ::: COMPARED: [success: 0, total: 0]
[INFO] Received report message: FinishedSuiteProcessingMessage{correlationId=xxxxx-xxxxx-uat-suite-1489587812901, status=OK, errors=[Failed to generate reports because it took too much time!]}
[INFO] Report is available at https://zzzzzzzz.yyyyyyyyyy.com/report.html?company=xxxxx&project=xxxxx&correlationId=xxxxx-xxxxx-uat-suite-1489587812901
[ERROR] Failed to obtain xUnit report from: http://10.x.y.z:8181/xunit?company=xxxxx&project=xxxxx&correlationId=xxxxx-xxxxx-uat-suite-1489587812901
[INFO] Suite processing finished.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:04 min
[INFO] Finished at: 2017-03-15T14:27:36+00:00
[INFO] Final Memory: 14M/137M
[INFO] ------------------------------------------------------------------------For every single page the following error is displayed:
Hey,
quite recently I encountered a bunch of AET issues. Unfortunately none of them was easy to troubleshoot and fix, mostly due to not enough feedback on client side. That's rather frustrating experience for someone who considers AET as a platform and just executes a bunch of test suites against predefined endpoint.
I have 3 stories that prove above standpoint.
First story
I have 2 environment in my project. Both of them have dedicated test suites and we used to run them on a quite regular basis. Everything was working fine on Thursday, we did app deployment and rerun AET Jenkins job the next day. It failed with the following error:
I opened the report, but for all the pages there was a message that said something about
Unknown error(w/o any description at all), but apparently all the pages were visited because JS and status code errors were present in the report (all confirmed by log entries on the server).Thankfully we had an AET engineer on board and he found out something was wrong with data collected in MongoDB. We made a few attempts to take some pattern IDs directly out of the database and injected that into report app URL. Unfortunately we ended up with the following in every single case:
All in all we decided to remove all the data related to that suite from MongoDB (that wasn't that important for us at that time) and rerun the test - all was fine this time. It's hard to say what exactly went wrong, but based just on client logs and w/o deep understanding of what's going on under the hood it was impossible to track that bug down. There was no information about MongoDB whatsoever in the log.
It's worth to mention that MongoDB itself was 100% healthy and had no downtime or availability issues.
Second story
Recently we made a few changes related the way HTTPS gets terminated and how we pass on HTTP traffic from load balancer to Apache web server. Unfortunately due to webserver misconfiguration we accidentally introduced basic auth dialog on certain paths.
The change was implemented and we run AET as usual, but we ended up with this:
As you can see there was information only about some timeout. Based on the output itself it's hard to say where exactly that timeout happened and what might be the root cause. As in previous story there were
Unknown errorson every single page. I wish there's a message about basic auth dialog somewhere to speed up troubleshooting.There was also a similar story with the identical effect - basic auth credentials were changed, but no one updated
Authorizationheader in the XML file, so the entire test failed. Again - no information about 401 or any other indication that something's wrong with the request itself.Third story
By default Apache Karaf comes with 5 collector workers. We stumbled upon a situation where we had to decrease that value.
The background story behind that was, as usual, triggered by failed Maven AET build:
URL that points to xUnit report returned 500 error code and the following body:
It didn't say much what's going on. After a session with AET engineer we figured out that the server we run Apache Karaf on wasn't powerful enough to handle 5 Firefox browsers simultaneously, so we decreased that value to 3 and magically it started to work.
As in previous stories there was no indication what's going on and without support from AET engineer it's almost impossible to track the problem down on your own.
As you can see it can be quite tricky to troubleshoot and resolve issues you may encounter during your day-to-day work with AET.
Is it possible to make AET client logs more descriptive to give users some hints what actually went wrong? I believe there's a huge room for improvement in this area.
I'm quite convinced that without help of AET engineers (btw huge thanks for that!) I wouldn't be able to fix those issues.
Thanks, Kuba