Adding execution time

[bluetechy]

Is it possible to display the execution time as well? This would help determine which tests are taking longer.

Although not necessary, but would be nice to have is the ability to define a maximum expected execution time in the config and then be able to compare it to the actual execution time so that we could see which tests are running slow or were made slow after making a change.

[dimitry12]

@bluetechy nice idea!

[powerc9000]

@bluetechy I will add the last few things in you've suggested today. It will include a large change to the format of the test file.

[powerc9000]

e2fb386

[bluetechy]

@powerc9000 This commit looks like it applies to #8, not #11.

[powerc9000]

Oops f247a50

[bluetechy]

@powerc9000 What is precisely being measured using ms?

1. MappingService
2. MappingService + Comparison
3. MappingService + Comparison + Upload to S3
4. Comparison
5. Comparison + Upload to S3
6. etc....

How can we define in the config file how to test the maximum number of milliseconds it takes to call MappingService so that we can spot-check what needs to be improved?

[powerc9000]

@bluetechy it pulls it off the queue, starts the timer makes the request and when the request comes back it stops the timer. So it is measuring 1. Do you want max execution time on the whole test group or just individual tests?

[bluetechy]

@powerc9000 I think it would make sense to do it as the individual test level. My thought is to still allow the test to complete even if it exceeds this maximum execution time so that we can also see if any differences exist, but just warn that it has exceeded max exe-time. Since max execution can be influenced by many factors, it is expected that this max exe-time may be exceeded from time-to-time.

[powerc9000]

@bluetechy

"testGroup": {
        "name": "SomeTest",
        "endpoint": "/example",
        "tests": [
            {
                "name": "t1",
                "description": "example",
                "ms": 100, //<--- add in MS
                "files": {
                    "inputs": {
                        "Name": "Name.input",
                        "Other": "Other"
                    },
                    "output": "output.json"
                }
            }
        ]

    }

[bluetechy]

@powerc9000 Great!

[bluetechy]

@powerc9000 I have added the "ms" property to OrderCalc. You trigger a refresh on a build on the "develop" branch in Travis-CI to verify that it works.

[powerc9000]

@bluetechy ms is now in gauntlet right now it just gives a warning if a test goes over time.

[powerc9000]

@bluetechy and you can see that here https://travis-ci.com/Unicity/OrderCalc.php/builds/20015497

[bluetechy]

@powerc9000 Awesome!!!! This will really help us fine tune our system.

[bluetechy]

I am reopening this issue to discuss an enhancement to this feature. Right now, when the call exceeds the value specified in the "ms" field Gauntlet reports just a warning. After talking this over with @dimitry12, it seems like we could improve our reporting by doing the following:

Assume a service has a test that is said to run in less than 75 ms as in this example:

"service%3DDoSomething": {
  "name": "service=DoSomething",
  "endpoint": "service=DoSomething",
  "tests": [
    {
      "name": "t1",
      "files": {
        "inputs": "input.json",
        "output": "output.json"
      },
      "ms": 75
    }
  ]
},

First solve for x:

(85 % / 100 %) = 75 ms / x ms

Next report as follows:

• When execution time is <= 75 ms, report a successful message (using green text).
• When execution time is > 75 ms but <= x ms, report a warning message (using yellow text).
• When execution time is > x ms, report an error message (using red text).

[dimitry12]

From my perspective it's single-feature anomaly detection with 15% threshold.

[powerc9000]

@bluetechy Alright this is currently in develop. I'll have to test it later, I just don't have the time today.

[powerc9000]

This is in gauntlet now