myzhan / locust4j

Locust4j is a load generator for locust, written in Java.
MIT License
81 stars 30 forks source link

Failure occurrences do not seem to be reflecting the actual number #31

Closed harkiranjeetkaur closed 2 years ago

harkiranjeetkaur commented 3 years ago

Hi,

The occurrences of the failure is giving an irrelevant number in the results. The way we are grabbing this variable is as following in the Locust master python class:

stats = environment.stats
    items = []
    entries = stats.entries
    for name, method in entries.keys():
        entry = entries.get((name, method))
        item = {"name": name,
                "method": method,
                "avg": round(entry.avg_response_time, 0),
                "avg_size": round(entry.avg_content_length, 0)
                }
        items.append(item)

    result["items"] = items

    errors = []
    for stats_error in stats.errors.values():
        error = {
            "name": stats_error.name,
            "method": stats_error.method,
            "error": stats_error.error,
            "occurrences": stats_error.occurrences
        }
        errors.append(error)

Task class

We are appending a unique random id to each execute() block, so that no two runs are similar.

public void execute() throws Exception {

        String workflowId = RandomStringUtils.randomNumeric(8);
        boolean failure = false;
        long category;

        User user = userQueue.pop();
        userQueue.addLast(user);
        String userAccessToken;
        .
        .
        .

//API call
       Response recent = EngagementController.getRecentEngagements(userAccessToken, 0);
        if ((recent.getStatusCode() != HttpStatus.SC_OK) && !failure) {
            failure = true;
            LocustUtil.recordFailure("GET " + workflowId, getName() + "/engagement-management/engagements/recent?size=3", recent,
                    userId, "recentEngagementsFailure_" + workflowId);
        }
}

Locust util record failure method

public static void recordFailure(final String requestType, final String taskName,
                                     final Response response, String userId, final String sampleWorkflowId) {
        Locust.getInstance().recordFailure(
                requestType,
                taskName,
                response.getTime(),
                "Response: StatusCode: " + response.getStatusCode() + "\n" +
                        "Header: Date: " + response.getHeader("date") + " " +
                        "Response Body: " + response.getBody() + " " +
                        "for userId: " + userId);
    }

And also that unique id is appended to the failure record. Ideally no two failures should have similar method or name. How is occurrence variable getting its value then?

failure occurences

ferristseng commented 2 years ago

I'm seeing this issue as well. I think this is because the occurrences are aggregated in Locust, and the occurrences counter is never reset. See:

https://github.com/locustio/locust/blob/abdf0dd28ae418821afb7e0594b7640c6e3f295f/locust/stats.py#L720-L731

I think the solution is to reset the occurrences counter to 0 before/after a message is sent to the master node.