CLI runs slow in CI/CD pipeline #1399

Closed azplanlos closed 10 months ago

Describe the bug TC CLI performs slow in our GitLab CI/CD pipeline, runtime is approx. 3-4 times slower than running requests via CLI on the local machine against the same endpoint when running a medium sized collection. Running 144 requests takes around 25 minutes in the pipeline while running in a couple of minutes locally.

Platform:

OS: linux

Are you using the free version/paid version/trial: paid version

Hi @azplanlos Thanks for reporting the issue.

What differences have you noticed in the Html report generated from local CLI and Gitlab CI/CD?

If you can share any further details of the issue, which might help to resolve the issue.

The execution times look approx. the same. Interesting thing is, if I add up the request execution times I get approx. 3-4 Minutes of time but total execution time is 25 Minutes.

Did you add a delay in between requests?
Do you have scripts that take longer time to execute?
Do you check retry logic that is working correctly?

No, there is no delay between requests and only simple pre-request and post-request scripts. No retry used.

What are duration time values in the Html Report for Local and Ci/CD?

local: Total Duration: 78762ms CI/CD: Total Duration: 1968393ms

same collection, run command: tc --col --log 8 --report cli,html --var-data url=...,token_endpoint=...,client_id=...,client_secret=... --ws --env

Something is an issue with Gitlab CI/Cd runtime.

Is it possible for you to create a sample collection using Test url - https://www.thunderclient.com/welcome and some sample test scripts that can re-produce the issue, so I can test it myself.

Hi @azplanlos the issue could http library used. Can you change to axios libary.

In your .vscode folder-> open settings.json and add below line

"thunder-client.httpLibrary": "axios"

Can you please test with axios and let me know the result

unfortunately this doesn't make a difference.

Thanks for confirmation. Did you check with other teams in your company do they have similar issue?

Hi @rangav, we did a little more digging and are able to reproduce the behaviour by running thunder client in a docker container and limiting the CPU ressources to the values used in our pipeline. First finding here is that using a node.js base image with alpine instead of Ubuntu runs much faster, but anyway maybe you have some more tooling to see how the performance could be improved more.

I used this Dockerfile for creating an alpine based image:

FROM node:21-alpine

RUN npm i -g @thunderclient/cli

VOLUME ["/home/tests"]

WORKDIR "/home/tests"
ENTRYPOINT ["/usr/local/bin/tc"]
CMD ["--help"]

and this docker-compose to setup our service and run the container with limited ressources on my Macbook:

version: "3.8"

services:
  testrunner:
    image: tc-runner:latest
    command: "--col 'collection_name' --env 'environment' --report 'cli,html' --var-data 'url=...' --log 0"
    volumes:
      - ./tests:/home/tests
    depends_on:
      - ...
    deploy:
      resources:
        limits:
          cpus: '0.150'
          memory: 375M
        reservations:
          cpus: '0.150'
          memory: 375M

My findings are: runtime for our collection locally is around 55 seconds, in docker with Ubuntu 16 minutes, with alpine and Debian bookworm approx 7-8 minutes.

Just a couple of additional notes to our findings yesterday: We were able to speed up the execution of one of our shorter test collections by increasing the CPU limit to approx 0.8, higher CPU values made the execution slower again (approx. 2 times). Also this approach didn't help to speed up the execution of a larger test collection (90 requests). But we found that these delays seem to be unrelated to prescript execution as throwing an exception from the prescript made the collection run really fast as the actual requests are not executed in this case.

Thanks for confirmation. Did you check with other teams in your company do they have similar issue?

Yes we are exchanging information on this topic and the other teams are following this ticket as well. Seems to be a general issue as no-one could come up with a solution for this yet.

@rangav are you aware of our new findings on this issue? Just asking as I didn't receive a reply for you on this one...

These are the findings from our CLI reports.

The time taken on the GitHub Actions is slower by 10x compared to local.
Total Requests in Collection: 54
Time is from the HTML report

Local - M2 Mackbook Pro (16GB RAM) - 3680ms Github Actions - 37768ms

**The issue could be with Device specs in the GitLab/Docker CLI

I have tested the same collection in Old Macbook Pro(2013, 8GB RAM). Which is 5x slower than the latest M2 Macbook.

OLD Macbook Pro (2013, 8GB RAM) - 14554ms

So the CLI execution time is dependent on device specifications from the above observations.

Hi @azplanlos I am closing this issue, as the execution time is dependent on the system specs.

Please increase the specs it will run faster.

If you have further questions, please continue discussion here

Hi @rangav, as this is still a big issue for all our teams I would appreciate if you could investigate further how the performance of the CLI could be optimized in CPU limited environments. I still don't see why Thunder Client would need more resources than our actual service that is tested. Thank you very much!

Hi @azplanlos sure will look into it again and see if there is anyway we can increase the speed of execution.

The CLI needs to load and run its code and all the libraries used to build it. So the CPU and RAM resources will have an effect on the speed of execution.

Thank you. Just found that using Node JS 16 instead of 21 also has a huge effect on performance of the TC CLI (approx. 50%!) while switching the http library from got to axios has no effect at all.

Thanks @azplanlos for the findings about performance improvements on node 16.

CLI recommend version is Node 16 or above

does it solve the issue now?

no unfortunately this is not solved by this finding. Still performance is not as expected.

Thanks for confirmation, Will investigate and get back to you

I did some tests - below are observations:

thunderclient cli - there is no minmum node version specified in the package.json, it is built on the lastet node version node 21 on my mac.
In Github actions, I tested with differenet node version 16, 18, 20. Using node version 20 is giving better performance results.

Using node version 20 is 4x faster than node 16 or 18

We have updated the CLI and Extension to show execution timings for Pre Request, Pre Script, and Post Script in the JSON report as below image.

Can you please update CLI to v1.11.2 and generate a JSON report and share the timings?

@azplanlos do you still have this issue?

thunderclient / thunder-client-support

CLI runs slow in CI/CD pipeline #1399