search

microsoft / jmeter-performance-analyzer-devops-extension

This task enables to run Performance testng using Apache Jmeter, Analyze report and post results. This task uses Apache Jmeter 5.5 and expects a valid parametrized JMX File, Any input Files, and property file for JMX. The task runs the jmx files according to the configured values in JMX and publishes the result to build artifacts and $web of your storage container. Provides control to fail pipeline if JMeter test fails More details at https://marketplace.visualstudio.com/items?itemName=id-az-pipeline.jmeter-perf-analyzer. You need to enable static hosting in the storage container in order to be able to view html results.
Other
10 stars 5 forks source link

Changing 'jmeterFolderName' parameter to custom one cause an error #20

Open ivan-zaitsev opened 1 year ago

ivan-zaitsev commented 1 year ago

Actually, I was investigation another problem when I have in one repo several JMeter tests, for each test I have separate pipeline, but when I run these pipelines see PerfAnalyzerReport and PerfAnalyzerLogs only for one project. I think it could happen because ' jmeter-performance-analyzer' move JMX files to the same folder '/home/vsts/work/1/s/apache-jmeter-5.5/bin'. I wanted to change the folder to check if it will solve the issue, but I found another one, changing 'jmeterFolderName' parameter to different folder cause an error.

Log:

18-Nov-2022 13:38:49:171 +0000  - /home/vsts/work/1/s  - Current Working directory: /home/vsts/work/1/s
18-Nov-2022 13:38:49:175 +0000  - /home/vsts/work/1/s  - JMETER_URL https://dlcdn.apache.org/jmeter/binaries/apache-jmeter-5.5.tgz
18-Nov-2022 13:38:49:175 +0000  - /home/vsts/work/1/s  - JMETER_FILE_Folder custom-jmeter-folder-name
18-Nov-2022 13:38:49:175 +0000  - /home/vsts/work/1/s  - JMETER_BIN_Folder custom-jmeter-folder-name/bin
18-Nov-2022 13:38:49:175 +0000  - /home/vsts/work/1/s  - JMETER_ABS_BIN_Folder /home/vsts/work/1/s/custom-jmeter-folder-name/bin
18-Nov-2022 13:38:49:175 +0000  - /home/vsts/work/1/s  - Current Working directory: /home/vsts/work/1/s
18-Nov-2022 13:38:49:175 +0000  - /home/vsts/work/1/s  - Start Downloading JMeter Binary
18-Nov-2022 13:38:49:176 +0000  - /home/vsts/work/1/s  - Downloading File: https://dlcdn.apache.org/jmeter/binaries/apache-jmeter-5.5.tgz to location: apache-jmeter.tgz
18-Nov-2022 13:38:49:867 +0000  - /home/vsts/work/1/s  - Download https://dlcdn.apache.org/jmeter/binaries/apache-jmeter-5.5.tgz Completed to :apache-jmeter.tgz
18-Nov-2022 13:38:49:868 +0000  - /home/vsts/work/1/s  - Completed Downloading JMeter Binary
18-Nov-2022 13:38:49:871 +0000  - /home/vsts/work/1/s  - Start Unzipping JMeter Binary
18-Nov-2022 13:38:51:953 +0000  - /home/vsts/work/1/s  - Completed Unzipping JMeter Binary
18-Nov-2022 13:38:51:953 +0000  - /home/vsts/work/1/s  - Custom Plugins not enabled
##[error]Error: ENOENT: no such file or directory, chdir '/home/vsts/work/1/s/custom-jmeter-folder-name/bin'
18-Nov-2022 13:38:51:953 +0000  - /home/vsts/work/1/s  - Error: ENOENT: no such file or directory, chdir '/home/vsts/work/1/s/custom-jmeter-folder-name/bin'
18-Nov-2022 13:38:51:954 +0000  - /home/vsts/work/1/s  - Please note this extension is in beta version. We will be fixing the issues as reported and hence it is important for us to be able to analyze logs and fix the issue. If you encounter any issue please create a github issue here https://github.com/microsoft/jmeter-performance-analyzer-devops-extension/issues
##[error]ENOENT: no such file or directory, chdir '/home/vsts/work/1/s/custom-jmeter-folder-name/bin'
5-k commented 1 year ago

Hi @ivan909020

Thankyou for highlighting a valid use case. However I would like to bring to your attention that modifying the JMETER_FOLDER_NAME won't help your use case.

From your statement, I assume you have separate pipeline for each test. If that is the case, running multiple pipeline should create their own virtual env. Since one instance of perf analyzer runs only 1 test JMX file, it should be able to run the test, publish it uniquely. You can modify the log and report folder name, along with the published artifact name.

In that case you can modify the variable JMeter Custom Unzipped Folder Name (jmeterCustomUnzippedFolderName in yaml). This is a new variable added to accommodate your request. Please do not modify the variable JMeter Folder Name (jmeterFolderName) unless you modify the JMeter source URL. This variable defines what folder name to expect inside the zip file downloaded from JMeter and not the folder name you want this to. Ideally running seperate pipeline should create seperate virtual env and not interfere with each other's runs.

In this case you should create different stage for each instance of PerfAnalyzer task and modify the following 4 variables

I have updated the default value for these variables and concatenated the stage display name. Hence for each run in a new stage, you will have reports published to a different folder, unique by stage name. Please ensure to keep the stage name different for multiple instance of perf analyzer running in same pipeline.

Let me know if this resolves your issue , else we can discuss further

ivan-zaitsev commented 1 year ago

Hello @5-k

Thank you for detailed answer. Interesting that folder name depends on JMeter source URL archive name. Having two variables JMeter Custom Unzipped Folder Name and JMeter Folder Name adding a little bit complexity. Don't you think that when you change JMeter Folder Name, the JMeter source URL unzip location also should be changed, because they depend on each other.

I will try to use new JMeter Custom Unzipped Folder Name and let you know, thank you again.

5-k commented 1 year ago

Hi @ivan909020

I have not published it yet, once done will let you know

5-k commented 1 year ago

Hi @ivan909020

I have deployed the newer version. In the advance section:

removed JMeter Folder Name kept JMeter Custom Unzipped Folder Name

and JMeter Custom Unzipped Folder Name is defaulted to a folder name which can be modified for each run in each pipeline.

For your use case, please modify the following variable in each pipeline , and for each task.

JMeter Log Folder Name (jmeterLogFolder) JMeter Report Folder Name (jmeterReportFolder) Artifact Name for Apache Jmeter Logs (artifactNameLog) Artifact Name for Apache Jmeter Reports (artifactNameReport) JMeter Custom Unzipped Folder Name

ivan-zaitsev commented 1 year ago

@5-k

Hello, thanks. Which task version should I use? Currently I am using - task: perfanalyzer@1 and I don`t see there new 'jmeterCustomUnzippedFolderName' variable.

5-k commented 1 year ago

Hi @ivan909020

Version 1.7.35 has been uploaded and now should have been auto downloaded to your devops instance, please rerun the pipeline using the updates in variable and let me know if everything works fine for you

Regards Prateek

ivan-zaitsev commented 1 year ago

@5-k 

##[error]Error: ENOENT: no such file or directory, rename '/home/vsts/work/1/s/apache-jmeter-5.5/LICENSE' -> '/home/vsts/work/1/s/custom-jmeter-folder-name/LICENSE'
23-Nov-2022 13:20:30:010 - Error: ENOENT: no such file or directory, rename '/home/vsts/work/1/s/apache-jmeter-5.5/LICENSE' -> '/home/vsts/work/1/s/custom-jmeter-folder-name/LICENSE'
23-Nov-2022 13:20:30:011 - Please note this extension is in beta version. We will be fixing the issues as reported and hence it is important for us to be able to analyze logs and fix the issue. If you encounter any issue please create a github issue here https://github.com/microsoft/jmeter-performance-analyzer-devops-extension/issues
##[error]ENOENT: no such file or directory, rename '/home/vsts/work/1/s/apache-jmeter-5.5/LICENSE' -> '/home/vsts/work/1/s/custom-jmeter-folder-name/LICENSE'
5-k commented 1 year ago

Hi @ivan909020

The issue has been fixed by adding a random generated string whenever Jmeter pipeline task should run. This will ensure 2 things

  1. Each run will have a separate folder while execution. Copying of JMX,Property, Input file will be isolated.
  2. Report and logs absolute path shall be unique.

With this change in place you need not change any advanced variables. You can modify if you need to.

However in case you have multiple task running in same pipeline, you should provide a unique name for the published artifact. This is because the published artifacts are per pipeline and running multiple task in same pipeline might overwrite the published artifact.

I can update the code to make the published artifact unique but that will overwrite user's preference of the output folder name. Hence please ensure to keep the following names unique.

Artifact Name for Apache Jmeter Logs (artifactNameLog) Artifact Name for Apache Jmeter Reports (artifactNameReport)

Please let me know if this solves the issue. Once sure please close the issue with comment.

ivan-zaitsev commented 1 year ago

@5-k Thanks, I will check it today.

Also noticed that this plugin moves JMX, properties, additional files to the 'bin' folder of JMeter. This is not critical, but it breaks relative paths for JMX file (E. g.: JMX test file reads from CSV file, or have test fragments with different JMX files). Currently I fixed this by not using relative paths, but it makes difficult to run it locally because you need to move scripts to common folder and only then you will be able to run it.

Example: /root/test-fragment.jmx /root/test1/test1.jmx /root/test1/resources/image.jpg /root/test1/resources/input.properties /root/test1/resources/data.csv

When we specify relative paths for '/root/test1/test1.jmx' for all that files (../test-fragment.jmx, resources/image.jpg, resources/input.properties, resources/data.csv) and run it locally everything works, but in pipeline it does not work.

ivan-zaitsev commented 1 year ago

@5-k

Thanks, this fixed this issue "##[error]ENOENT: no such file or directory, rename '/home/vsts/work/1/s/apache-jmeter-5.5/LICENSE' -> '/home/vsts/work/1/s/custom-jmeter-folder-name/LICENSE'".

But it didn`t resolved the main issue when I have several tests in one repo, different pipelines show the same results for different JMX files. I started to dig deeper and found out that the issue was because this plugin moves JMX, properties, additional files to the 'bin/' folder of JMeter.

For example, we have folder structure:

/root/test-fragment.jmx /root/test1/test.jmx /root/test2/test.jmx /root/test3/test.jmx

And if we specify "jmxInputFolderSourcePath" as "/root/" (because we need test-fragment.jmx) it will move all files from subfolders to bin/ folder of JMeter. And even if we are running /root/test1/test.jmx currently, it will overwrite that file with the last one /root/test3/test.jmx

Currently I fixed this by creating subfolder /root/common/test-fragment.jmx. I think that logic that you added with random int already is not useful anymore.

Maybe the best option to consider is not to move the test files to "bin/" folder? This will also resolve issue with relative paths.

5-k commented 1 year ago

Hi @ivan909020

Can we have a call to discuss the issue you face. I feel this can be handled with the unique folder name, but if you say so, lets discuss

ivan-zaitsev commented 1 year ago

@5-k Do you have skype? My login is ivan909020

archerixx commented 1 year ago

Hi @5-k ,

Latest changes negatively impacted whole build system for me. Extracted apache-jmeter-5.5_ folder (with random sufix) can not be used in our build flow. Is there a way for you to at least mark these changes as new version of Jmeter-Performance-Analyzer ? Currently it is version 1, if you could please update it to 2* or leave/app option to use default apache-jmeter-5.5 name without sufix

Thank you

5-k commented 1 year ago

@archerixx

Can you share the error/output logs

archerixx commented 1 year ago

It is not problem with JmeterPrefernece as a Task, but with a flow that comes after it. Since it was setup like there will be folder "apache-jmeter-5.5" (but now Directory created successfully: apache-jmeter-5.5_155524671 is created, so i cant find it unless finding some way to enable wildcard on folder names) JMeter_Custom_Unzipped_Folder_Name is set to "apache-jmeter-5.5", but it still adds suffix at the end.

5-k commented 1 year ago

Hi @archerixx

I will add an checkbox option to enable random suffix generation and keep it default to false. It would solve the issue

Hi @ivan909020 Please enable the checkbox to true for generating random suffix for jmeter folder, log folder, report folder, artifact folder (one common checkbox under advanced). Also we are working on your other ask for direct reference instead of copy file

Fix will be deployed in an hour Thanks

archerixx commented 1 year ago

Thank you very much @5-k

ivan-zaitsev commented 1 year ago

Hi @archerixx

I will add an checkbox option to enable random suffix generation and keep it default to false. It would solve the issue

Hi @ivan909020 Please enable the checkbox to true for generating random suffix for jmeter folder, log folder, report folder, artifact folder (one common checkbox under advanced). Also we are working on your other ask for direct reference instead of copy file

Fix will be deployed in an hour Thanks

Hello, I don't think that option with random suffix is needed, I just changed the folder name to unique one for each pipeline.

5-k commented 1 year ago

Sure then looks it works for both use case, Version upgraded to 1.7.45

archerixx commented 1 year ago

Hi @5-k again, Sorry to bother. It seems that "JMeter Log Folder Name" and "JMeter Report Folder Name" folder names have been affected by this suffix change. They now have "_" sign added at the end. Will you be able to remove it ?

Also, what I noticed is that extracted zip/tar now have subfolder named identicaly, so path to apache-jmeter is like this now: "${wokringDirectory}/apache-jmeter-5.5/apache-jmeter-5.5/...". In previous version, there was no additional subfolder. Could you let me know if this will be reverted or will it stay as is, so I can make changes on my end ?

5-k commented 1 year ago

Hi @archerixx Moving this to a different issue: https://github.com/microsoft/jmeter-performance-analyzer-devops-extension/issues/33