FileNotFoundException when executing tests on Azure

[mikeliucc]

My test runs find locally. When I tried to run the same test on Azure (following the instruction here https://abstracta.github.io/jmeter-java-dsl/guide/#azure-load-testing), I got the following error on Azure:

2023-08-01 06:12:17,338 WARN o.a.j.JMeter: LogLevel: INFO
2023-08-01 06:12:17,839 ERROR c.b.c.RandomCSVReader: Cannot initialize RandomCSVReader, because of error: 
java.io.FileNotFoundException: /root/artifacts/accountAndGuestIds.csv (No such file or directory)

Looks like the test data file needs to be uploaded to Azure. How would I go about doing this?

Thanks in advance, Mike

[rabelenda]

Hello, thank you for asking about this.

I have just implemented this feature in the same way as BlazeMeter, but for some reason I am currently experiencing some weird behavior with Azure Load Testing, and I need to get an answer from Azure Load Testing before I release this feature.

Here is the jar in case you want to try it. It will automatically upload csv resources in the same way as BlazeMeterEngine currently does.

To try it, you can install it with maven like this:

mvn install:install-file -Dfile=jmeter-java-dsl-azure-1.18-SNAPSHOT.jar

and then just add the dependency to your project:

<dependency>
  <groupId>us.abstracta.jmeter</groupId>
  <artifactId>jmeter-java-dsl-azure</artifactId>
  <version>1.18-SNAPSHOT</version>
  <scope>test</scope>
</dependency>

Once I solve the issue I am facing with Azure, I will release this with next version of the DSL

[mikeliucc]

Cool! I'll give it a spin in a few minutes.

Meanwhile, here's a few related things for your consideration:

1. I think Azure Load Testing only support "stock" JMeter JMX. So plugins might be a no-no. In our case, I happened to use randomOrder() on my CSV data set, which in turn resulted in a different <CSVDataSet> component in the generated JMX (I think it's the one from blazemeter). Once I turn off randomOrder() on my CSV data set, I was able to run my tests on Azure.
2. Any chance we might be able to include a few of the Azure niceties in the DSL? When I manually create a test in Azure Load Testing, I see these options - any chance we can include (some of) them in the DSL?
3. Lastly, in a more complex scenario, we might have multiple CSV data sets. I think Azure can support this. Please consider also supporting multiple CSV file uploads as well.

I'll report back my findings after trying out the new jar.

[rabelenda]

Hello,

1. Are you sure plugins don't work? I just tested one with random access (which uses RandomCSVDataSetConfig), and worked as expected. Might it be related to the account or something? I will try to dig about this with Azure Load Testing team.

2. What in particular do you want to use, and why? :). We can add many features, but would like to understand better the use case, and some of them might not be that relevant when using JMeter DSL vs JMeter GUI. For instance, I don't know yet a scenario where using environment variables is needed to do something in JMeter DSL, vs just using java variables and potential environment variables or system properties in JMeter DSL execution. I am quite sure that if you describe scenarios where you use some of those features, it will make sense to implement support for some of them.

3. The provided jar automatically uploads all csvs used in test plan, alongside with files used in http requests. You can also use asset method to upload any other potential file you may need (even jars for plugins or dependencies). Is that enough? Or are you referring to something else? Here is an example of using assets to upload dependency jars.

[mikeliucc]

OK... first on the findings on the new jar.

The short of it, it worked. Thank you! It was slow going for a while due to the way we set up our POM file and also due to access restrictions on our Azure environment. The Azure issues showed up really funky... My initial report regarding RandomCSVDataSetConfig was actually just a massive convoluted Azure misconfiguration.

But, after much strenuous struggle with Azure, we finally got it to work with the 1.18-SNAPSHOT jar. Thanks again!!

To respond to your comment:

1. Yes, random CSV access works. My bad.
2. Here's a possible use case for your consideration. When we first create a Azure load test, we can only configure via DSL its name, number of engines and timeout value. However such test won't provide the monitoring and test criteria we would like to have. Here's an example of what I manually configure after the test has been created:

With the above configuration, our load test would show up with the desired monitoring for both the client-side and server-side metrics.

If we are re-running the same test each time, then I suppose it doesn't matter -- I don't mind the one-time manual work. However we are planning various scenarios (i.e. different Azure tests) and probably more to come in the future.

It's true that I don't need the environment variable configuration anymore -- found another way to get around it.

Another useful feature to consider is to name the test run. Currently they all show up as TestRun_YYYY-MM-DD_HH:MM:SS:

Any chance you would consider letting us change the test run name or the description?

3. Yes, confirmed! Very cool!

Thanks again!

[rabelenda]

Hello, I am glad the new jar worked :), I am still trying to figure out some issue with Azure Load Test execution, but as soon as I solve it we will release a new version with the changes included in the jar I previously shared.

Regarding the configuration you sent. Thank you for the detailed info!

I see you even marked the main things in yellow, but just to clarify:

1. You need support for splitCsvBetweenEngines. Thats easy to support and easy to understand why you would need it :).
2. Even though JMeter DSL provides means to do assertions on statistics, you actually need auto stop logic. We discussed about it in this issue https://github.com/abstracta/jmeter-java-dsl/issues/50, and in discord channel there have been users requesting similar feature, and even one implemented one approach that I think shared with other users. In any case, I think is something we should implement. Not particularly for Azure, but for the most general use case.
3. Resource monitoring in Azure. We could add something like monitoredResource() to AzureEngine.
4. Test run names which could easily be added to AzureEngine. Any particular reason for changing the test run names? Maybe to include CI/CD pipeline id or something like that? Would like to know about your need so as to provide examples to users of the method on when makes sense to specify a name.

[mikeliucc]

1. Awesome!
2. Awesomer!
3. Most Awesomous!
4. What could be meaningful would be release name, build number... Something that help us to associate a particular run to some significant moment in time past.

[rabelenda]

1. 3. and 4. are easy to implement.
2. Would take more analysis and potentially time.

One alternative for 2. for the meantime might be to use jmeter-java-dsl-wrapper module with auto stop plugin. Eg:

Add these dependencies:

<dependency>
  <groupId>us.abstracta.jmeter</groupId>
  <artifactId>jmeter-java-dsl-wrapper</artifactId>
  <version>1.17</version>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>kg.apc</groupId>
  <artifactId>jmeter-plugins-autostop</artifactId>
  <version>0.2</version>
  <scope>test</scope>
  <exclusions>
    <exclusion>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-nop</artifactId>
    </exclusion>
    <exclusion>
      <groupId>org.apache.jmeter</groupId>
      <artifactId>jorphan</artifactId>
    </exclusion>
  </exclusions>
</dependency>

And use the plugin in a test plan, like this example:

  @Test
  public void test() throws Exception {
    testPlan(
        threadGroup(1, Duration.ofSeconds(10),
          dummySampler("FAIL")
              .responseTime(Duration.ofSeconds(1))
              .simulateResponseTime(true)
              .successful(false),
          testElement(buildAutoStop(90, 3))
        )
    ).run();
  }

  private TestElement buildAutoStop(int errorRatePercent, int errorRateSeconds) {
    AutoStop ret = new AutoStop();
    ret.setName("AutoStop");
    ret.setProperty(TestElement.GUI_CLASS, AutoStopGui.class.getName());
    ret.setProperty(TestElement.TEST_CLASS, ret.getClass().getName());
    ret.setProperty("avg_response_latency", 0);
    ret.setProperty("avg_response_latency_length", 0);
    ret.setProperty("avg_response_time", 0);
    ret.setProperty("avg_response_time_length", 0);
    ret.setProperty("error_rate", errorRatePercent);
    ret.setProperty("error_rate_length", errorRateSeconds);
    return ret;
  }

[mikeliucc]

Sure, will give that a try! Thanks!

[rabelenda]

We have just release a new version, which includes features to support 1. 3. & 4.