Runtime configurable FilePaths

[Milton-Xiong]

Dataloader FilePaths is configurable at design time by annotation, but only accepting constant. As a data driven test designer, I'd like to be able to re-config dataloader filepaths during runtime - provide APIs to update or modify FilePaths.

[anujgandharv]

Hi Milton-Xiong, EasyTest Core already supports specifying filepaths at runtime. You can specify a variable value as part of the DataLoader annotation and provide the value of the variable at runtime, or you can simply specify Data Loader annotation without any attributes and specify the variable "testDataFiles" during runtime. Here is the detailed description as well as usage on both scenarios, which you can also find in the README file.

A user can now specify a variable value as part of the DataLoader's filePaths attribute. Thus it is now possible to use DataLoader annotation like this :

@DataLoader(filePaths = {"${my.data.file}" , "${my.second.data.file}"}) Using the above way, a user can specify properties of the above variables "my.data.file" and "my.second.data.file" as System property using -D option of Java System Properties.

A new System Property "testDataFiles" to provide a comma separated list of input test data files at runtime has been included in EasyTest. In order to use this option simply specify @DataLoader annotation at the top of your class without any input data. Thus in such a case DataLoader annotation acts as a marker annotation telling the EasyTest system that it has to fetch the value of filePaths attribute from the system property "testDataFiles".

NOTE If a user has specified both "testDataFiles" System Property AND a value for "dataFiles" attribute, then the System Property files(specified using testDataFiles System Property) will override the files specified using the "dataFiles" attribute of DataLoader annotation.

I hope this is helpful. Please let me know if you need any more information.

Thanks, Anuj Kumar

[Milton-Xiong]

Hi Anuj, I tried the following code but both still failed. I'm using version 1.3.2.@RunWith(DataDrivenTestRunner.class) @DataLoader() public class SimpleOrderManagerTest { @BeforeClass public static void beforeClass() { System.setProperty("testDataFiles", Config.getInstance().getTestRootFolder()

• TESTCASE_FOLDER + Config.getInstance().getDataFolder() + "SimpleOrderManagerTest.xls"); }

@Test
public void testWithDataDriven(@Param(name="name")String name , @Param(name="age")int age , @Param(name="expectedOutput")int expectedOutput) throws Exception {
    System.out.println("Name : " + name);
    System.out.println("Age : " + age);
    System.out.println("ExpectedOutput : " + expectedOutput);
} or 

@RunWith(DataDrivenTestRunner.class) @DataLoader(filePaths = {"${SimpleOrderManagerTest.testDataFiles}"}, loaderType= LoaderType.EXCEL)public class SimpleOrderManagerTest { @BeforeClass public static void beforeClass() { System.setProperty("SimpleOrderManagerTest.testDataFiles", Config.getInstance().getTestRootFolder() + TESTCASE_FOLDER + Config.getInstance().getDataFolder() + "SimpleOrderManagerTest.xls"); }

@Test
public void testWithDataDriven(@Param(name="name")String name , @Param(name="age")int age , @Param(name="expectedOutput")int expectedOutput) throws Exception {
    System.out.println("Name : " + name);
    System.out.println("Age : " + age);
    System.out.println("ExpectedOutput : " + expectedOutput);
}Below is the exception and call stack:java.lang.AssertionError: Could not load the resource with path testDataFiles as either a Classpath, FileSystem or a URL resource. Please check the path and try again at org.junit.Assert.fail(Assert.java:88) at org.easetech.easytest.io.ResourceLoaderStrategy.getResource(ResourceLoaderStrategy.java:115) at org.easetech.easytest.loader.DataLoaderUtil.loadData(DataLoaderUtil.java:250) at org.easetech.easytest.runner.RunnerUtil.loadClassLevelData(RunnerUtil.java:76) at org.easetech.easytest.runner.DataDrivenTestRunner.<init>(DataDrivenTestRunner.java:119) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:80) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:57) at java.lang.reflect.Constructor.newInstance(Constructor.java:540) at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:104) at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:86) at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59) at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26) at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59) at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:41) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)

In debug mode, I found @BeforeClass is not invoked. is this the reason? why is @BeforeClass not invoked if I run test with @RunWith(DataDrivenTestRunner.class)? Thanks,Milton

 On Tuesday, March 17, 2015 11:16 PM, Anuj Kumar <notifications@github.com> wrote:

Hi Milton-Xiong, EasyTest Core already supports specifying filepaths at runtime. You can specify a variable value as part of the DataLoader annotation and provide the value of the variable at runtime, or you can simply specify Data Loader annotation without any attributes and specify the variable "testDataFiles" during runtime. Here is the detailed description as well as usage on both scenarios, which you can also find in the README file.A user can now specify a variable value as part of the DataLoader's filePaths attribute. Thus it is now possible to use DataLoader annotation like this :@DataLoader(filePaths = {"${my.data.file}" , "${my.second.data.file}"}) Using the above way, a user can specify properties of the above variables "my.data.file" and "my.second.data.file" as System property using -D option of Java System Properties.A new System Property "testDataFiles" to provide a comma separated list of input test data files at runtime has been included in EasyTest. In order to use this option simply specify @DataLoader annotation at the top of your class without any input data. Thus in such a case DataLoader annotation acts as a marker annotation telling the EasyTest system that it has to fetch the value of filePaths attribute from the system property "testDataFiles".NOTE If a user has specified both "testDataFiles" System Property AND a value for "dataFiles" attribute, then the System Property files(specified using testDataFiles System Property) will override the files specified using the "dataFiles" attribute of DataLoader annotation.I hope this is helpful. Please let me know if you need any more information.Thanks, Anuj Kumar— Reply to this email directly or view it on GitHub.

[anujgandharv]

Hi Milton. This would fail, because before calling the BeforeClass method, EasyTest tries to load all the annotations at the Class level. If you want to set the System Property in the beforeClass method, then instead of using @DataLoader annotation at Class level, you should use it at Method level. I have personally never tried it, but looking at the code base, I am confident that it will work. Please let me know in case you have any issues.

Thanks, Anuj Kumar

[Milton-Xiong]

Dear Anuj, Unfortunately it is still failed, like what you said I changed my code as below:@RunWith(DataDrivenTestRunner.class) public class SimpleOrderManagerTest { @BeforeClass public static void beforeClass() { System.setProperty("simplTestMethod_DataFile", "C:\Workplace\Java\javatestflextrade\SimpleTest.xls"); }

@Test
//@DataLoader(filePaths = {"C:\\Workplace\\Java\\javatestflextrade\\SimpleTest.xls"}, loaderType = LoaderType.EXCEL)
@DataLoader(filePaths = {"${simplTestMethod_DataFile}"}, loaderType = LoaderType.EXCEL)
public void simplTestMethod(@Param(name = "name") String name, @Param(name = "age") int age, @Param(name = "expectedOutput") int expectedOutput) {
    System.out.println(name);
    System.out.println(age);
    System.out.println(expectedOutput);
}It is not working, throw same exception in my previous email.Any idea?

Thanks,Milton

 On Wednesday, March 18, 2015 4:40 PM, Anuj Kumar <notifications@github.com> wrote:

Hi Milton. This would fail, because before calling the BeforeClass method, EasyTest tries to load all the annotations at the Class level. If you want to set the System Property in the beforeClass method, then instead of using @DataLoader annotation at Class level, you should use it at Method level. I have personally never tried it, but looking at the code base, I am confident that it will work. Please let me know in case you have any issues.Thanks, Anuj Kumar— Reply to this email directly or view it on GitHub.

[anujgandharv]

Hi Milton, Sorry to hear that it is still not working for you. Few questions to understand the problem clearly: 1) Does the DataLoader Annotation that you have commented out on the Test Method in the above example work? So this one : //@DataLoader(filePaths = {"C:\Workplace\Java\javatestflextrade\SimpleTest.xls"}, loaderType = LoaderType.EXCEL)

If it works then it would rule out the issue of finding the file on the Filesystem.

2) Can you try running your test class from command line using JUnitCore class to run the tests. Here is the command you can use : java -cp org.junit.runner.JUnitCore -DsimplTestMethod_DataFile=C:\Workplace\Java\javatestflextrade\SimpleTest.xls

and see if it works.

In the mean time I will investigate from my side why it doesn't work.

Thanks, Anuj Kumar

[anujgandharv]

So in the above command, Github removed something. Here it is again for easy understanding java -cp path\to\Junit.jar org.junit.runner.JUnitCore path\to\your\test\class -DsimplTestMethod_DataFile=C:\Workplace\Java\javatestflextrade\SimpleTest.xls

[Milton-Xiong]

Hi Anuj, Both 2 cases you mentioned in your email work well.I spent a little bit time to have a look at the code, found that test data file path and content are getting loaded quite early, even before @BeforeClass method being invoked.This causes it is impossible to change/modify data file path during runtime, there is no place I can write java code to change it when I write unit test. the only way I can specify data file path is through -D option. this does not meet our requirements, since we want to load data from different places during runtime which depends on some between status/values on the fly.We are evaluating a few data driven approaches with junit, and think easytest is one of the best. It would be very appreciated if you can fix this issue asap. Thanks,Milton.

 On Thursday, March 19, 2015 10:23 PM, Anuj Kumar <notifications@github.com> wrote:

Hi Milton, Sorry to hear that it is still not working for you. Few questions to understand the problem clearly: 1) Does the DataLoader Annotation that you have commented out on the Test Method in the above example work? So this one : //@DataLoader(filePaths = {"C:\Workplace\Java\javatestflextrade\SimpleTest.xls"}, loaderType = LoaderType.EXCEL)If it works then it would rule out the issue of finding the file on the Filesystem.2) Can you try running your test class from command line using JUnitCore class to run the tests. Here is the command you can use : java -cp org.junit.runner.JUnitCore -DsimplTestMethod_DataFile=C:\Workplace\Java\javatestflextrade\SimpleTest.xlsand see if it works.In the mean time I will investigate from my side why it doesn't work.Thanks, Anuj Kumar— Reply to this email directly or view it on GitHub.

[anujgandharv]

HI Milton, I would like to understand the actual Use case you have where you want to specify the file in the before class method and not on the class level. TBH, this is not a runtime scenario as you are still specifying the file path inside the class file. So I am currently not sure, what is your intention for providing the System Property as part of the before Class method.

Changing the current EasyTest project is possible, but yours is a feature request and since it changes the flow of how EasyTest loads various assets, it would be a Big feature request that can not be delivered in the coming months. It will take a lot of effort.

[Milton-Xiong]

Hi Anuj, Here is one of the use cases:Public TestClass{    @Test    @DataLoader(filePaths = {"${ReadFromConfigFile}"}    Public void AccountRegisterTest() {        ...    }     @Test    @DataLoader(filePaths = {"${filePathsBasedOnTheAccountTypeRegistered}"}    Public void AccountCheckTest() {        ...    }} Scenario 1 - We'd like to have the data file path is configurable - can be read from a java property/XML/Json file. Thus, if we want to exercise the tests by using different set of data which are located different server or folder, what we need to do is just changing the configuration file. You might ask why not use -D option? we have thousands test classes / methods which need to read data from thousands folders, we don't want to specify thousands of -D options at command line. :-)Below is an example of the folder hierarchy for the test data:C: |-DataSet_1 |          |-AccountRegister |          |-AccountCheck |-DataSet_2 |          |-AccountRegister |          |-AccountCheck Scenario 2 - We have some system test, which requires tests can be performed in a certain order, and there are some dependencies between them. E.g. AccountCheckTest has to be performed after AccountRegisterTest. and most importantly, AccountCheckTest needs to be feeding different data, which depends on the Account Type registered, which is one of the outcomes of AccountRegisterTest. unfortunately, the Account Type is not predictable. Thanks,Milton

 On Friday, March 20, 2015 6:35 PM, Anuj Kumar <notifications@github.com> wrote:

HI Milton, I would like to understand the actual Use case you have where you want to specify the file in the before class method and not on the class level. TBH, this is not a runtime scenario as you are still specifying the file path inside the class file. So I am currently not sure, what is your intention for providing the System Property as part of the before Class method. Changing the current EasyTest project is possible, but yours is a feature request and since it changes the flow of how EasyTest loads various assets, it would be a Big feature request that can not be delivered in the coming months. It will take a lot of effort.— Reply to this email directly or view it on GitHub.

[anujgandharv]

Hi Milton, Sorry for replying late. I was caught up in some other stuff. Ok, now that I understand your Use Case, I am in a better position to provide a solution to it.

Have a look at the following gist : https://gist.github.com/anujgandharv/ca144065d855842ec762

Here I have used the EasyTest specific annotations @TestConfigProvider , @TestProperties and @TestBean to realize your use case.

In essence, I have decoupled the loading of your Configuration files from your actual test class.

The idea is to do the following: 1) Create a config file(Java properties file) that contains the path to the test data files. 2) Load this config file within the EasyTest framework using the @TestProperties annotation. 3) Use the loaded properties inside a method annotated with @TestBean to set the System Property. 4) Next, use the path name inside the DataLoader annotation specified either at the class level or method level, to load the test data.

Please have a look at the gist and let me know if you need any clarifications. Also let me know if it works in your scenario or you need something else.

Thanks, Anuj Kumar

[anujgandharv]

Hi Milton, Just wondering if the above solution worked for you?

Thanks, Anuj Kumar

[Milton-Xiong]

Hi Anuj, Great! It resolves the issue scenario 1!It would be perfect if scenario 2 also can be fixed - Data file path is dependent on some outcomes of previous tests. The outcomes are not predictable and unknown before hand. for more detail, please see my previous email scenario 2. Thanks,Sheng

 On Tuesday, March 24, 2015 4:39 PM, Anuj Kumar <notifications@github.com> wrote:

Hi Milton, Just wondering if the above solution worked for you? Thanks, Anuj Kumar— Reply to this email directly or view it on GitHub.

[anujgandharv]

Hi Milton, It is an excellent Use Case for which currently there is no support in EasyTest. But, I would be more than happy to add such support as I believe it would be a good addition to EasyTest. Let me start a new thread to track it cleanly. I would require your knowledge and expertise in defining clear requirements.

Thanks, Anuj Kumar

[anujgandharv]

Please have a look at issue #114 Please provide your input comments and let me know how you, as an end user of easytest would liketo have this functionality defined in the most user friendly way.

Thanks, Anuj Kumar

[Milton-Xiong]

Hi Anuj, Issue #114 is not exact what I want. Most of the time test dependency ordering can be addressed by using FixMethodOrder (JUnit API) annotation. What I want is much further, but still regarding Runtime configurable FilePaths, which is dependent on some outcomes of previous tests. please look at the following example:Below is an example of the folder hierarchy for the test data:C: |-DataSet            |-AccountRegister            |-AccountCheck                          |-OrdinaryAccount                          |-priviledgeAccount This is the test class: Public TestClass{    @Test    @DataLoader(filePaths = {"${ReadFromConfigFile}"}    Public void AccountRegisterTest() {        ...        //We are not getting what account type it is until AccountRegisterTest() is done.     }     @Test    @DataLoader(filePaths = {"${filePathsBasedOnTheAccountTypeRegistered}"} //Based on the account type, different data would be loaded from the corresponding folder.    Public void AccountCheckTest() {        ...    }} Thanks,Sheng

 On Wednesday, March 25, 2015 5:16 AM, Anuj Kumar <notifications@github.com> wrote:

Please have a look at issue #114 Please provide your input comments and let me know how you, as an end user of easytest would liketo have this functionality defined in the most user friendly way.Thanks, Anuj Kumar— Reply to this email directly or view it on GitHub.

[anujgandharv]

Interesting use case. But I have quite a few questions. 1) It might be the case that AccountRegisterTest is run 20 times (20 test data rows defined in Excel file.) Each run of the test may potentially produce different accountType values.. How do you envision handling such a scenario. 2) How do you envision handling a test failure or a test exception? 3) Do you envision loading the test data for the dependent test method at each run of the method it depends on. It can potentially have an impact on the performance of the system tests as well as potentially failing tests because of unavailable memory, in case a lot of test methods are executed with a lot of test data.

[anujgandharv]

btw, just out of curiosity, what company are you working for?

[Milton-Xiong]

Hi Anuj, Let me answer your first question first:Here is how we can do it based on Junit @RunWith(Parameterized.class): Public class DataDrivenByJunit{    private int input;    private String expected;    private boolean isPriviledge;     @Parameterized.Parameters(name = "{index}: DataDrivenByJunit({0})={1}")    public static Iterable<Object[]> data() {        return Arrays.asList(new Object[][]{                {4, "X"}, {5, "Y"}, {6, "Z"}        });    }

    public DataDrivenByJunit(int input, String expected) {        this.input = input;        this.expected = expected;    }     @Test    public void AccountRegisterTest() {        isPriviledge = RegisterAccount(input);    }

    @Test    public void AccountCheckTest() {        if (isPriviledge)            //Load data from priviledgeAccount folder;        else            //Load data from OrdinaryAccount folder;;    }} I think the major difference here is, Junit inject the test data as instance member variables. we have full control on it.In the case that AccountRegisterTest is run 20 times, 20 instances of class DataDrivenByJunit would be created, and all the test methods would be invoked against these instances respectively, so no data interference would happen.

Thanks,Sheng

 On Wednesday, March 25, 2015 5:04 PM, Anuj Kumar <notifications@github.com> wrote:

Interesting use case. But I have quite a few questions. 1) It might be the case that AccountRegisterTest is run 20 times (20 test data rows defined in Excel file.) Each run of the test may potentially produce different accountType values.. How do you envision handling such a scenario. 2) How do you envision handling a test failure or a test exception? 3) Do you envision loading the test data for the dependent test method at each run of the method it depends on. It can potentially have an impact on the performance of the system tests as well as potentially failing tests because of unavailable memory, in case a lot of test methods are executed with a lot of test data. — Reply to this email directly or view it on GitHub.

[anujgandharv]

Hi Sheng, You can do it using Parameterized Runner, but personally I would not advise you to do it, simply because of the coupling of test data with test class as well as the need to define a constructor. I am sure you are aware of the drawbacks of using Parameterized runner, specifically in scenarios where there can be multiple tests part of the same test class, that expects different input test data. You can have a look at my blog post on Parameterized runner for more indepth details on Parameterized runner : http://www.kumaranuj.com/2012/08/junits-parameterized-runner-and-data.html

Keeping it aside, what I would like to say is that I think your use case is a good use case and it would be good if I can implement it in EasyTest code base, But I have to be clear on the requirements. A simple explanation from you would suffice to clear the requirements. This is what I have gathered so far from the discussion. Also is it possible to use gist for code examples. They make the readability of the comment a lot easier. I am trying again to visualize your above example as requirements:

1) Let's assume we have two methods in our test class : M1 and M2. Lets also assume that M2 should run after M1. Also, lets assume that the test data for M1 is loaded in the memory and it has three rows of test data. This means that M1 will run 3 times. 2) Then for each execution of M1, you would like to do the following: a) Load the test data for M2 based on the outcome of M1. b) Run test M2 using the loaded test data. Which means, if the test data loaded for M2 contains 10 rows, then M2 will run 10 times.

Thus if the test data file for M1 has 3 rows and each of different M2 test data files have 10 rows, then the total number of tests executed will be : 1 X 10 + 1 X 10 + 1 X 10 = 30

If the three test data files for M2 are different and each contains 5, 6, 7 test data rows respectively, then the total number of tests executed will be : 1 X 5 + 1 X 6 + 1 X 7 = 18

Do you think its a clear representation of your above example? Do you think if EasyTest provides such a functionality then you will use the functionality.

Thanks, Anuj Kumar

[Milton-Xiong]

Hi Anuj, Perfect! your representation is very clear, that's exactly what I want.To be frank, I also don't like the coupling between tests and test data. But we have some very long long system test cases, or user acceptance tests, which perform some kind of "end-to-end" job sequences which from account creation, account checking, ... all the way to the account closing. Do I have a better choice? Thanks,Sheng

 On Thursday, March 26, 2015 6:59 PM, Anuj Kumar <notifications@github.com> wrote:

Hi Sheng, You can do it, but personally I would not advise you to do it, simply because of the coupling of test data with test class as well as the need to define a constructor. I am sure you are aware of the drawbacks of using Parameterized runner, specifically in scenarios where there can be multiple tests part of the same test class, that expects different input test data. You can have a look at my blog post on Parameterized runner for more indepth details on Parameterized runner : http://www.kumaranuj.com/2012/08/junits-parameterized-runner-and-data.htmlKeeping it aside, what I would like to say is that I think your use case is a good use case and it would be good if I can implement it in EasyTest code base, But I have to be clear on the requirements. A simple explanation from you would suffice to clear the requirements. This is what I have gathered so far from the discussion. Also is it possible to use gist for code examples. They make the readability of the comment a lot easier. I am trying again to visualize your above example as requirements:1) Let's assume we have two methods in our test class : M1 and M2. Lets also assume that M2 should run after M1. Also, lets assume that the test data for M1 is loaded in the memory and it has three rows of test data. This means that M1 will run 3 times. 2) Then for each execution of M1, you would like to do the following: a) Load the test data for M2 based on the outcome of M1. b) Run test M2 using the loaded test data. Which means, if the test data loaded for M2 contains 10 rows, then M2 will run 10 times.Thus if the test data file for M1 has 3 rows and each of different M2 test data files have 10 rows, then the total number of tests executed will be : 1 X 10 + 1 X 10 + 1 X 10 = 30 If the three test data files for M2 are different and each contains 5, 6, 7 test data rows respectively, then the total number of tests executed will be : 1 X 5 + 1 X 6 + 1 X 7 = 18Do you think its a clear representation of your above example? Do you think if EasyTest provides such a functionality then you will use the functionality.Thanks, Anuj Kumar— Reply to this email directly or view it on GitHub.

[anujgandharv]

Ok now that I know what you want, let me investigate how I can provide such a functionality within easytest. I will get back to you soon on this.

[anujgandharv]

Hi Sheng, Because of the unique functionality request, I found that there are quite a few changes that needs to be done to EasyTest. I think it would be a better idea if I give you a separate Test Runner class instead of DataDrivenTestRunner, having the specific functionality as specified above. Do you think it would suffice your needs?

Thanks, Anuj Kumar

[Milton-Xiong]

Hi Anuj, In my mind, a new test runner makes sense.But I prefer the new test runner is some kind of a super set of DataDrivenTestRunner - all the good functionality in DataDrivenTestRunner should be kept. Thanks,Sheng

 On Tuesday, March 31, 2015 4:13 PM, Anuj Kumar <notifications@github.com> wrote:

Hi Sheng, Because of the unique functionality request, I found that there are quite a few changes that needs to be done to EasyTest. I think it would be a better idea if I give you a separate Test Runner class instead of DataDrivenTestRunner, having the specific functionality as specified above. Do you think it would suffice your needs?Thanks, Anuj Kumar— Reply to this email directly or view it on GitHub.

[anujgandharv]

Hi Sheng, I was updating the Clients list of EasyTest and wanted to include the name of the company you are working with. Could you please provide me with the same? I will be pitching for including EasyTest as Apache Incubator Project and having the clients list will help me a lot.

[pebelsberger]

Any news on this?