Defining Test Data

[hevayo]

Question : What is the best way to define test data?. Test data will be useful for following

• Data driven tests
• Mocking
• Providing data for library users to mock library functions
• Use the data for machine learning ( in data mapper )

Option 1 : Define test data in the source it self

• Can we recommend to define test data as ballerina values in source code itself. The test data can reside in the test source or a separate module. Pros:

  • Data will be automatically validated with types and make it far easier to maintain.
  • Data will be readily available to write tests without needing to write code to load from files.
  • Ballerina allows you to store data directly as records, json or xml.

  Cons:

  • Data has to be kept off from packing in to executable.

• Spec changes required to keep data in ballerina source.

  • Currently to implement test cases you can keep the test data inside the test directory (test modules). But these test data will not be able to be shared with balos to be used by external users and machine learning BE.
  • Going forward if we have a mechanism to export test only modules with a package we can keep all the test data in that module

[miyurud]

+1 for having test data in a separate module directory. We need to have a means of packing the sample data attached to test cases into the balo file.

Having test data embedded within the test case file directly is one option to consider. For example, we may use special annotation or directly specify sample data as Ballerina records. However, there are multiple practical issues.

• This will increase the file size of the test case files by significant amount. This will also introduce issues of version control.
• Sometimes this may reduce the readability of the test sources.
• How are we going to handle the case of Personal Identity Information (PII) getting committed to the git repositories?

Ensuring the quality Irrespective from which source we extract the test data we will have to ensure the quality of the test data provided. For example, we should validate to a major extent the test data we get are the data that we need to run with the test cases. Data with missing attributes, data with different attribute types will be problematic.

It is good to have both valid as well as invalid data to run with test cases (see). But for scenarios like data mapping its always better to have high quality valid data.

Leverage Test Data Generator Tools There is an on going effort to auto generate test data artificially. We could integrate that with this effort so that developer does not need to worry too much about how to specify the sample data, but use the intelligent suggestions and modify them a bit to generate a suitable data set.

In this case the developer will have to be aware of how to generate such test data artificially using the test data generator tools.