Custom DataAttribute causes catastrophic error

[jcuello]

I'm having an odd issue when running my tests with the following NewLineTextFileDataAttribute. The code is in F# but to summarize what it's doing, the attribute takes input/output filepaths which searches the filepaths for input/output file pair.

namespace Bioinformatics.Tests.Attributes
open System
open System.Collections.Generic
open System.Reflection
open Xunit.Sdk
open System.IO

type public NewLineTextFileDataAttribute(inputFilepath:string, inputLineCount:int, outputFilepath:string, outputLineCount:int) =
  inherit DataAttribute()
  override this.GetData(testMethod:MethodInfo) =
    if testMethod = null then
      raise (ArgumentNullException(nameof(testMethod)))
    else   
      let readFile filepath lineCount =
        let results = new List<obj>()
        use file = File.OpenText(filepath)
        for _ in 1..lineCount do
          results.Add(file.ReadLine())
        results

      let currentDirectory = Directory.GetCurrentDirectory()
      let inputPath = 
        if Path.IsPathFullyQualified(inputFilepath) then inputFilepath
        else Path.Combine(currentDirectory, inputFilepath)

      let outputPath = 
        if Path.IsPathFullyQualified(outputFilepath) then outputFilepath 
        else Path.Combine(currentDirectory, outputFilepath)

      let getFiles (path:string) = 
        let rootInputPath = Path.GetPathRoot(path)
        if path.Contains('*') || path.Contains('?')  then  
          Directory.GetFiles(rootInputPath, Path.GetFileName(path), SearchOption.TopDirectoryOnly) 
        else [| path |]

      let files = Seq.zip (getFiles(inputPath)) (getFiles(outputPath))
      let results = new List<obj array>()
      for inputFile, outputFile in files do
        if File.Exists(inputFile) && File.Exists(outputFile) then
          let inputs = readFile inputFile inputLineCount
          let outputs = readFile outputFile outputLineCount
          inputs.AddRange(outputs)
          results.Add(inputs.ToArray())
        else
          raise (ArgumentException($"Unable to find files %s{inputPath} and %s{outputPath}"))

      results

And this is how I'm trying to run it:

namespace Bioinformatics.Tests

open Bioinformatics.Tests.Attributes

module Chapter1 =
  open Bioinformatics
  open Xunit    

  [<Theory>]
  [<NewLineTextFileData("Chapter1/PatternCount/inputs/*.txt", 2, "Chapter1/PatternCount/outputs/*.txt", 1)>]  
  let ``My test`` (text:string, pattern:string, expectedOutput:int) =
      let result = Chapter1.patternCount text pattern
      Assert.Equal(result, expectedOutput)

Originally everything was going smoothly, I was able to debug my NewLineTextFileDataAttribute class without any problems. However, when I added the feature where you can use the '*' or '?' character to search for specific file patterns, I'm now unable to debug my custom attribute and it started to throw this error:

========== Test discovery finished: 1 Tests found in 732.3 ms ==========
========== Starting test run ==========
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.5+1caef2f33e (64-bit .NET 7.0.3)
[xUnit.net 00:00:00.74] Bioinformatics.Tests: Catastrophic error deserializing item #1: System.ArgumentException: An item with the same key has already been added. Key: 076c8e82e08d85c0a1f8145c093310d6cba3129f
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at Xunit.Runner.VisualStudio.VsTestRunner.RunTestsInAssembly(IRunContext runContext, IFrameworkHandle frameworkHandle, LoggerHelper logger, TestPlatformContext testPlatformContext, RunSettings runSettings, IMessageSinkWithTypes reporterMessageHandler, AssemblyRunInfo runInfo) in /_/src/xunit.runner.visualstudio/VsTestRunner.cs:line 584
[xUnit.net 00:00:00.75]   Starting:    Bioinformatics.Tests
[xUnit.net 00:00:00.80]     Bioinformatics.Tests.Chapter1.My test [FAIL]
[xUnit.net 00:00:00.81]       System.InvalidOperationException : No data found for Bioinformatics.Tests.Chapter1.My test
Failed to add result for test 'Bioinformatics.Tests.Chapter1.My test' with ID '6144782c-ab6b-94ec-09e1-fb6a54964d65'.
[xUnit.net 00:00:11.32]   Finished:    Bioinformatics.Tests
========== Test run finished: 2 Tests (1 Passed, 1 Failed, 0 Skipped) run in 11.7 sec ==========

Another odd behavior is that when I debug the test, I'm able to get the data from the first input/output file, but after that it throws the error and fails:

[bradwilson]

This error usually means there is a duplicated data row, since the unique ID for the test is computed based on several pieces of data that are expected to be unique and that includes the data row when trying to get a single test case per data row.

If you cannot guarantee that the data rows are not duplicated, you will need to disable theory data pre-enumeration.

[jcuello]

Hi @bradwilson thanks for taking the time to answer, however I'm currently using DataAttribute to inject the data and it does not have that DisableDiscoveryEnumeration property. Do I have inherit from MemberDataAttributeBase instead? If so that kind of defeats the purpose of creating this custom DataAttribute since now I'll have to explicitly tell it what method to use on each test case.

[bradwilson]

@jcuello In v2, the only way to do this from DataAttribute is to also define your own data discoverer (derived from DataDiscoverer). You can use DataDiscoverer as the base class for your discoverer, override SupportsDiscoveryEnumeration to return false, and then decorate your NewLineTextFileDataAttribute with [DataDiscovererAttribute] like we do with DataAttribute: https://github.com/xunit/xunit/blob/19a9f685758cfab7efe8e63506b77bda6a14acfa/src/xunit.core/Sdk/DataAttribute.cs#L13

The two parameters are the fully qualified type name, and the assembly name where the type is defined.

[jcuello]

EDIT: I just created the test project in C# and it works as intended. I guess it must have been because xUnit doesn't play so well with F#.

@bradwilson I tried what you suggested and simplified my code (and fixed some bugs), but now I get a new error message:

========== Starting test run ==========
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.5+1caef2f33e (64-bit .NET 7.0.3)
[xUnit.net 00:00:00.62] Bioinformatics.Tests: Catastrophic error during deserialization: System.ArgumentNullException: Value cannot be null. (Parameter 'type')
   at System.ArgumentNullException.Throw(String paramName)
   at System.Reflection.RuntimeReflectionExtensions.GetRuntimeMethods(Type type)
   at Xunit.Sdk.ReflectionTypeInfo.GetMethod(String methodName, Boolean includePrivateMethod) in /_/src/xunit.execution/Sdk/Reflection/ReflectionTypeInfo.cs:line 97
   at Xunit.Sdk.TestMethod.Deserialize(IXunitSerializationInfo info) in /_/src/xunit.execution/Sdk/Frameworks/TestMethod.cs:line 53
   at Xunit.Serialization.XunitSerializationInfo.DeserializeSerializable(Type type, String serializedValue) in /_/src/common/XunitSerializationInfo.cs:line 207
   at Xunit.Serialization.XunitSerializationInfo.Deserialize(Type type, String serializedValue) in /_/src/common/XunitSerializationInfo.cs:line 110
   at Xunit.Serialization.XunitSerializationInfo.DeserializeTriple(String value) in /_/src/common/XunitSerializationInfo.cs:line 93
   at Xunit.Serialization.XunitSerializationInfo.DeserializeSerializable(Type type, String serializedValue) in /_/src/common/XunitSerializationInfo.cs:line 198
   at Xunit.Serialization.XunitSerializationInfo.Deserialize(Type type, String serializedValue) in /_/src/common/XunitSerializationInfo.cs:line 110
   at Xunit.Sdk.SerializationHelper.Deserialize[T](String serializedValue) in /_/src/common/SerializationHelper.cs:line 40
   at Xunit.Sdk.TestFrameworkExecutor`1.Deserialize(String value) in /_/src/xunit.execution/Sdk/Frameworks/TestFrameworkExecutor.cs:line 66
   at Xunit.Sdk.XunitTestFrameworkExecutor.Deserialize(String value) in /_/src/xunit.execution/Sdk/Frameworks/XunitTestFrameworkExecutor.cs:line 88
   at Xunit.DefaultTestCaseBulkDeserializer.<BulkDeserialize>b__2_0(String serialization) in /_/src/xunit.runner.utility/Descriptor/DefaultTestCaseBulkDeserializer.cs:line 22
   at System.Linq.Utilities.<>c__DisplayClass2_0`3.<CombineSelectors>b__0(TSource x)
   at System.Linq.Enumerable.SelectListIterator`2.ToList()
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Xunit.DefaultTestCaseBulkDeserializer.BulkDeserialize(List`1 serializations) in /_/src/xunit.runner.utility/Descriptor/DefaultTestCaseBulkDeserializer.cs:line 22
   at Xunit.Xunit2.BulkDeserialize(List`1 serializations) in /_/src/xunit.runner.utility/Frameworks/v2/Xunit2.cs:line 74
   at Xunit.XunitFrontController.BulkDeserialize(List`1 serializations) in /_/src/xunit.runner.utility/Frameworks/XunitFrontController.cs:line 122
   at Xunit.Runner.VisualStudio.VsTestRunner.RunTestsInAssembly(IRunContext runContext, IFrameworkHandle frameworkHandle, LoggerHelper logger, TestPlatformContext testPlatformContext, RunSettings runSettings, IMessageSinkWithTypes reporterMessageHandler, AssemblyRunInfo runInfo) in /_/src/xunit.runner.visualstudio/VsTestRunner.cs:line 565
[xUnit.net 00:00:00.62]   Starting:    Bioinformatics.Tests
[xUnit.net 00:00:00.67]   Finished:    Bioinformatics.Tests
========== Test run finished: 0 Tests (0 Passed, 0 Failed, 0 Skipped) run in 1.4 sec ==========

And here's the updated code, I don't know if it's F# yet, but I'll try with C# to see if I get the same error. If none of this still works I'll just close this issue. I already think I spent too much time on this.

namespace Bioinformatics.Tests.Attributes
open System
open System.Collections.Generic
open System.Reflection
open Xunit.Sdk
open System.IO
open Xunit.Abstractions

type public FileDataDiscoverer() = 
  inherit DataDiscoverer()
  override _.SupportsDiscoveryEnumeration(_: IAttributeInfo, _: IMethodInfo) = false

[<DataDiscoverer("Bioinformatics.Tests.Attributes.FileDataDiscoverer", "Bioinformatics.Tests")>]
type public NewLineTextFileDataAttribute(
  inputFilepath:string, inputLineCount:int,
  outputFilepath:string, outputLineCount:int) =
  inherit DataAttribute()

  override this.GetData(testMethod:MethodInfo) =
    if testMethod = null then
      raise (ArgumentNullException(nameof(testMethod)))
    else
      let readFile filepath lineCount =
        let results = new List<obj>()
        use file = File.OpenText(filepath)
        for _ in 1..lineCount do
          results.Add(file.ReadLine())
        results   

      let getFiles (path:string) = 
        let fullPath = Path.GetFullPath(path)
        let filePathDirectory = Path.GetDirectoryName(fullPath)
        if fullPath.Contains('*') || fullPath.Contains('?')  then  
          Directory.GetFiles(filePathDirectory, Path.GetFileName(fullPath), SearchOption.TopDirectoryOnly) 
        else Directory.GetFiles(filePathDirectory)

      let files = Seq.zip (getFiles inputFilepath) (getFiles outputFilepath)
      let results = new List<obj array>()
      for inputFile, outputFile in files do
        if File.Exists(inputFile) && File.Exists(outputFile) then
          let inputs = readFile inputFile inputLineCount
          let outputs = readFile outputFile outputLineCount
          inputs.AddRange(outputs)
          results.Add(inputs.ToArray())
        else
          raise (ArgumentException($"Unable to find files %s{Path.GetFullPath(inputFilepath)} and %s{Path.GetFullPath(outputFilepath)}"))

      results