Read file from tests

[advait]

Hello!

It would be really awesome if we were able to read files from our elm tests! As an example, consider canned JSON responses from an API. It would be great to keep these as separate .json files instead of bringing them into my elm source files as multi-line strings.

See this related Stack Overflow post: https://stackoverflow.com/questions/48378601/load-external-data-into-elm-test-suite

The solution in the post proposed using Native Modules. Unfortunately, because native modules have been (mostly) deprecated as of 0.19, it seems that the only way to reasonably use this approach to read files would be through elm-explorations/test. I think this a really reasonable use case for testing and would like to propose it as a feature request. I'm happy to help propose what an API might look like and implement a prototype if there is support from the admins.

Thanks, Advait

[MartinSStewart]

To add to this idea, what if there was a new function Test.testWithFiles : String -> List String -> (Dict String Bytes -> Expectation) -> Test? List String would be a list of file paths and Dict String Bytes would be the filepath and data contained in the file. If a file can't be found then that test fails.

One thing I'm not sure about is how fuzz testing should be handled. Should there just be a fuzzWithFiles, fuzz2WithFiles, fuzz3WithFiles, and fuzzWith_WithFiles?

[lydell]

Random note: node-test-runner and elm-test-rs probably want to watch the provided file paths for changes, to trigger test re-runs if they change.

Would be nice to have an example of what using a single JSON file (with some decoder) in a test could look like with Test.testWithFiles.

[jfmengels]

I don't know if working with Bytes would be that nice. My initial reaction is that I wouldn't know how to transform that to a String or a more practical type.

My initial feeling is that we could/should have a way to transform the data into something that is usable. If you want to load some data (like a user session or a Model snapshot) as a JSON file, then you indicate which file to read as well as how to interpret it as more palatable data.

I was thinking of an API like the following:

Test.testWithFiles "my test name"
  [ ( "fixtures/data.json", Test.File.json myDecoder ) ]
  (\filesDict ->
    case Dict.get "fixtures/data.json" filesDict of
        Just data ->
           -- My assertion...
        Nothing ->
           -- Should be impossible?
           Expect.fail "Did not load the data file somehow"
  )

where Test.File.json myDecoder would lead the file to be considered to be a JSON string, to be decoded with myDecoder (and if that fails, then the whole test fails with the decoding error message).

If instead of providing data fixtures, the intent for the test is to make sure that a decoder can correctly decode a JSON file, then one could do

Test.testWithFiles "my test name"
  [ ( "api_response/data.json", Test.File.json myDecoder ) ]
  (\filesDict ->
    case Dict.get "fixtures/data.json" filesDict of
        Just data ->
           -- My assertion...
        Nothing ->
           -- Should be impossible?
           Expect.fail "Did not load the data file somehow"
  )

That said, after writing all of this, the problem with the above approach is that it doesn't work in practice, since you'd have a Dict with potentially different value types. It does work however if you work with a pre-determined number of elements.

And I would probably imagine that the common use-case would be to load a single file, rather than multiple. In that case, it would be more practical to have a variant that only takes a single file (and have the test automatically fail if it could not fail).

Test.testWithFile "my test name"
  ( "fixtures/data.json", Test.File.json myDecoder )
  (\data ->
     -- My assertion...
  )

which feels a lot smoother, because I don't have to pattern match on data that elm-test has already verified. I think it could make sense to have a testWithFile2, testWithFile3, etc. to have the same benefit (and would resemble fuzz2, fuzz3, ...), and to actually type-check.

Even if we don't go with my suggested approach of trying to decode things ahead of the body's implementation and we go with the Bytes approach, I believe having testWithX functions would be nice because it prevents the case Dict.get ... of bit.

That said, my proposal is somewhat easily re-creatable (in the package or in user-land) from @MartinSStewart's base proposal. Having the base capability is the most important bit (but having a nice API is obviously a wish)

---

There's also the question of whether we should want to support globbing ("get all the files like fixtures/*.json), but I'm not sure this is useful at this point. Or rather, I'm not sure that I see particular test cases that would really benefit from this.

[MartinSStewart]

I didn't think too long on my API proposal. There's probably something nicer out there. That said,

I don't know if working with Bytes would be that nice. My initial reaction is that I wouldn't know how to transform that to a String or a more practical type.

could be addressed by having a Test.bytesToString : Bytes -> Maybe String helper function.

[lydell]

API idea I got (which replaces fuzz, fuzz2, fuzzWith):

tests =
    describe "stuff"
        [ Test.test "good old test" <|
            \_ ->
                Expect.equal 1 1
        , Test.coolTest "test with fuzzers and/or files"
            |> Test.withFuzzer Fuzz.int
            |> Test.withFuzzer Fuzz.float
            |> Test.withFuzzRuns 100
            |> Test.withFuzzDistribution myDistribution
            |> Test.withStringFile "test.txt"
            |> Test.withStringFile "test2.txt"
            |> Test.withJsonFile "names.json" (Json.Decode.list Json.Decode.string)
            |> Test.withBytesFile "test.jpg" myBytesDecoder
            |> Test.run
                (\fuzzedInt fuzzedFloat testFile1 testFile2 names bytes ->
                    Expect.equal 1 1
                )
        ]

This makes it easy to combine fuzzers and files, and to provide decoders for files. It also avoids having to do lookups in a dict.

I’m not sure if that API is possible, but it feels like it could be.

[gampleman]

I like that idea, since it reduces the total number of functions that just combine orthogonal concepts. You could use that also to add tabular tests, etc.

[MartinSStewart]

@lydell's approach is my favorite. I tried making the types work in a slightly different context but couldn't but that could just be my own limitations or it not being exactly the same situation. Worst case though I'm pretty sure it will work if you move the test code into coolTest as a second parameter.

[mpizenberg]

I also like @lydell idea. @MartinSStewart I have something like that in a parser context.

succeed (\a b d -> ...)
  |> keep parseA
  |> keep parseB
  |> ignore parseC
  |> keep parseD

keep : Parser a -> Parser (a -> b) -> Parser b
keep val fun =
    map2 (<|) fun val

ignore : Parser ignore -> Parser keep -> Parser keep
ignore skipper keeper =
    map2 always keeper skipper

It reverses the way @lydell wrote it since the runner is provided first instead of last but maybe it should be possible to write it in the reverse way?

[mpizenberg]

Worst case though I'm pretty sure it will work if you move the test code into coolTest as a second parameter.

ah right, ok you got it.