UnkindPartition / tasty

Modern and extensible testing framework for Haskell
637 stars 108 forks source link

Not possible to figure out how to write patterns tasty-1.4.3. #400

Closed 1chb closed 7 months ago

1chb commented 9 months ago

I run via stack and the following works for me:

> stack test --test-arguments '-p foo'
> stack test --test-arguments '-p /foo/'

For everything more advanced I get: option -p: Could not parse pattern

> stack test --test-arguments '-p /foo/bar/'  -- suggested by ChatGPT 4
> stack test --test-arguments '-p $2\ ==\ "Two"' -- suggested by documentation

Edit: I found out another acceptable and usable syntax (not from doc but from suggestion after an error): > stack test --test-arguments '-p /foo/&&/bar/'

Bodigrim commented 9 months ago

It's non-trivial to guess how to escape arguments passed through --test-arguments. One layer is how your shell escapes things and another one is how Stack splits the string received into a list of arguments.

Please upgrade to tasty-1.5 (put it into extra-deps of your stack.yaml). The reason is that tasty-1.5 at least shows what exactly it received. E. g.,

$ stack test --test-arguments '-p $2\ ==\ "Two"'
your-package> test (suite: your-package-tests, args: -p $2\ ==\ Two)

option -p: Could not parse: $2\ is not a valid pattern

What is happenning is that despite your efforts to escape a space with \, Stack still decided that it's the end of the first argument. Even if we avoid spaces at all, the result is not amusing:

$ stack test --test-arguments '-p $2=="Two"'
your-package> test (suite: your-package-tests, args: -p $2== Two)

option -p: Could not parse: $2== is not a valid pattern

Somehow Stack decides to split arguments by ". I don't really know why, maybe an additional level of escaping is needed?.. @mpilgrem maybe you can advise? FWIW I think that Cabal has the same or worse habits.


Bottom line is that if --test-arguments mangles things, it's a question for Stack issue tracker. The recommended workaround is to compile test suite and run it manually with whichever flags desired. My go-to incantation is cabal run tests -- -p whatever, maybe Stack has the same?..

mpilgrem commented 9 months ago

Stack's argument parser is Data.Attoparsec.Args.argsParser (with escaping mode enabled for --test-arguments):

-- | A basic argument parser. It supports space-separated text, and
-- string quotation with identity escaping: \x -> x.
argsParser :: EscapingMode -> P.Parser [String]
argsParser mode = many (P.skipSpace *> (quoted <|> unquoted)) <*
                  P.skipSpace <* (P.endOfInput <?> "unterminated string")
 where
  unquoted = P.many1 naked
  quoted = P.char '"' *> str <* P.char '"'
  str = many ( case mode of
                 Escaping -> escaped <|> nonquote
                 NoEscaping -> nonquote
             )
  escaped = P.char '\\' *> P.anyChar
  nonquote = P.satisfy (/= '"')
  naked = P.satisfy (not . flip elem ("\" " :: String))
mpilgrem commented 9 months ago

I not entirely clear what arguments you are trying to pass, but if it is (a) -p and (b) $2 == "Two" then I think you need to specify -p "$2 == \"Two\"". EDIT: so stack test --ta -p --ta "$2 == \"Two\"".

Bodigrim commented 9 months ago

Thanks @mpilgrem, it works:

$ stack test --test-arguments '-p "$2 == \"Two\""'
your-package> test (suite: your-package-tests, args: -p "$2 == \"Two\"")

All 0 tests passed (0.00s)
mpilgrem commented 9 months ago

I'll improve Stack's online documentation of this.