search

NorfairKing / sydtest

A modern testing framework for Haskell with good defaults and advanced testing features.
113 stars 25 forks source link

American spellings don't work in config file #75

Open brandonchinn178 opened 12 months ago

brandonchinn178 commented 12 months ago

randomize-execution-order: false doesn't seem to work in the config file, but randomise-execution-order: false does. Is that expected?

NorfairKing commented 12 months ago

@brandonchinn178 expected: yes. PR welcome: also yes. I wouldn't mind supporting both.

FWIW: there's a really good reason why the default is true and you probably want to use sequential instead. Do you want to show me where you're using sydtest? Maybe I can make a helpful suggestion.

brandonchinn178 commented 12 months ago

I understand why the default is true; in general, I agree that randomizing the test order is good to prevent a implicit dependency on the order of tests.

In my case, all my tests are unit tests and I want the output order to match the order in the source code, to match the spec. It's just aesthetics, so I don't think sequential is the right thing here.

brandonchinn178 commented 12 months ago

Also, can you explain why this is expected? I do see this: https://github.com/NorfairKing/sydtest/blob/a3ba5f3096c3dcc58920f61ba8a45191d9020eb4/sydtest/src/Test/Syd/OptParse.hs#L268-L269

And the help text does say it's allowed:

# any of
[ randomise-execution-order: # optional
    # Randomise the execution order of the tests in the test suite
    <boolean>
, randomize-execution-order: # optional
    # American spelling
    <boolean>
]

--no-randomize-execution-order also works, it just doesn't work in the config file

NorfairKing commented 11 months ago

@brandonchinn178 It looks like I made two mistakes in one message:

  1. I was supposed to say doNotRandomiseExecutionOrder instead of sequential.
  2. I mistakenly thought the American spelling wasn't supposed to be supported yet.

It also looks like there's an issue with the parsing. I'm investigating now if it's autodocodec that's broken.

NorfairKing commented 11 months ago

@brandonchinn178 I just managed to reproduce this failure. Autodocodec has tests against this, so I'll continue digging.

NorfairKing commented 11 months ago

I've added another test to autodocodec and haven't found the issue, but now I think I have an idea what the issue is.

NorfairKing commented 11 months ago

I figured out why this happened:

https://github.com/NorfairKing/autodocodec/commit/c8d56ace5b4c5b1137d9a8eaac11c64531e9be14#diff-d42217527abcf98c817a6d39cb84537548449dbefe2ce1eb569dcc249a935331R1618-R1627

It turns out that I misunderstood (my own) autodocodec API and caused this bug. I'll have to revisit the autodocodec API to make this kind of issue solveable.