search

JuliaTesting / ReferenceTests.jl

Utility package for comparing data against reference files
https://juliatesting.github.io/ReferenceTests.jl/latest/
Other
82 stars 14 forks source link

[WIP] make pipeline cleaner (Cont. #33) #36

Open johnnychen94 opened 4 years ago

johnnychen94 commented 4 years ago

Cont. #33

In this PR, a "constant" global runtime flag TestMode is introduced so that it's easier to handle environment variables mentioned in #8

This may have many use cases, for example, in this PR I dispatch_create_new_reference and _update_reference on TestMode.

The first commit doesn't change the behavior of reference_test. I'd love to hear advice about what you'd like to have in further commits.

coveralls commented 4 years ago

Coverage Status

Coverage decreased (-3.9%) to 78.333% when pulling 6b3e9032bedbfe393f985ccbbef9e04300ca4cbc on jc/pipeline into 46d650a2efc73177bc9df7408a579c78488c6849 on master.

johnnychen94 commented 4 years ago

The coverage decreases because CI can't capture codes for InteractiveMode. Any advice on how to test this?

johnnychen94 commented 4 years ago

I'm not satisfied with the current codes in this PR, but I don't have better ideas now.

One key question is: what advantages it brings to us if it doesn't only rely on isinteractive for triggering the input prompt? (#8 ) The only use case I can think about is to automatically generate all reference files(but not update) in a non-interactive mode (even when isinteractive()==true).

oxinabox commented 4 years ago

I like this concept overall.

I think InteractiveMode is wrong name for this. We have 3 behavours:

In all of them you get at least a warning and a display of the differences.

And those behavours can be applied in 2 circumstances:

We can handle the combination via multiple dispatch.

There is a case to be made for making the default setting be: Fail on Disagree, Create on Missing. That way if you knew a bunch of things had changed, you could delete all the ones you expect to be invalidated. Then rerun the tests.

Or if you expect everything to be invalidated you just switch to Create on Both.

Having used this inpractice a bit, this is fine, because one reviews the new referenced files during code-review.