t/store.t fails

[eserte]

My smokers report the following failure:

#   Failed test at t/store.t line 37.
#     Structures begin differing at:
#          $got->{author}{id}[0]{description} = 'schilder, tekenaar, hofschilder, museumbeheerder'
#     $expected->{author}{id}[0]{description} = 'hofschilder, schilder, tekenaar'
# Looks like you failed 1 test of 2.
t/store.t .............. 
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/2 subtests 

Statistical analysis shows that there are no pass reports anymore with newer Catmandu versions (>= 1.2002):

****************************************************************
Regression 'mod:Catmandu'
****************************************************************
Name                   Theta          StdErr     T-stat
[0='const']           1.0000          0.0362      27.63
[1='eq_1.0306']       0.0000          0.1241       0.00
[2='eq_1.05']         0.0000          0.2401       0.00
[3='eq_1.0502']       0.0000          0.1717       0.00
[4='eq_1.0503']       0.0000          0.1417       0.00
[5='eq_1.0504']      -0.0000          0.1717      -0.00
[6='eq_1.0505']       0.0000          0.2401       0.00
[7='eq_1.0506']       0.0000          0.1241       0.00
[8='eq_1.0601']       0.0000          0.2401       0.00
[9='eq_1.0602']      -0.3333          0.0712      -4.68
[10='eq_1.0603']         -0.4167          0.0775      -5.38
[11='eq_1.0604']         -1.0000          0.1121      -8.92
[12='eq_1.0605']         -0.0000          0.2401      -0.00
[13='eq_1.0606']         -0.9545          0.0622     -15.34
[14='eq_1.07']       -0.0000          0.1034      -0.00
[15='eq_1.08']       -0.0000          0.1417      -0.00
[16='eq_1.09']       -0.0000          0.0914      -0.00
[17='eq_1.10']       -0.1250          0.0695      -1.80
[18='eq_1.20']       -0.0000          0.1717      -0.00
[19='eq_1.2002']         -1.0000          0.2401      -4.17
[20='eq_1.2007']         -1.0000          0.1417      -7.06
[21='eq_1.2009']         -1.0000          0.1241      -8.06
[22='eq_1.2010']         -1.0000          0.0613     -16.31

R^2= 0.791, N= 182, K= 23
****************************************************************

[eserte]

Just realized that this issue tracker is for both Catmandu-RKD and Catmandu-Store-RKD. But both show the same failure.

[netsensei]

This package contains an implementation of a remote API. This is a functional test that fails because it tries to assert the output based on external data, which is tenuous.

Instead of sending a query against the remote API and comparing the live output, this test should assert whether or not functionality works against a mock of the API.

i.e. use something like https://metacpan.org/pod/Test::MockModule

[eserte]

I am not a fan of mocking, because mocking may hide real problems. On the other hand, accessing real live interfaces which may change means that an author needs to be prepared to do test-only releases every now and then. Just my $0.02.

[netsensei]

That's valuable feedback! Thanks!

Yes, interfaces may change their contract, and testing against that, instead of a mock, may help in catching these changes in a timely fashion. This pertains to modelling through URI's, data structures (naming of fields, entities, relationships,...), authentication,...

Then there's the data itself: values, data or information. Functional tests could test whether a value is present or absent if that directly ties into the API contract (i.e. the required presence of a field). But I don't think they should do hard assertions against the actual - changeable - information returned by the API.

So, my recommendation would be: use a regular expression to check whether the value returned by the API adheres to relevant constraints:

• Whether it can or can't be empty.
• Expected structure of the value (i.e. value MUST be an URI, MUST be numeric,...)
• Length of the value (i.e. max 255 characters)
• ...

Even so, we have to be wary: we're testing the client implementation; not the API itself.