Weird random errors during repeated validations

[NGEfreak]

I have a schema with several type => ['string', 'null'] constraints that randomly fail with valid data.

Here is an example script:

#!/usr/bin/env perl

use 5.010;
use strict;
use warnings;
use utf8;

use JSON::Validator;

my $schema = {
    items => {
        properties => {
            prop1 => { type => [qw(string null)] },
            prop2 => { type => [qw(string null)], format => 'ipv4' },
            prop3 => { type => [qw(string null)], format => 'ipv4' },
            prop4 => { type => 'string', enum => [qw(foo bar)] },
            prop5 => { type => [qw(string null)] },
            prop6 => { type => 'string' },
            prop7 => { type => 'string', enum => [qw(foo bar)] },
            prop8 => { type => [qw(string null)], format => 'ipv4' },
            prop9 => { type => [qw(string null)] },
        },
        type => 'object',
    },
    type => 'array',
};

my $data = [ {
    prop1 => undef,
    prop2 => undef,
    prop3 => undef,
    prop4 => 'foo',
    prop5 => undef,
    prop6 => 'foo',
    prop7 => 'bar',
    prop8 => undef,
    prop9 => undef,
} ];

for my $i ( 1 .. 1000 ) {
    my $validator = JSON::Validator->new();
    $validator->schema($schema);

    my @errors = $validator->validate($data);
    if (@errors) {
        say "failed $i: @errors";
        exit;
    }
}

say 'ok';

For example, running this script 20 times I got this result:

$ for run in {1..20}; do perl example.pl; done

ok
ok
failed 170: /0/prop1: /anyOf/1 Expected string - got null.
ok
failed 1: /0/prop5: /anyOf/1 Expected string - got null.
failed 86: /0/prop1: /anyOf/1 Expected string - got null.
ok
ok
failed 1: /0/prop5: /anyOf/1 Expected string - got null.
failed 27: /0/prop1: /anyOf/1 Expected string - got null. /0/prop5: /anyOf/1 Expected string - got null.
failed 171: /0/prop1: /anyOf/1 Expected string - got null.
ok
ok
failed 3: /0/prop1: /anyOf/1 Expected string - got null.
ok
failed 3: /0/prop5: /anyOf/1 Expected string - got null.
failed 1: /0/prop3: /anyOf/1 Expected string - got null.
failed 29: /0/prop1: /anyOf/1 Expected string - got null.
failed 5: /0/prop3: /anyOf/1 Expected string - got null.
failed 53: /0/prop1: /anyOf/1 Expected string - got null.

It doesn't matter if I move the JSON::Validator initialization outside the for-loop.

I have also tried some simpler schemas:

• same schema without format constraints => still fails
• same schema without enum constraints => doesn't fail as often anymore
• same schema with one property removed (for example prop9) => still fails
• same schema with two properties removed (for example prop8 and prop9) => doesn't appear to fail anymore

Since this problem is really strange and perhaps related to my Perl installation I have also run this code on some other machines. However, the problem occurs on all of them:

• Perl 5.24.1 on Debian (perl-5.24.1-3+deb9u4)
• Strawberry Perl 5.24.4 on Windows 10
• Strawberry Perl 5.28.1 on Windows 10

I have also tried several older versions but with no difference:

• JSON::Validator 2.18, Mojolicious 8.07
• JSON::Validator 2.18, Mojolicious 8.09
• JSON::Validator 2.17, Mojolicious 8.09
• JSON::Validator 2.16, Mojolicious 8.09
• JSON::Validator 2.15, Mojolicious 8.09
• JSON::Validator 2.14, Mojolicious 8.09
• JSON::Validator 2.13, Mojolicious 8.09

EDIT: Just tried it on Red Hat with Perl 5.16.3 and it looks like it works! Perhaps hash order randomization (since 5.18.x) is the cause of the problem?

[jhthorsen]

Could it be this line that needs improvements? https://github.com/jhthorsen/json-validator/blob/master/lib/JSON/Validator.pm#L485

[NGEfreak]

Yes, that line is part of the problem. I have attached some files with an example.

good.txt contains a debug report of a successful validation. bad.txt contains the debug report of the failed validation. Both are from the same process.

method_calls.txt lists all _validate() calls of the bad run. Each segment shows @_ (without invocant) and $seen_addr.

As you can see the memory address 94152271142056 of $schema is the same in call 10 and 20. Of course, the content of $schema is different in both calls. This means that the reference of call 10 gets destroyed and the memory address is set free again. Later, the memory address is reused again during call 20.

bad.txt good.txt method_calls.txt

[jhthorsen]

I hope 2.19 fixes this issue. I've added random-errors.t which reflects the test case you created above.