search

karenetheridge / JSON-Schema-Modern

Validate data against a schema using a JSON Schema
https://metacpan.org/release/JSON-Schema-Modern/
Other
10 stars 1 forks source link

leap seconds tests fail on some systems #58

Closed eserte closed 2 years ago

eserte commented 2 years ago

On some of my smoker systems the test suite is failing:

    # Failed test 'evaluation result is incorrect'
    # at t/additional-tests-draft2019-09.t line 20.
    # expected true; got false

# Failed test 'format-date-time.json: "leap seconds" - "a valid date-time with a leap second, UTC"'
# at t/additional-tests-draft2019-09.t line 20.
    # Failed test 'evaluation result is incorrect'
    # at t/additional-tests-draft2019-09.t line 20.
    # expected true; got false

# Failed test 'format-date-time.json: "leap seconds" - "a valid date-time with a leap second, with minus offset"'
# at t/additional-tests-draft2019-09.t line 20.
# 
# 
# Results using Test::JSON::Schema::Acceptance 1.013
# specification version: draft2019-09
# using custom test directory: t/additional-tests-draft2019-09
# optional tests included: no
# 
# filename                                  pass  todo-fail  fail
# ---------------------------------------------------------------
# anchor.json                                  2          0     0
# badRef.json                                  3          0     0
# faux-buggy-schemas.json                      2          0     0
# format-date-time.json                        2          0     2
# format-date.json                             1          0     0
# format-duration.json                         9          0     0
# format-ipv4.json                             4          0     0
# format-relative-json-pointer.json            7          0     0
# format-time.json                            13          0     0
# id.json                                      6          0     0
# keyword-independence.json                  812          0     0
# loose-types-const-enum.json                  8          0     0
# recursive-dynamic.json                       2          0     0
# ref-and-id.json                              6          0     0
# ref.json                                    12          0     0
# short-circuit.json                          20          0     0
# unknownKeyword.json                          4          0     0
# vocabulary.json                              5          0     0
# ---------------------------------------------------------------
# TOTAL                                      918          0     2
# 
# WARNING: some non-optional tests are failing! This implementation is not fully compliant with the specification!
# 
# Looks like you failed 2 tests of 923.
t/additional-tests-draft2019-09.t ....... 
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/923 subtests 
    # Failed test 'evaluation result is incorrect'
    # at t/additional-tests-draft2020-12.t line 20.
    # expected true; got false

# Failed test 'format-date-time.json: "leap seconds" - "a valid date-time with a leap second, UTC"'
# at t/additional-tests-draft2020-12.t line 20.
    # Failed test 'evaluation result is incorrect'
    # at t/additional-tests-draft2020-12.t line 20.
    # expected true; got false

# Failed test 'format-date-time.json: "leap seconds" - "a valid date-time with a leap second, with minus offset"'
# at t/additional-tests-draft2020-12.t line 20.
# 
# 
# Results using Test::JSON::Schema::Acceptance 1.013
# specification version: draft2020-12
# using custom test directory: t/additional-tests-draft2020-12
# optional tests included: no
# 
# filename                                  pass  todo-fail  fail
# ---------------------------------------------------------------
# anchor.json                                  2          0     0
# badRef.json                                  3          0     0
# dynamicRef.json                              5          0     0
# faux-buggy-schemas.json                      2          0     0
# format-date-time.json                        2          0     2
# format-date.json                             1          0     0
# format-duration.json                         9          0     0
# format-ipv4.json                             4          0     0
# format-relative-json-pointer.json            7          0     0
# format-time.json                            13          0     0
# id.json                                      6          0     0
# keyword-independence.json                  884          0     0
# loose-types-const-enum.json                  8          0     0
# recursive-dynamic.json                       2          0     0
# ref-and-id.json                              6          0     0
# ref.json                                    12          0     0
# short-circuit.json                          20          0     0
# unknownKeyword.json                          4          0     0
# vocabulary.json                             11          0     0
# ---------------------------------------------------------------
# TOTAL                                     1001          0     2
# 
# WARNING: some non-optional tests are failing! This implementation is not fully compliant with the specification!
# 
# Looks like you failed 2 tests of 1006.
t/additional-tests-draft2020-12.t ....... 
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/1006 subtests 
    # Failed test 'evaluation result is incorrect'
    # at t/additional-tests-draft7.t line 20.
    # expected true; got false

# Failed test 'format-date-time.json: "leap seconds" - "a valid date-time with a leap second, UTC"'
# at t/additional-tests-draft7.t line 20.
    # Failed test 'evaluation result is incorrect'
    # at t/additional-tests-draft7.t line 20.
    # expected true; got false

# Failed test 'format-date-time.json: "leap seconds" - "a valid date-time with a leap second, with minus offset"'
# at t/additional-tests-draft7.t line 20.
# 
# 
# Results using Test::JSON::Schema::Acceptance 1.013
# specification version: draft7
# using custom test directory: t/additional-tests-draft7
# optional tests included: no
# 
# filename                                  pass  todo-fail  fail
# ---------------------------------------------------------------
# badRef.json                                  3          0     0
# faux-buggy-schemas.json                      2          0     0
# format-date-time.json                        2          0     2
# format-date.json                             1          0     0
# format-duration.json                         9          0     0
# format-ipv4.json                             4          0     0
# format-relative-json-pointer.json            7          0     0
# format-time.json                            13          0     0
# id.json                                      9          0     0
# keyword-independence.json                  636          0     0
# loose-types-const-enum.json                  8          0     0
# not-an-anchor.json                           2          0     0
# not-an-id.json                               2          0     0
# ref-and-id.json                              2          0     0
# ref.json                                    10          0     0
# short-circuit.json                          16          0     0
# unknownKeyword.json                          8          0     0
# vocabulary.json                              3          0     0
# ---------------------------------------------------------------
# TOTAL                                      737          0     2
# 
# WARNING: some non-optional tests are failing! This implementation is not fully compliant with the specification!
# 
# Looks like you failed 2 tests of 742.
t/additional-tests-draft7.t ............. 
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/742 subtests

Unfortunately I don't see a pattern when this is happening. Statistical analysis also does not give any regression candidates.

eserte commented 2 years ago

Possible reason: DateTime::Format::RFC3339 is missing or too old.

karenetheridge commented 2 years ago

Fixed in 0.523.