nimble run_conformance_tests: CONFORMANCE SUITE FAILED: 9 unexpected failures

[timotheecour]

@oswjk did it ever work? before https://github.com/oswjk/nimpb/pull/10 I ran into https://github.com/oswjk/nimpb/issues/4; after https://github.com/oswjk/nimpb/pull/10 I get:

ERROR, test=Required.Proto3.ProtobufInput.RepeatedScalarSelectsLast.DOUBLE.JsonOutput: Output was not equivalent to reference message: modified: optional_double: 2.2250738585072014e-308 -> 2.2250738585072009e-308
. request=protobuf_payload: "a\232\231\231\231\231\231\271?a\377\377\377\377\377\377\357\177a\000\000\000\000\000\000\020\000" requested_output_format: JSON message_type: "protobuf_test_messages.proto3.TestAllTypesProto3", response=json_payload: "{\"optionalDouble\":2.225073858507201e-308}"
ERROR, test=Required.Proto3.ProtobufInput.ValidDataRepeated.DOUBLE.JsonOutput: JSON output we received from test was unparseable. request=protobuf_payload: "\321\002\232\231\231\231\231\231\271?\321\002\377\377\377\377\377\377\357\177\321\002\000\000\000\000\000\000\020\000" requested_output_format: JSON message_type: "protobuf_test_messages.proto3.TestAllTypesProto3", response=json_payload: "{\"repeatedDouble\":[0.1,1.797693134862316e+308,2.225073858507201e-308]}"
WARNING, test=Recommended.Proto3.JsonInput.TrailingCommaInAnObject: Should have failed to parse, but didn't. request=json_payload: "{\"fieldname1\":1,}" requested_output_format: JSON message_type: "protobuf_test_messages.proto3.TestAllTypesProto3", response=json_payload: "{\"fieldname1\":1}"
WARNING, test=Recommended.Proto3.JsonInput.TrailingCommaInAnObjectWithSpace: Should have failed to parse, but didn't. request=json_payload: "{\"fieldname1\":1 ,}" requested_output_format: JSON message_type: "protobuf_test_messages.proto3.TestAllTypesProto3", response=json_payload: "{\"fieldname1\":1}"
WARNING, test=Recommended.Proto3.JsonInput.TrailingCommaInAnObjectWithSpaceCommaSpace: Should have failed to parse, but didn't. request=json_payload: "{\"fieldname1\":1 , }" requested_output_format: JSON message_type: "protobuf_test_messages.proto3.TestAllTypesProto3", response=json_payload: "{\"fieldname1\":1}"
WARNING, test=Recommended.Proto3.JsonInput.TrailingCommaInAnObjectWithNewlines: Should have failed to parse, but didn't. request=json_payload: "{\n        \"fieldname1\":1,\n      }" requested_output_format: JSON message_type: "protobuf_test_messages.proto3.TestAllTypesProto3", response=json_payload: "{\"fieldname1\":1}"
WARNING, test=Recommended.Proto3.JsonInput.JsonWithComments: Should have failed to parse, but didn't. request=json_payload: "{\n        // This is a comment.\n        \"fieldname1\": 1\n      }" requested_output_format: JSON message_type: "protobuf_test_messages.proto3.TestAllTypesProto3", response=json_payload: "{\"fieldname1\":1}"

...

[oskaritimperi]

There are a few conformance tests that are failing, but shouldn't affect normal usage much. Like those JSON input tests: failures come from the fact that Nim stdlib json module allows that kind of usage. That's why those are marked as WARNING, not as ERROR.

Then there are a few tests that are really failing. Those are marked as ERROR.

I can see that there is a small regression from 2 unexpected failures to 9. The previous list of failures can be found from https://github.com/oswjk/nimpb/blob/master/tests/conformance/failures.txt

These are currently failing:

  Required.Proto3.JsonInput.Int32FieldLeadingZero
  Required.Proto3.JsonInput.Int32FieldNegativeWithLeadingZero
  Required.Proto3.JsonInput.Struct.ProtobufOutput
  Required.Proto3.JsonInput.Uint64FieldMaxValueNotQuoted.JsonOutput
  Required.Proto3.JsonInput.Uint64FieldMaxValueNotQuoted.ProtobufOutput
  Required.Proto3.JsonInput.ValueAcceptNull.JsonOutput
  Required.Proto3.JsonInput.ValueAcceptNull.ProtobufOutput
  Required.Proto3.ProtobufInput.RepeatedScalarSelectsLast.DOUBLE.JsonOutput
  Required.Proto3.ProtobufInput.ValidDataRepeated.DOUBLE.JsonOutput

[timotheecour]

could that be due to not-nil strings / seqs and the changes from https://github.com/oswjk/nimpb/pull/10 ?

[oskaritimperi]

Yes. That, and I made a change to make the parseInt (for parsing an integer from a JsonNode) compile. Previously I had two parseInt procs with different generic args and that worked, but it failed to compile with 0.19. So I changed the unsigned one to parseUInt and that is now somehow messed up somewhere.