Valid json (?) is refused

mratsim commented 3 years ago

I'm trying to isolate a bug with distinct type and create a minimal repro but I ended up getting stuck in another issue:

import jsony

type
  SecretWord = distinct uint64

  CryptographicType = object
    data: array[4, SecretWord]

  TestVector[T] = object
    vectors: seq[T]

proc parseHook(src: string, pos: var int, value: var CryptographicType) =
  var str: string
  parseHook(src, pos, str)
  discard "value.fromHex(str)"

let s = """
{
    "vectors": [{
        "data": "0x2523648240000001ba344d80000000086121000000000013a700000000000013"
    }]
}
"""
let v = s.fromJson(TestVector[CryptographicType])

According to https://jsonlint.com/ this is valid JSON but somehow I get

/[...]/Programming/Nim/constantine/build/jsony_distinct.nim(24) jsony_distinct
/[...]/.nimble/pkgs/jsony-0.0.5/jsony.nim(434) fromJson
/[...]/.nimble/pkgs/jsony-0.0.5/jsony.nim(318) parseHook
/[...]/.nimble/pkgs/jsony-0.0.5/jsony.nim(170) parseHook
/[...]/Programming/Nim/constantine/build/jsony_distinct.nim(14) parseHook
/[...]/.nimble/pkgs/jsony-0.0.5/jsony.nim(44) parseHook
Error: unhandled exception: Expected ". At offset: 15 [JsonError]

End goal

I'll open an issue if this ends up being an issue in jsony

I am trying to replace nim-json-serialization for my test vectors, it brings many dependencies and one is crashing my CI at the moment for a git clone issue: https://github.com/mratsim/constantine/pull/155/checks?check_run_id=1865342674#step:15:60 and unfortunately there is no easy way out of it until Nimble supports task level dependencies (it's a library tested with nim-json-serialization that is brought) (https://github.com/nim-lang/nimble/issues/482).

I use distinct types in there but they are all caught by the 2 readValue in nim-json-serialization (https://github.com/mratsim/constantine/blob/c4a2dee/tests/t_ec_sage_template.nim#L95-L178) and example file (https://github.com/mratsim/constantine/blob/c4a2dee/tests/vectors/tv_BN254_Nogami_scalar_mul_G1.json). Somehow at the moment jsony is trying to parseHook those.

/[...]/Programming/Nim/constantine/tests/t_ec_sage_bls12_381.nim(18, 28) template/generic instantiation of `run_scalar_mul_test_vs_sage` from here
/[...]/Programming/Nim/constantine/tests/t_ec_sage_template.nim(186, 26) template/generic instantiation of `loadVectors` from here
/[...]/Programming/Nim/constantine/tests/t_ec_sage_template.nim(172, 19) template/generic instantiation of `fromJson` from here
/[...]/.nimble/pkgs/jsony-0.0.5/jsony.nim(434, 4) template/generic instantiation of `parseHook` from here
/[...]/.nimble/pkgs/jsony-0.0.5/jsony.nim(318, 20) template/generic instantiation of `parseHook` from here
/[...]/.nimble/pkgs/jsony-0.0.5/jsony.nim(170, 14) template/generic instantiation of `parseHook` from here
/[...]/.nimble/pkgs/jsony-0.0.5/jsony.nim(318, 20) template/generic instantiation of `parseHook` from here
/[...]/.nimble/pkgs/jsony-0.0.5/jsony.nim(197, 14) Error: type mismatch: got <string, int, SecretWord>
but expected one of: 
proc parseHook(s: string; i: var int; v: var JsonNode)
  first type mismatch at position: 3
  required type for v: var JsonNode
  but expression 'value' is of type: SecretWord
proc parseHook(s: string; i: var int; v: var SomeFloat)
  first type mismatch at position: 3
  required type for v: var SomeFloat
  but expression 'value' is of type: SecretWord

mratsim commented 3 years ago

Edit: I didn't solve the original issue but found what was wrong for the end goal, it seems like due to Nim visibility constraint with generics parseHook must be exported or Nim won't use the proper one. This might need an entry in the README

treeform commented 3 years ago

I could make the error message better. Its not saying the json is invalid is saying it's expecting something like a string but gets some thing else.

You had hook on CryptographicType, but that contains {"data":.... still but you where asking for a string. I changed it to look for var array[4, SecretWord] instead.

... and yes the parse hook needs to be visible where you are doing the parsing. I should add that to the doc.

This works:

import jsony, strutils

type
  SecretWord = distinct uint64

  CryptographicType = object
    data: array[4, SecretWord]

  TestVector[T] = object
    vectors: seq[T]

proc parseHook(src: string, pos: var int, value: var array[4, SecretWord]) =
  var str: string
  parseHook(src, pos, str)
  for i in 0 ..< 4:
    let hexStr = str[2 + i*16 ..< 2 + i*16 + 16]
    echo hexStr
    value[i] = SecretWord(fromHex[uint64](hexStr))

let s = """
{
    "vectors": [{
        "data": "0x2523648240000001ba344d80000000086121000000000013a700000000000013"
    }]
}
"""
let v = s.fromJson(TestVector[CryptographicType])
echo v

treeform commented 3 years ago

Added a message about needed to export as well as have all examples use exports stars.

https://github.com/treeform/jsony/commit/907bd3cdbda8f6a0ec0bc2f4aa3cbc40b22c9ef7

treeform / jsony

Valid json (?) is refused #8

End goal