Non-deterministic codegen for Option deserialization

[hubertp]

I'm trying to replace Jackson serialization with jsoniter and encountering some issues with serialization.

The problems appears to be during deserialization of Option values in case classes. Macro for writing foo: Option[String] field will always generate something similar to

    Option vxx = x.foo();
    if (vxx != None.MODULE$) {
        out.writeNonEscapedAsciiKey("foo");
        out.writeVal((String)vxx.get());
    }

(decompiled code, for reference only)

While reading will generate either 1:

    case -244668668:
        if (in.isCharBufEqualsTo(l, "foo")) {
            if ((p0 & 32) == 0) {
                throw in.duplicatedKeyError(l);
            }

            p0 ^= 32;
            if (in.isNextToken((byte)110)) {
                var10000 = (Option)in.readNullOrError(_foo, "expected value or null");
            } else {
                in.rollbackToken();
                var10000 = new Some(in.readString((String)null));
            }

            _foo = var10000;
        } else {
            in.skip();
        }
        break;

or 2:

    case -244668668:
        if (in.isCharBufEqualsTo(l, "foo")) {
            if ((p0 & 32) == 0) {
                throw in.duplicatedKeyError(l);
            }

            p0 ^= 32;
            _foo = this.d99(in, (Option)_foo);
        } else {
            in.skip();
        }
        break;

where d99 is roughly generated as

    private Option<String> d99(final JsonReader in, final Option<String> default) {
        in.setMark();
        if (in.isNextToken((byte)123)) {
            if (in.isCharBufEqualsTo(in.readKeyAsCharBuf(), "type")) {
                int l = in.readStringAsCharBuf();
                if (in.isCharBufEqualsTo(l, "None")) {
                    in.rollbackToMark();
                    return this.d101(in, scala.None..MODULE$);
                } else if (in.isCharBufEqualsTo(l, "Some")) {
                    in.rollbackToMark();
                    return this.d100(in, (Some)null);
                } else {
                    throw in.discriminatorValueError("type");
                }
            } else {
                throw in.decodeError("expected key: \"type\"");
            }
        } else {
            return (Option)in.readNullOrTokenError(default, (byte)123);
        }
    }

Obviously, given the serialization code, I would expect only option 1, which is generated in the macro. But sometimes it generates deserialization for JSON value in 2 and complains with the usual

[info]   com.github.plokhotnyuk.jsoniter_scala.core.JsonReaderException: expected '{' or null, offset: 0x00000447, buf:
[info] +----------+-------------------------------------------------+------------------+
[info] |          |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f | 0123456789abcdef |
[info] +----------+-------------------------------------------------+------------------+
[info] | 00000420 | 61 6e 64 61 72 64 2e 42 61 73 65 2e 44 61 74 61 | andard.Base.Data |
[info] | 00000430 | 2e 53 65 74 2e 53 65 74 22 2c 22 70 61 72 65 6e | .Set.Set","paren |
[info] | 00000440 | 74 54 79 70 65 22 3a 22 53 74 61 6e 64 61 72 64 | tType":"Standard |
[info] | 00000450 | 2e 42 61 73 65 2e 41 6e 79 2e 41 6e 79 22 7d 2c | .Base.Any.Any"}, |
[info] | 00000460 | 22 61 63 74 69 6f 6e 22 3a 7b 22 74 79 70 65 22 | "action":{"type" |
[info] +----------+-------------------------------------------------+------------------+

where parentType: Option[String]. PR demonstrating the issue https://github.com/enso-org/enso/pull/10035 in a target project. Anyone encountered this as well?

[plokhotnyuk]

@hubertp Hi, Hubert!

Thanks for trying and sending your feedback!

Generated codecs are always the same for the same jsoniter-scala version, derivation configuration and a way of injection of custom codecs using implicit val and def functions.

I've tried to compile polyglot-api module from your demonstrating PR, but, unfortunately, it has a lot of required prerequisites to be installed...

Do you have any isolated example to see your expectations and results clearly?

From your PR I learned that jsoniter-scala docs should be improved to clearly state that CodecMakerConfig can be used only in compile time, as an expression parameter of JsonCodecMaker.make call. If you want to have derivation configuration in one place then need to create a separated module with a macro function that has your pre-configured JsonCodecMaker.make implementation. So after that you can call your macro instead of JsonCodecMaker.make everywhere without copying the configuration expression.

[hubertp]

Thanks for the info. Moving serde logic to a separate module appears to have solved the problem. Probably a combination of incremental compilation + macros led to this non-determinism.