Feature Request: Improve support for encoding and decoding 64-bit integers to and from JSON

[austinmilt]

Problem

Currently there is no way to safely encode and decode 64-bit integers to and from JSON using dart:convert. For example:

import 'dart:convert';

void main() {
  print(jsonDecode("18446744073709551615"));
}

prints 18446744073709552000 on web due to well-documented truncation

and trying to encode a BigInt - recommended above for arbitrarily-sized integers -

import 'dart:convert';

void main() {
  print(jsonEncode(BigInt.parse("18446744073709551615")));
}

results in an error since BigInt is not supported for JSON encoding by default.

While JSON and JavaScript are heavily related and therefore it's normal to expect JS levels of precision, JSON is used widely for non-web applications.

Existing workarounds are insufficient

Right now, the most straightforward workaround is to encode and decode BigInt to and from String by passing custom converters to json.encode. However, this creates an ambiguous string value at the other end of the wire when performing remote calls, which could be cumbersome if the other end is in a different language such that encoding/decoding libraries aren't shared. It also makes the JSON less human-readable.

Other workarounds have similar issues.

Qualities of a better approach

It is prudent for dart:convert to support 64-bit (and larger) integers out-of-the-box. A better approach should

1. Avoid silent failures such as truncation.
2. Enable developers to decide their own tradeoffs wherever they are using JSON.
3. Facilitate low-dependency development where core package features can solve major issues with incremental changes.

Option A - Allow overriding stringifying objects and decoding tokens

With this nonbreaking change, developers can customize how an object is added to the JSON string produced by jsonEncode. This is a layer deeper than the current toEncodable argument which is limited to return JSON-encodable Dart objects. This would enable behavior like jsonEncode({'a': BigInt.from(1)}) => '{"a": 1}' rather than the current workaround like jsonEncode({'a': BigInt.from(1)}) => '{"a": "1"}' (where BigInt becomes JSON string).

Also with this change, developers can customize how a JSON token is converted from a string back into an object within jsonDecode. This would enable a developer to reverse the flow above to produce untruncated BigInt from valid JSON numbers.

While more involved for the developer that Option B below, this creates maximum flexibility extending beyond decoding numbers.

Option B - Optional flags in JsonCodec::encode/decode

With this nonbreaking change, developers can opt to customize treatment of arbitrarily-sized integers by passing flags to (a) use BigInt for integers and (b) throw a runtime error when parsing of a number results in truncation. With this change, the signatures for encoding/decoding functions would look like

// signature of jsonEncode would not change but BigInt would be supported out-of-the-box by writing as a JSON number
String jsonEncode(Object? object, {Object? toEncodable(Object? nonEncodable)?})

// when [useBigInt] is true, all integers are parsed to BigInt
// when [throwOnNumericTruncation] is true, an error is thrown when an integer is too large to fit in an int
dynamic jsonDecode(String source, {Object? reviver(Object? key, Object? value)?, bool? useBigInt, bool? throwOnNumericTruncation})

This option is very specific and doesnt cover some important edge cases (such as high-precision floats) and generally adds work to the developer. It does, however, give the developer an option to do something about large integers coming over the wire.

Platform optimizations

I believe I understand from #45856 (and specifically this comment) that the above recommendations are moot on web since JavaScript's JSON parsing is used before the Dart parser gets a stab. However, given the importance of Dart (and particularly Flutter) for multi-platform development, it warrants considering how to create consistent behavior across all deployment environments.

[lrhn]

A platform independent ability to parse 64-bit integers (just signed or unsigned too?) requires parsing into a type that is cross-platform.

We don't have an Int64 class in the platform libraries. There is a package for that, and a JSON parser using that package could probably do something reasonable. (Although I'd prefer decoding to a different Int64 implementation which doesn't try to implement int operations. More like an Int64Value that can be converted to string, to int ... with all the inherent losses if on web, to Int64, or written into typed data, but doesn't have to be optimized for doing operations on it.)

So maybe that's the idea:

• Introduce Int64Value in the platform libraries, with no guarantees about the implementation (but it'll be an extension type on int on native, and something else on web, like two 32-bit integers). Only supported operation is ==.
• Introduce a JSON parser (perhaps a more efficient one than the one we have today) which can provide the values of int fields in a configurable way, with one option being Int64Value. (Generally have the parser provide interpretations of the source in whatever way the user wants, rather than build list/map structures of values.)
• Allow converting Int64Value to any number of types, including BigInt and Int64 from the package (support added by that package).
• Profit!

[austinmilt]

@lrhn that all sounds like a solid fix to me, and thanks for the thoughtful reply. The biggest bang for the buck IMO is

(Generally have the parser provide interpretations of the source in whatever way the user wants, rather than build list/map structures of values.)

The JSON spec already provides a relatively short list of possible element types, so being able to provide custom mapping from those types would be really powerful, e.g. (without considering efficiency)

abstract class JsonParser {
  static JsonParser defaultParser = BaseJsonParser();

  dynamic fromNumber(JsonNumber element);
  dynamic fromObject(JsonObject element);
  // ... etc for other JSON element types

  JsonElement toJson(dynamic value);

  dynamic fromJson(JsonElement element) {
    // builds an object from the mapping functions for each element type
  }

  JsonElement _parseJsonString(String value) {
    // parses the raw string into a JsonElement before
    throw UnimplementedError();
  }
}

class BaseJsonParser extends JsonParser {
  @override
  double fromNumber(JsonNumber element) {
    return double.parse(element.value);
  }

  @override
  Map<String, dynamic> fromObject(JsonObject element) {
    final Map<String, dynamic> result = {};
    for (MapEntry<JsonName, JsonElement> entry in element.value.entries) {
      // ... recursion
    }
    return result;
  }

  @override
  JsonElement toJson(dynamic value) {
    // switch case on type of value and return an appropriate conversion
    throw UnimplementedError();
  }
}

class JsonNumber implements JsonElement {
  JsonNumber(this.value);
  final String value;

  @override
  String encode() {
    return value.toString();
  }
}

class JsonObject implements JsonElement {
  JsonObject(this.value);
  final Map<JsonName, JsonElement> value;

  @override
  String encode() {
    // JSON object encoding with recursion
    throw UnimplementedError();
  }
}

class JsonName implements JsonElement {
  JsonName(this.value);
  final String value;

  @override
  String encode() {
    return '"$value"';
  }
}

abstract interface class JsonElement {
  String encode();
}

I'd be happy to contribute code to the work 🤝