Open lultimouomo opened 2 years ago
Hi @lultimouomo,
Please apologize for the long delay; I wanted to take the time to give you a complete answer.
I find that I can need more memory for the JsonDocument than for the serialized Json."
This is perfectly normal! With the same logic, you could contact the Chromium team and say, "the DOM is signicantly larger than the HTML on my disk".
Remember that ArduinoJson is a serialization library, not a generic container library.
JsonDocument
is supposed to be a short-lived, throw-away object, only used during serialization and deserialization.
If you need to keep the JSON document in memory for a long period, extract the information in another data structure, as shown in JsonConfigFile.ino, or keep the serialized version in a String
.
The main reason for not using JsonDocument
as a container class is the monotonic allocator.
I plan on removing this limitation, but the experience with v6.6 showed me that the monotonic allocator is a well-suited solution, especially for 8-bit microcontrollers.
In v7, I'll stop focusing on 8-bit MCUs and change the allocation strategy entirely.
pretty sizeable amount of memory
This statement is exaggerated: 16-bytes on a device with 320 KB of RAM (and often 4MB of PSRAM) is a reasonable number. As a matter of fact, rapidjson's objects have the same size, and I'm sure they're well optimized.
From my experience, many users define ARDUINOJSON_USE_DOUBLE and
ARDUINOJSON_USE_LONG_LONG to 1, so I wanted to make it the default in the next revision.
I'm also considering removing these flags for v7.
In this context, trying to reduce the size of VariantSlot
seems to go in the wrong direction, but I'm opened to the discussion.
make support for serialized() optional
Why not? After all, very few users are interested in this feature, so we could make it optional.
Unfortunately, this change alone will not reduce the size of VariantSlot
; as you said, we also need to remove _tail
.
What bothers me, though, is that I intended to use the size
field for strings as well; this would allow NUL in strings, providing a satisfying solution to #1646, and build the foundation for #922.
Also, adding feature flags complexifies testing and makes you embed multiple copies of the library when you link several compilation units with different flags.
Removing
_tail
The early versions of ArduinoJson had only a _head
pointer, but the performance was significantly reduced; indeed, when you populate a JsonDocument
, you frequently append to arrays and objects.
Still, we could reduce the size of _head
and _tail
by using VariantSlotDiff
as for VariantSlot::_next
.
The problem is that this change will slightly increase the code size and will only benefit the users that configured ARDUINOJSON_USE_DOUBLE
, ARDUINOJSON_USE_LONG_LONG
, and the hypothetical ARDUINOJSON_ENABLE_SERIALIZED
to 0 at the same time.
I would need to do some experiments to see how many bytes of code this change would add.
add support for short strings
I intended to add SSO in v7, but I can include it in the v6 branch.
I think it will work great if the VariantSlot
stays at 16 bytes because there are many words with seven characters or less.
However, if we reduce VariantSlot
and only store a three-character string, SSO will occur very rarely, so it won't be worth it. Indeed, short words, like "id," "url," "min," and "max" tend to be used in keys and not in values (which indicated that an SSO for the key, would be a good idea, even if limited to 3 characters).
I was wondering if you would consider PRs
Thanks for offering your help. To be honest, I had mixed experiences with PRs in this repository: most contributors write a quick-and-dirty fix without any consideration for coverage and code size. You can submit a PR, but I'll probably not merge it as is. I'll, however, use it as a source of inspiration and credit you in the changelog.
In summary, I don't think reducing the size of VariantSlot
is relevant because:
JsonDocument
will always be bigger than the serialized string, and 16-bytes per object is already quite good.JsonDocument
out of the serialization process, thus stretching the library's arm.asRaw
goes against #1646 and #922.VariantSlot
renders small string optimization useless.Apart from that, I'm all in favor of SSO; it's been on my (secret) backlog for a long time, and I'll probably implement it in v6.
PS: I deleted your messages in #1343 because they were irrelevant to the conversation; please open a new issue so we can work on your problem without polluting another thread. Thank you for your understanding.
Best regards, Benoit
1. Your expectation is unrealistic: the `JsonDocument` will always be bigger than the serialized string, and 16-bytes per object is already quite good.
That's very large for a uint32_t
or smaller value. Being a text-based format I'd expect the best case scenario is that the JsonDocument
requires less memory than the serialised string.
An alternative to #1074 (which is still a problem when the estimate JSON_ARRAY_SIZE(50) + 50 * (JSON_ARRAY_SIZE(2) + JSON_ARRAY_SIZE(3))
for serialising a list of 50 sets of 3 uint8_t
values is nearly 4KB) for serialisation would be to iteratively output to a file/stream/buffer. It would place restrictions on the order of function calls because earlier objects in an array/map would no longer be modifiable (having already been written), but it would make it possible to free those objects.
The main reason for not using JsonDocument as a container class is the monotonic allocator. I plan on removing this limitation, but the experience with v6.6 showed me that the monotonic allocator is a well-suited solution, especially for 8-bit microcontrollers. In v7, I'll stop focusing on 8-bit MCUs and change the allocation strategy entirely.
Does this mean using it as container class becomes a valid use case in v7?
Yes, you can use long-lived JsonDocument
in ArduinoJson 7.
Now, just because you can doesn't mean you should.
Weigh the pros and cons before committing to a solution.
Migrated to v7 today. Looks very good! I had a memory usage / capacity watch in my ui but removed that and heap usage looks better than before !
Hi Benoit,
I noticed that ArduinoJson needs a pretty sizeable amount of memory for each object (16 bytes on ESP32); even by using zero-copy I find that I can need more memory for the
JsonDocument
than for the serialized Json. I see that a lot of work has been put into saving memory, but I was wondering if you would consider PRs for the following changes:make support for
serialized()
optionalBy removing support for
serialized()
via a preprocessor definition we could then removeasRaw
fromVariantContent
and also remove_tail
fromCollectionData
and reduce the size ofVariantContent
to the size of a pointer/float. Removing_tail
would mean that to add a collection element we need to walk the linked list each time, but in certain situations it could be a reasonable tradeoff.add support for short strings
In case of very short strings we could apply short-string optimization and store up to 3 characters for 32bit archs and 7 characters for 64bit archs (+1 for null termination) in place of the 'char *'. This is probably a marginal improvement, especially because 64bit archs where it can be applied more easily don't probably need it as much. Also it won't matter for zero-copy.