The purpose of JSON tokenizer is to process .json files and parse JSON objects to extract strings. As a result it allows to measure an entropy of each string.
There is a bug that instead of returning separate strings from a json object, would return a single string with concatenated values. Therefore, it didn't support nested objects too well.
What changed?
JSON tokenizer was updated to support nested objects with recursion
The hard limit for max depth in parsing nested objects is set to 10, but is configurable in the settings
It resolves a bug reported in #64
References
It fixes #64
Testing
[x] This change adds test coverage for new/changed/fixed functionality
Checklist
[x] All active GitHub checks for tests, formatting, and security are passing
[x] The correct base branch is being used, if not master
Description
The purpose of JSON tokenizer is to process
.jsonfiles and parse JSON objects to extract strings. As a result it allows to measure an entropy of each string.There is a bug that instead of returning separate strings from a json object, would return a single string with concatenated values. Therefore, it didn't support nested objects too well.
What changed?
References
Testing
Checklist
master