search

ianka / xjson.tcl

A Tcl library that validates, collects, composes JSON data against supplied schemas and to/from Tcl data.
Other
5 stars 0 forks source link
json validation

readme

NAME

xjson - extended JSON functions for Tcl

Table Of Contents

SYNOPSIS

package require Tcl 8.6-
package require itcl 4.0-
package require struct::set
package require struct::list
package require xjson ?1.10?

::xjson::decode json ?indexVar?
::xjson::encode decodedJson ?indent? ?tabulator? ?nest?
::xjson::recode decodedJson
::xjson::diff oldDecodedJson newDecodedJson
::xjson::patch decodedJson patch
::xjson::rpatch decodedJson patch
::xjson::makeCollectorClass ?options? collectorClassName ?methodName methodDefinition ...?
collectorClassName collectorObjName ?options? ?nestedCollectorName nestedCollectorObjName ...? schema
collectorObjName collect decodedJson ?path?
collectorObjName printSchema ?indent?
::xjson::makeComposerClass ?options? composerClassName ?methodName methodDefinition ...?
composerClassName composerObjName ?options? ?nestedComposerName nestedComposerObjName ...? schema
composerObjName compose tclData ?path?
composerObjName printSchema ?indent?

DESCRIPTION

This package is a set of extended JSON functions for Tcl. It allows decoding, encoding, and pretty-printing of JSON structures from Tcl structures and vice versa. In addition, decoded JSON that was created by functions outside of this package may be recoded. A set of diff and patch functions tailored to JSON allows to track changes in complicated JSON structures easily.

The main feature of this package however are two class factories that produce itcl classes that construct validator and data collector/composer objects. Those objects take a schema in a simple nested list syntax on construction, so they are then prepared for validating against that schema again and again. The schema may also feature various operators that manipulate the validated data so that further configuration specific processing (e.g. data formatting) can be specified in the realm of the administrator rather than the programmer.

Schemas may be nested and libraries of commonly used collector/composer objects and their schemas can be constructed easily.

The objects also feature automatic sandboxing, so they can specify Tcl code for doing more complicated tests or data manipulations. In addition, the programmer may define their own barebone, non-sandboxed methods when creating a new class with the class factory. For sandboxes and those methods, an additional security mechanism exists. Schemas may be marked as trusted (programmer/administrator supplied) and only such trusted schemas may use Tcl code or user-defined methods that are marked unsafe.

The procedures and objects return error messages that indicate the offending data and the non-matching schema in a human-readable, pretty-printed way. They are fit for supplying them to the interface user, which is usually a programmer or administrator of an another software interfacing the software using xjson.

BUGS

This manpage is intimidating. Please go right to the various EXAMPLES sections at the bottom for maintaining a slow pulse.

PROCEDURES

The package defines the following public procedures:

Read the following section COLLECTOR AND COMPOSER CLASS USAGE on how to use the classes made by those procedures.

COLLECTOR AND COMPOSER CLASS USAGE

The ::xjson::makeCollectorClass and ::xjson::makeComposerClass class factory procedures each make a class that is meant to be used to construct one or more collector respective composer objects using the following constructors:

Read the following section COLLECTOR OBJECT USAGE respective COMPOSER OBJECT USAGE on how to use the objects constructed by the class.

COLLECTOR OBJECT USAGE

The objects constructed by the collector class define the following methods:

COMPOSER OBJECT USAGE

The objects constructed by the composer class define the following methods:

COLLECTOR AND COMPOSER SCHEMAS

Collector and composer schemas are nested lists of collecting/composing functions and their arguments.

As an important detail, both collector and composer schemas describe the JSON side of things. Validation always happens on the JSON side of things. This gets crucial as soon your schemas feature data manipulations.

Schemas can be very simple. For example, a schema integer that is made from the collecting method of the same name validates JSON input data as -5 but not JSON input data as 3.2 (not an integer) or "5". Mind the quotes that mark that character as a string value. For the composing direction, the Tcl input data is validated instead.

You can make that schema an argument of another schema. For example, the array collecting method takes a schema as an argument. The resulting schema {array integer} validates JSON input data as [-4, 7, 8, 3], or for the composing direction Tcl input data {-4 7 8 3}.

Many collecting methods take options that allow to specify further constraints. For example, you can tell the array method to allow only up to three array members in the JSON input data. The schema {array -max 3 integer} validates [], [156], [1, 0], and [3, 99, 3], but not [-4, 7, 8, 3]. And of course the integer collecting method also allows further constraints. A schema {array -max 3 {integer -min 0 -max 99}} validates [], [1, 0], and [3, 99, 3], but not [-4, 7, 8, 3] and neither [156]. Likewise for the composing direction with Tcl lists of integers.

The collector objects do not just validate the decoded JSON input data but also return it as plain Tcl data. For manipulating what is returned, there are special collecting methods meant for output formatting. For example, a schema {array -max 3 {format "%02d" {integer -min 0 -max 99}}} returns the JSON input [3, 99, 3] as {03 99 03}.

It works the reverse for composing. The Tcl data is validated to be an array -max 3, the single elements formatted according to the format "%02d" method, then validated according to the integer -min 0 -max 99 method and finally returned as decoded JSON.

Symetrical schemas may be the same for collector and composer objects. As soon there is asymetrical data manipulation (as that format method) involved, you usually need separate schemas for both directions.

There are also methods that work as control structures. For example, the schema {anyof {{integer -max -10} {integer -min 10}}} validates any integer value that is either <=-10 or >=10. And of course, you can also mix the allowed JSON input types that way if it makes sense in your application: e.g. {array {anyof {number string boolean null}}} validates an array of all kinds of valid simple JSON types. Note that you will lose the JSON type information that way so you usually don't want to do such stunts without further type-specific output formatting.

See the section BUILTIN METHODS and the various EXAMPLES sections for more details.

BUILTIN METHODS

The following methods are built into each collector/composer class (and object) unless the class was created specifying the -nobuiltins option when calling the ::xjson::makeCollectorClass or ::xjson::makeComposerClass class factory procedure. The methodName as used by the class factory procedure is the same as the method name below as it is used inside the schema, unless otherwise noted.

CUSTOM METHODS

If neither the above collecting/composing methods nor sandboxed Tcl code from within the schema are sufficient to solve the particular validation and collecting/composing problem, you may want to create custom collecting methods. This can be done with relative ease. They have to be specified on the call to the ::xjson::makeCollectorClass or ::xjson::makeComposerClass class factory procedure with a unique methodName and a methodDefinition.

See the files "builtinCollectingMethods.tcl" and "builtinComposingMethods.tcl" from the library installation directory (often "/usr/share/tcl/xjson1.10/") for examples on how to write your own custom methods.

NESTING

With each collector/composer object you construct from the classes produced by ::xjson::makeCollectorClass or ::xjson::makeComposerClass, you may specify other collector/composer objects that should be accessible from within the registered schema by a nestedCollectorName/nestedComposerName alias. The rationale of this is creating libraries of different collector/composer objects for often used JSON aggregates in your application, and calling them from an uplevel or the toplevel schema.

The nest method makes use of this function. It takes an alias name and calls the collect/compose method of the nested object with the decoded JSON input data at that point, and the path. The nested object takes care of the input data, validates it with its own schema, and returns the result to the calling object.

The specified nested objects do not have to exist when the calling object is constructed. It is also not checked which class the nested object has. You may specify any object that has a collect/compose method with the same semantics as those produced by ::xjson::makeCollectorClass resp. ::xjson::makeComposerClass.

The dubious/trusted flag is local to each object. This may be used to create collector/composer objects with application provided schemas and elevated rights that a object with user-provided schem and restricted rights may call.

SANDBOXING

User supplied data is never evaluated as code by any builtin method. All the considerations below are about configuration-supplied rather than programmer-supplied schemas.

On construction of the collector/composer object, you may specify the -trusted option to enable Tcl code evaluation from the schema. If not specified, using those methods and options in the supplied schema will throw an error instead and the object won't be constructed at all.

Schemas may specify collecting/composing methods (e.g. apply, expr, lmap) or options (e.g. -test, -transform) that rely on Tcl code supplied from within the schema. To use such schemas in a safe fashion, all that Tcl code is executed in a safe interpreter (a sandbox) as supplied by Tcl's interp -safe command.

Sandbox creation and destruction after use happens automatically whenever data is collected/composed. That sandbox is shared by all methods in the schema and may also be used to pass values in global variables between methods. As a shortcut, all of the methods that have arguments or options allowing to specify Tcl code also have an option -isolate, that creates a local sandbox just for that method automatically.

NULL HANDLING

The procedures ::xjson::encode, ::xjson::recode, and ::xjson::decode treat JSON null values literally. As with the JSON boolean values true and false that are coded as literal true resp. literal false, JSON null values are decoded as literal null by ::xjson::decode and the same literal needs to be specified to ::xjson::encode to produce a JSON null value. ::xjson::decode returns an empty list on empty JSON input, and ::xjson::encode and ::xjson::recode throw an error on an attempt to encode an empty list.

In contrast, the collector/composer objects constructed from the classes produced by ::xjson::makeCollectorClass and ::xjson::makeComposerClass treat JSON null values symbolically.

A literal null a collector class finds in its decoded JSON input is treated as if the data field it fills isn't there. JSON nulls in arrays are simply skipped, and they also don't count for the array length. JSON nulls as the value of an object field are treated the same as if that field wasn't specified in that object. Schemas may specify a null collecting method that validates a literal null. It is however treated as such and will trigger the above null handling in the uplevel schema. An empty list in the decoded JSON input data is treated the same as a literal null. A missing data field is also treated like a literal null.

In composer classes, the methods that emit JSON types have a special option -null that allows the schema author to tell which Tcl input value should be treated as null, if any. If not specified, there will never be a null value emitted at that place. The optional method has a special option -emitnull that allows the schema author to specify if downlevel nulls should be inserted into the emitted JSON object or array literally instead of simply leaving out that field.

To change that symbolic treatment of JSON nulls at specific places, you can use the default collecting method and tell a default value that should be used whenever a null is encountered in the JSON input at that place.

DATA FORMATS

DECODED JSON FORMAT

The decoded JSON format as returned by the ::xjson::decode and accepted by the ::xjson::encode and ::xjson::recode commands is a nested list of type-data pairs.

JSON PATCH FORMAT

The JSON patch format as returned by the ::xjson::diff and accepted by the ::xjson::patch and ::xjson::rpatch commands is a nested list of diff operations.

EXAMPLES

DECODING EXAMPLES

Decode an array of array of numbers.

% ::xjson::decode {[[1,2],[3,4]]}
array {{array {{number 1} {number 2}}} {array {{number 3} {number 4}}}}

Decode an object of various types.

% ::xjson::decode {{"foo":"hello","bar":42,"quux":null}}
object {foo {string hello} bar {number 42} quux {literal null}}

Same with arbitrary whitespace.

% ::xjson::decode {
{
    "foo":  "hello",
    "bar":  42,
    "quux": null
}
}
object {foo {string hello} bar {number 42} quux {literal null}}

ENCODING EXAMPLES

Encode an array of array of numbers.

% ::xjson::encode {array {{array {{number 1} {number 2}}} {array {{number 3} {number 4}}}}} 0 {}
[[1,2],[3,4]]

Encode an object of various types.

% ::xjson::encode {object {foo {string hello} bar {number 42} quux {literal null}}} 0 {}
{"foo":"hello","bar":42,"quux":null}

Same with pretty printing.

% ::xjson::encode {object {foo {string hello} bar {number 42} quux {literal null}}}
{
    "foo":  "hello",
    "bar":  42,
    "quux": null
}

Encode with pre-encoded data.

% set json {"hello"}
% ::xjson::encode [list object [list foo [list encoded $json] bar {number 42} quux {literal null}]] 0 {}
{"foo":"hello","bar":42,"quux":null}

Encode with nested decoded data.

% set type decoded
% set data {string hello}
% ::xjson::encode [list object [list foo [list $type $data] bar {number +42} quux {literal null}]] 0 {}
{"foo":"hello","bar":42,"quux":null}

RECODING EXAMPLES

Recode pre-encoded data.

% set json {"oof rab"}
% ::xjson::recode [list object [list foo [list encoded $json] bar {number +42} quux {literal null}]]
object {foo {string {oof rab}} bar {number 42} quux {literal null}}

Recode with nested decoded data.

% set type decoded
% set data {string "oof rab"}
% ::xjson::recode [list object [list foo [list $type $data] bar {number +42} quux {literal null}]]
object {foo {string {oof rab}} bar {number 42} quux {literal null}}

DIFF EXAMPLES

Feed two sets of slightly different JSON data into the decoder and remember the result.

% set old [::xjson::decode {
    {
        "articles": [
            {
                "id":    101,
                "name":  "Pizzapane bianca",
                "price": 4.95
            },
            {
                "id":    120,
                "name":  "Pizza Regina",
                "price": 9.8
            },
            {
                "id":    139,
                "name":  "Wunschpizza",
                "price": 12.70
            },
            {
                "id":    201,
                "name":  "Rucola",
                "extra": true,
                "price": 1
            }
        ]
    }
}]

% set new [::xjson::decode {
    {
        "articles": [
            {
                "id":    101,
                "name":  "Pizzapane bianca",
                "extra": true,
                "price": 4.95
            },
            {
                "id":    120,
                "name":  "Pizza Regina",
                "price": 9.80
            },
            {
                "id":    138,
                "name":  "Pizza Hawaii",
                "price": 12.00
            },
            {
                "id":    139,
                "name":  "Wunschpizza",
                "price": 13.50
            },
            {
                "id":    201,
                "name":  "Rucola",
                "price": 1
            }
        ]
    }
}]

Calculate the patch data.

% ::xjson::diff $old $new
keys {articles {indices {0 {keys {extra {add {literal true}}}} 2 {remove {{object {id {number 139} name {string Wunschpizza} price {number 12.7}}} {object {id {number 201} name {string Rucola} extra {literal true} price {number 1}}}}} 2 {insert {{object {id {number 138} name {string {Pizza Hawaii}} price {number 12.0}}} {object {id {number 139} name {string Wunschpizza} price {number 13.5}}} {object {id {number 201} name {string Rucola} price {number 1}}}}}}}}

PATCH EXAMPLES

Feed JSON data into the decoder and remember the result.

% set old [::xjson::decode {
    {
        "articles": [
            {
                "id":    101,
                "name":  "Pizzapane bianca",
                "price": 4.95
            },
            {
                "id":    120,
                "name":  "Pizza Regina",
                "price": 9.8
            },
            {
                "id":    139,
                "name":  "Wunschpizza",
                "price": 12.70
            },
            {
                "id":    201,
                "name":  "Rucola",
                "extra": true,
                "price": 1
            }
        ]
    }
}]

Dig out matching patch data.

% set patch {keys {articles {indices {0 {keys {extra {add {literal true}}}} 2 {remove {{object {id {number 139} name {string Wunschpizza} price {number 12.7}}} {object {id {number 201} name {string Rucola} extra {literal true} price {number 1}}}}} 2 {insert {{object {id {number 138} name {string {Pizza Hawaii}} price {number 12.0}}} {object {id {number 139} name {string Wunschpizza} price {number 13.5}}} {object {id {number 201} name {string Rucola} price {number 1}}}}}}}}}

Apply the patch and encode it.

% ::xjson::encode [::xjson::patch $old $patch]
{
    "articles": [
        {
            "id":    101,
            "name":  "Pizzapane bianca",
            "price": 4.95
                    "extra": true
        },
        {
            "id":    120,
            "name":  "Pizza Regina",
            "price": 9.8
        },
        {
            "id":    138,
            "name":  "Pizza Hawaii",
            "price": 12.0
        },
        {
            "id":    139,
            "name":  "Wunschpizza",
            "price": 13.5
        },
        {
            "id":    201,
            "name":  "Rucola",
            "price": 1
        }
    ]
}

JSON VALIDATION AND DATA COLLECTING EXAMPLE

Create a collector class with the class factory and the builtin methods. Even for advanced usage, you only ever need to do this once. More than once only if you want to create multiple collector classes with a different set of builtin and your own collection methods, or different options.

% ::xjson::makeCollectorClass ::myCollectorClass

Create a collector object with a schema. You need to do this once per different schema you want to use.

% set ::mycollector [::myCollectorClass #auto {
    object {
        "articles" {
            dictby "id" {array -max 100 {object {
                "id"    integer
                "extra" {default false boolean}
                "name"  string
                "price" {format "%.2f" number}
            }}}
        }
    }
}]

Feed JSON data into the decoder, and the decoded json data into the collector object.

% $::mycollector collect [::xjson::decode {
    {
        "articles": [
            {
                "id":    101,
                "name":  "Pizzapane bianca",
                "price": 4.95
            },
            {
                "id":    120,
                "name":  "Pizza Regina",
                "price": 9.8
            },
            {
                "id":    139,
                "name":  "Wunschpizza",
                "price": 12.70
            },
            {
                "id":    201,
                "name":  "Rucola",
                "extra": true,
                "price": 1
            }
        ]
    }
}]
articles {101 {extra false name {Pizzapane bianca} price 4.95} 120 {extra false name {Pizza Regina} price 9.80} 139 {extra false name Wunschpizza price 12.70} 201 {extra true name Rucola price 1.00}}

JSON COMPOSING EXAMPLE

Create a composer class with the class factory and the builtin methods. Even for advanced usage, you only ever need to do this once. More than once only if you want to create multiple composer classes with a different set of builtin and your own composer methods, or different options.

% ::xjson::makeComposerClass ::myComposerClass

Create a composer object with a schema. You need to do this once per different schema you want to use.

% set ::mycomposer [::myComposerClass #auto {
    array {
        object {
            id   integer
            name string
        }
    }
}]

Feed Tcl data into the composer object, and the result into the encoder.

% ::xjson::encode [$::mycomposer compose {{id 7 name foo} {id 3 name bar}}]
[
    {
        "id":   7,
        "name": "foo"
    },
    {
        "id":   3,
        "name": "bar"
    }
]

JSON RECOMPOSING EXAMPLE

This toy example decodes and validates a JSON input, collects the data, then recomposes the data into JSON.

% ::xjson::makeCollectorClass ::myCollectorClass
% ::xjson::makeComposerClass ::myComposerClass

% set ::mycollector [::myCollectorClass #auto {
dictbyindex 1 {array {object -values {id integer name {stringop -toupper {0 end} string} comment string}}}
}]

% set ::mycomposer [::myComposerClass #auto {
dictbyindex 1 {array {object -values {id integer name {stringop -tolower {0 end} string} comment string}}}
}]

% set step0 {[{"id": 1, "name": "abc", "comment": "foo"}, {"id": 981, "name": "xyz", "comment": "bar"}]}
[{"id": 1, "name": "abc", "comment": "foo"}, {"id": 981, "name": "xyz", "comment": "bar"}]

% set step1 [::xjson::decode $step0]
array {{object {id {number 1} name {string abc} comment {string foo}}} {object {id {number 981} name {string xyz} comment {string bar}}}}

% set step2 [$mycollector collect $step1]
ABC {1 foo} XYZ {981 bar}

% set step3 [$mycomposer compose $step2]
array {{object {id {number 1} name {string abc} comment {string foo}}} {object {id {number 981} name {string xyz} comment {string bar}}}}

% set step4 [::xjson::encode $step3 0 {}]
[{"id":1,"name":"abc","comment":"foo"},{"id":981,"name":"xyz","comment":"bar"}]

KEYWORDS

diff, json, patch, tcl, validation

COPYRIGHT

Copyright © 2021 Jan Kandziora jjj@gmx\.de, BSD-2-Clause license