Open sergei-maertens opened 1 year ago
Additional thoughts: At the moment we have:
Before my PR, we were using python data for the json logic. If the PR is merged we would have:
Additional example: dealing with currency / number component
€ 123,00
.blocked until we get the go-ahead for this
Another issue related to types: https://github.com/open-formulieren/open-forms/issues/2707
The work on this cannot block releasing intermediate versions. So, working on this needs to be done outside of master OR as feature flag.
Chris mentioned that we can essentially do some type inference and "rewrite"/compile/transpile JSON logic expressions into equivalents that can be evaluated on both backend and frontend.
E.g. a string datetime + relative delta -> convert to unix timestamp (number) + delta (number) & the result can then be compared in terms of primitives.
To apply logic:
data (1) -> convert (2) -> preproces (3) -> jsonlogic (4) -> convert (5) -> result (6)
To load data from the database / load data from the user submission:
data (1) -> convert (2) -> result (6)
Example:
"2023-03-15"
, 2) Date(2023, 3, 15)
, 3) 12345
(Unix timestamp) 4) Apply rule, like add timedelta, 5) Date(2023, 3, 16)
6) {{ date|format }}
Refinement: We should create concrete things todo (=issues) with this ticket as the main epic.
See also #2251 See also #2305 (using local type information instead of requiring formio definitions would be very useful)
In short - we need to ensure that all (input) data has the correct type for processing in python and when calculations are done with json-logic, we need to serialize back to json-types and all data needs to be properly normalized.
We have essentially the following flow of information:
The 'problem' with JSON is that it only has a number of primitives that are used for richer Python types:
This is further complicated with the formio component types and the notion of single/multiple values (array vs. primitive).
Using the python datatypes
Simply just using JSON types (complex & primitives) is not sufficient because we cannot do smart operations on them. We must support (non-exhaustive list) the following operations:
|date
,|time
,|floatformat
)timestamp + timedelta
glom
orjq
to read/write certain data)Identified boundaries
We can identify "our own code" as the system boundary. This implies:
FormVariable
type information (seeSubmissionValueVariable.to_python
)note: 'as JSON' implies the result of
json.loads(...)
here, so we have python dicts/lists/strings/ints/floats/NoneType.handling JSON logic
The JSON logic library essentially operates on JSON primitives (or complex objects) and we should deal with that. This is particularly challening when comparing datetimes (or dates and datetimes) for example:
2022-11-08T14:12:00+01:00
is equal to2022-11-08T13:12:00+00:00
and2022-11-08T13:12:00Z
- but simple string operations will not give the same result.We need to normalize JSON logic expressions with the available type information at save time so that runtime is as simple as possible:
settings.TIMEZONE
)A conclusion may be that we need to pass python-objects (datetimes) down to json logic rather than just serialized versions.
Tasks taken from refinement