atviriduomenys / spinta

Spinta is a framework to describe, extract and publish data (a DEP Framework).
MIT License
10 stars 4 forks source link

Add swap user function #508

Closed sirex closed 8 months ago

sirex commented 9 months ago

Implement swap(old, new) function, as documented here:

https://atviriduomenys.readthedocs.io/dsa/formules.html#func.swap

Example manifest table:

model property type ref source prepare
City
name string
Vėlnius swap("Vilnius")
Vilna swap("Vilnius")

Here, prepare statement is given in so called append mode:

https://github.com/atviriduomenys/spinta/blob/1fac5b6b75ec65188d815078fd135bc05d49b31c/spinta/manifests/tabular/helpers.py#L631

Appendable property.prepare expression above is identical to:

model property type ref source prepare
City
name string swap("Vėlnius", "Vilnius").swap("Vilna", "Vilnius")

Pilnas swap() funkcijos argumentų sąrašas būtų toks:

swap(self, old, new)

Kur self yra kontekstinė reikšmė, kuri ateina iš duomenų šaltinio.

See: https://atviriduomenys.readthedocs.io/dsa/formules.html#alternatyvus-funkcijos-iskvietimas

swap() function should be implemented using SqlResultBuilder:

https://github.com/atviriduomenys/spinta/blob/1fac5b6b75ec65188d815078fd135bc05d49b31c/spinta/datasets/backends/sql/commands/read.py#L37-L38

That means, this function should be run on results. Probably it is possible to achieve similar functionality with SQL query, but for now, it is better to do a universal python based implementation, that would work on all backends. Later, if needed, an optimized SQL based solution, could be added.

cast() function could be used as an example:

https://github.com/atviriduomenys/spinta/blob/1fac5b6b75ec65188d815078fd135bc05d49b31c/spinta/datasets/backends/sql/commands/query.py#L978-L993

sirex commented 8 months ago

Found few bugs:

d | r | b | m | property | type     | ref | source | prepare
example                  |          |     |        |
  |   |   | Data         |          |     |        |
  |   |   |   | name     | string   |     | name   |
  |   |   |   |          |          |     | "A"    | swap("B")

This gives:

Traceback (most recent call last):
  File "lark/lexer.py", line 478, in lex
    yield lexer.next_token(lexer_state, parser_state)
  File "lark/lexer.py", line 398, in next_token
    raise UnexpectedCharacters(lex_state.text, line_ctr.char_pos, line_ctr.line, line_ctr.column,
lark.exceptions.UnexpectedCharacters: No terminal matches 'A' in the current parser context, at line 1 col 8

swap(""A"", "B")
       ^
Expected one of: 
        * DOT
        * RSQB
        * RPAR
        * COMP
        * LSQB
        * VBAR
        * FACTOR
        * COMMA
        * TERM
        * AMPERSAND

Previous tokens: Token('STRING', '""')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "spinta/manifests/tabular/helpers.py", line 159, in _parse_spyna
    return spyna.parse(formula)
  File "spinta/spyna.py", line 86, in parse
    ast = _parser.parse(rql)
lark.exceptions.UnexpectedToken: Unexpected token Token('NAME', 'A') at line 1, column 8.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "typer/main.py", line 683, in wrapper
    return callback(**use_params)  # type: ignore
  File "spinta/cli/show.py", line 24, in show
    store = prepare_manifest(context, verbose=False)
  File "spinta/cli/helpers/store.py", line 132, in prepare_manifest
    store = load_manifest(
  File "spinta/cli/helpers/store.py", line 116, in load_manifest
    commands.load(
  File "multipledispatch/dispatcher.py", line 278, in __call__
    return func(*args, **kwargs)
  File "spinta/manifests/yaml/commands/load.py", line 108, in load
    commands.load(
  File "spinta/manifests/tabular/commands/load.py", line 58, in load
    load_manifest_nodes(context, into, schemas, source=manifest)
  File "spinta/manifests/helpers.py", line 134, in load_manifest_nodes
    for eid, schema in schemas:
  File "spinta/manifests/tabular/helpers.py", line 1234, in read_tabular_manifest
    yield from _read_tabular_manifest_rows(
  File "spinta/manifests/tabular/helpers.py", line 1192, in _read_tabular_manifest_rows
    yield from state.release()
  File "spinta/manifests/tabular/helpers.py", line 1153, in release
    parent.leave()
  File "spinta/manifests/tabular/helpers.py", line 697, in leave
    self._parse_prepare()
  File "spinta/manifests/tabular/helpers.py", line 705, in _parse_prepare
    self.data["prepare"] = _parse_spyna(self, self.data["prepare"])
  File "spinta/manifests/tabular/helpers.py", line 161, in _parse_spyna
    reader.error(f"Error while parsing formula {formula!r}:\n{e}")
  File "spinta/manifests/tabular/helpers.py", line 216, in error
    raise TabularManifestError(f"{self.path}:{self.line}: {message}")
spinta.manifests.tabular.helpers.TabularManifestError: /tmp/manifest.txt:4: Error while parsing formula 'swap(""A"", "B")':