dynatrace-oss / dt-cli

Command line tool for Dynatrace
https://pypi.org/project/dt-cli/
Apache License 2.0
17 stars 12 forks source link

extension validate-schema is incompatible with schemas > 1.290.0 #178

Closed osd42 closed 2 days ago

osd42 commented 4 days ago

extension validate-schema with schemas >= 1.291.0 From schema 1.291.0, dt-cli won't validate an extension. I am using dt-cli 1.6.18 (from the latest dt-extensions-sdk[cli]) (1.6.18)

To Reproduce Steps to reproduce the behavior:

  1. Import schemas 1.290.0 and 1.291.0 and later
  2. Try this CLI on a valid extension: dt extension validate-schema --instance extension/extension.yaml --schema-entrypoint ~/.vscode-server/data/User/globalStorage/dynatraceplatformextensions.dynatrace-extensions/1.290.0/extension.schema.json
  3. Try all later versions like 1.291.0 dt extension validate-schema --instance extension/extension.yaml --schema-entrypoint ~/.vscode-server/data/User/globalStorage/dynatraceplatformextensions.dynatrace-extensions/1.291.0/extension.schema.json
  4. See error
(.venv) $ dt extension validate-schema --instance extension/extension.yaml --schema-entrypoint /home/charly/.vscode-server/data/User/globalStorage/dynatraceplatformextensions.dynatrace-extensions/1.295.0/extension.schema.json

Traceback (most recent call last):
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/bin/dt", line 8, in <module>
    sys.exit(main())
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/dtcli/__main__.py", line 5, in main
    dtcli.scripts.dt.main()
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/dtcli/scripts/dt.py", line 840, in validate_schema
    errors = _validate_schema.validate_schema(
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/dtcli/validate_schema.py", line 80, in validate_schema
    return list(map(process_validation_error, validator.iter_errors(instance)))
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/jsonschema/validators.py", line 384, in iter_errors
    for error in errors:
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/jsonschema/_keywords.py", line 296, in properties
    yield from validator.descend(
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/jsonschema/validators.py", line 432, in descend
    for error in errors:
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/jsonschema/_legacy_keywords.py", line 135, in items_draft6_draft7_draft201909
    yield from validator.descend(item, items, path=index)
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/jsonschema/validators.py", line 432, in descend
    for error in errors:
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/jsonschema/_keywords.py", line 355, in oneOf
    errs = list(validator.descend(instance, subschema, schema_path=index))
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/jsonschema/validators.py", line 432, in descend
    for error in errors:
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/jsonschema/_keywords.py", line 275, in ref
    yield from validator._validate_reference(ref=ref, instance=instance)
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/jsonschema/validators.py", line 482, in _validate_reference
    return list(self.descend(instance, resolved))
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/jsonschema/validators.py", line 432, in descend
    for error in errors:
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/jsonschema/_keywords.py", line 296, in properties
    yield from validator.descend(
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/jsonschema/validators.py", line 432, in descend
    for error in errors:
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/jsonschema/_keywords.py", line 275, in ref
    yield from validator._validate_reference(ref=ref, instance=instance)
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/jsonschema/validators.py", line 482, in _validate_reference
    return list(self.descend(instance, resolved))
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/jsonschema/validators.py", line 432, in descend
    for error in errors:
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/jsonschema/_keywords.py", line 296, in properties
    yield from validator.descend(
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/jsonschema/validators.py", line 432, in descend
    for error in errors:
  File "/home/charly/workspace/snmp-asa-cisco-device/.venv/lib/python3.10/site-packages/jsonschema/_keywords.py", line 218, in pattern
    and not re.search(patrn, instance)
  File "/home/charly/.pyenv/versions/3.10.14/lib/python3.10/re.py", line 200, in search
    return _compile(pattern, flags).search(string)
  File "/home/charly/.pyenv/versions/3.10.14/lib/python3.10/re.py", line 303, in _compile
    p = sre_compile.compile(pattern, flags)
  File "/home/charly/.pyenv/versions/3.10.14/lib/python3.10/sre_compile.py", line 788, in compile
    p = sre_parse.parse(p, flags)
  File "/home/charly/.pyenv/versions/3.10.14/lib/python3.10/sre_parse.py", line 955, in parse
    p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
  File "/home/charly/.pyenv/versions/3.10.14/lib/python3.10/sre_parse.py", line 444, in _parse_sub
    itemsappend(_parse(source, state, verbose, nested + 1,
  File "/home/charly/.pyenv/versions/3.10.14/lib/python3.10/sre_parse.py", line 841, in _parse
    p = _parse_sub(source, state, sub_verbose, nested + 1)
  File "/home/charly/.pyenv/versions/3.10.14/lib/python3.10/sre_parse.py", line 444, in _parse_sub
    itemsappend(_parse(source, state, verbose, nested + 1,
  File "/home/charly/.pyenv/versions/3.10.14/lib/python3.10/sre_parse.py", line 841, in _parse
    p = _parse_sub(source, state, sub_verbose, nested + 1)
  File "/home/charly/.pyenv/versions/3.10.14/lib/python3.10/sre_parse.py", line 444, in _parse_sub
    itemsappend(_parse(source, state, verbose, nested + 1,
  File "/home/charly/.pyenv/versions/3.10.14/lib/python3.10/sre_parse.py", line 555, in _parse
    code1 = _class_escape(source, this)
  File "/home/charly/.pyenv/versions/3.10.14/lib/python3.10/sre_parse.py", line 350, in _class_escape
    raise source.error('bad escape %s' % escape, len(escape))
re.error: bad escape \p at position 5

Expected behavior The new schemas (>1.290.0) should be compatible with dt-cli extension validate-schema feature.

Desktop (please complete the following information):

Additional context I am using the validate-schema feature on a CI pipeline because it catches errors that vscode does not. With this bug, we are stuck to extensions using schema 1.290.0 max. Thank your for the dt-cli tool !

vduseev commented 2 days ago

Thank you for creating the issue @osd42! I appreciate your time and effort!

This is reproducible. I tested it with every schema and the problem with introduced in 1.291, when a new regex, that restricts symbols in metric's displayName and description was added.

Here is this regex:

^(([^\\p{Cn}\\p{Cc}\\p{Cf}\\p{Co}\\p{Cs}\\p{Zl}\\p{Zp}]*)|([\\uD83E\\uDD00-\\uD83E\\uDDFF]*))+$

The thing is, there is only one flavor of regex engine, that supports that expression: Java 8+. All other engines, such as Python's re (used by dt-cli), Golang, PERL, JavaScript, do not support this expression and would fail trying to compile it.

The expression was added to schema 1.291 in April 2024. There is no way to go back and remove it. And there is no regex library for Python that supports this kind of expression. So, the only way to solve that is to ignore unsupported regex expressions in dt-cli during schema validation, as if they didn't exist, just like in 1.290.

I'm opening a PR that will do exactly that.

vduseev commented 2 days ago

Release v1.6.21 addressing the issue has been published to PyPi.