search

dbt-labs / dbt-core

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
https://getdbt.com
Apache License 2.0
9.52k stars 1.58k forks source link

[Bug] Artifact schemas are inferred to be incorrect types #10442

Closed nakamichiworks closed 1 month ago

nakamichiworks commented 1 month ago

Is this a new bug in dbt-core?

Current Behavior

I want to use WritableManifest class to parse and analyze manifest.json of my dbt projects.

However, code editor (vscode) and associated type checker (pyright) incorrectly infer the type of WritableManifest to be VersionedSchema, which means that code completion does not work for the fields of the class.

# script.py
import json
from dbt.artifacts.schemas.manifest import WritableManifest

with open("manifest.json") as f:
    data = json.load(f)
    manifest = WritableManifest(**data)
    print(manifest.nodes)
    # vscode warning:
    # Cannot access attribute "nodes" for class "VersionedSchema"
    # Attribute "nodes" is unknownPylance[reportAttributeAccessIssue]
$ pyright script.py
/private/tmp/use_dbt_artifacts/script.py
  /private/tmp/use_dbt_artifacts/script.py:8:20 - error: Cannot access attribute "nodes" for class "VersionedSchema"
    Attribute "nodes" is unknown (reportAttributeAccessIssue)
1 error, 0 warnings, 0 informations

Expected Behavior

Instances of WritableManifest should be inferred to be WritableManifest type.

Steps To Reproduce

See the code snippet above.

Relevant log output

No response

Environment

- OS: macOS 14.5
- Python: 3.12.4
- dbt: 1.8.3

Which database adapter are you using with dbt?

Not relevant for this issue.

Additional Context

No response

dbeatty10 commented 1 month ago

Thanks for reaching out @nakamichiworks !

We use mypy as our type checker, and when I run your example script with it, it doesn't flag any issues:

$ mypy script.py

Success: no issues found in 1 source file

Could this be an issue with pyright rather than any issue with the way the type annotation is expressed?

nakamichiworks commented 1 month ago

Sorry I didn't notice the difference between mypy and pyright. I will look into the difference between the two type checkers in more detail.

That said, my proposal in https://github.com/dbt-labs/dbt-core/pull/10443 can satisfy both mypy and pyright. The proposed usage of TypeVar for class objects is even suggested in the official mypy docs. Any reason or concern not to accept my proposal?

dbeatty10 commented 1 month ago

Thanks for that info @nakamichiworks -- just let us know what you find out in relation to the difference between mypy and pyright.

We can have one of our engineers take a look at your proposed change in https://github.com/dbt-labs/dbt-core/pull/10443 and see what they think too.

MichelleArk commented 1 month ago

@nakamichiworks thank you for opening this and for for linking the mypy docs! Your proposed change looks good 👍