great-expectations / great_expectations

Always know what to expect from your data.
https://docs.greatexpectations.io/
Apache License 2.0
10k stars 1.54k forks source link

INFO: _get_default_value called with key "table", but it is not a known field in logs #10553

Open VolkovGeoPhy opened 3 weeks ago

VolkovGeoPhy commented 3 weeks ago

Describe the bug For instance I use this type of expectations. (or many other types) class ExpectColumnValuesToMatchRegex(ColumnMapExpectation): `class ColumnMapExpectation(BatchExpectation, ABC): """Base class for ColumnMapExpectations.

ColumnMapExpectations are evaluated for a column and ask a yes/no question about every row in the column.
Based on the result, they then calculate the percentage of rows that gave a positive answer.
If the percentage is high enough, the Expectation considers that data valid.

ColumnMapExpectations must implement a `_validate(...)` method containing logic
for determining whether the Expectation is successfully validated.

ColumnMapExpectations may optionally provide implementations of `validate_configuration`,
which should raise an error if the configuration will not be usable for the Expectation. By default,
the `validate_configuration` method will return an error if `column` is missing from the configuration.

Raises:
    InvalidExpectationConfigurationError: If `column` is missing from configuration.
Args:
    domain_keys (tuple): A tuple of the keys used to determine the domain of the
        expectation.
    success_keys (tuple): A tuple of the keys used to determine the success of
        the expectation.
"""  # noqa: E501

column: StrictStr = Field(min_length=1, description=COLUMN_DESCRIPTION)
mostly: Mostly = 1.0  # type: ignore[assignment] # TODO: Fix in CORE-412

catch_exceptions: bool = True

map_metric: ClassVar[Optional[str]] = None
domain_keys: ClassVar[Tuple[str, ...]] = (
    "batch_id",
    "table",
    "column",
    "row_condition",
    "condition_parser",
)`

To Reproduce Add expectation ExpectColumnValuesToMatchRegex

Expected behavior Empty logs

Environment (please complete the following information):

Additional context Add any other context about the problem here.

adeola-ak commented 3 weeks ago

Hi there! Redshift isn’t a fully supported or tested data source in Great Expectations. This means that while it can work with SQLAlchemy, certain features or expectations, like ColumnMapExpectation, might run into issues if Redshift handles SQL slightly differently than other databases.

We generate SQL queries to validate data against your expectations. If Redshift doesn’t fully support the type of SQL query being generated, that can cause the validation to fail. Redshift may not support certain functions or data operations that the expectation relies on.

I’d recommend reviewing the SQL queries generated by the expectation to ensure they’re compatible with Redshift’s SQL dialect. You can enable SQL query logging in SQLAlchemy to see what’s being executed and compare it with Redshift’s SQL requirements. If there’s a mismatch, it may be necessary to adjust the_validate() method in your custom expectation to work better with Redshift.

Unfortunately, as Redshift isn't supported, I will be closing this issue. But I do encourage you to call on our community for assistance if the above tip I shared doesn't get you in the right direction. Thank you for reaching out!

VolkovGeoPhy commented 3 weeks ago

@adeola-ak nice but it doesn't relate to Redshift dialect. https://greatexpectationstalk.slack.com/archives/CUTCNHN82/p1729850880800489?thread_ts=1729691473.828959&cid=CUTCNHN82 It just inconsistency in your code. Check the classes that I meantioned in the description!

VolkovGeoPhy commented 3 weeks ago

You have one class that inherits another. And in your child class you use validation that fails because of the structure of the parent class

VolkovGeoPhy commented 1 week ago

@adeola-ak the same with 1.2.2 version