Unable to retrieve validation definition name='XXX' id='XXX' from store (type=value_error)

[SiddhantSadangi]

Describe the bug Cannot convert an ephemeral data context to a file context

To Reproduce

great_expectations.yml

# Welcome to Great Expectations! Always know what to expect from your data.
#
# Here you can define datasources, batch kwargs generators, integrations and
# more. This file is intended to be committed to your repo. For help with
# configuration please:
#   - Read our docs: https://docs.greatexpectations.io/docs/guides/connecting_to_your_data/connect_to_data_overview/#2-configure-your-datasource
#   - Join our slack channel: http://greatexpectations.io/slack

# config_version refers to the syntactic version of this config file, and is used in maintaining backwards compatibility
# It is auto-generated and usually does not need to be changed.
config_version: 4

# This config file supports variable substitution which enables: 1) keeping
# secrets out of source control & 2) environment-based configuration changes
# such as staging vs prod.
#
# When GX encounters substitution syntax (like `my_key: ${my_value}` or
# `my_key: $my_value`) in the great_expectations.yml file, it will attempt
# to replace the value of `my_key` with the value from an environment
# variable `my_value` or a corresponding key read from this config file,
# which is defined through the `config_variables_file_path`.
# Environment variables take precedence over variables defined here.
#
# Substitution values defined here can be a simple (non-nested) value,
# nested value such as a dictionary, or an environment variable (i.e. ${ENV_VAR})
#
#
# https://docs.greatexpectations.io/docs/guides/setup/configuring_data_contexts/how_to_configure_credentials

config_variables_file_path: uncommitted/config_variables.yml

# The plugins_directory will be added to your python path for custom modules
# used to override and extend Great Expectations.
plugins_directory: plugins/

stores:
# Stores are configurable places to store things like Expectations, Validations
# Data Docs, and more. These are for advanced users only - most users can simply
# leave this section alone.
  expectations_store:
    class_name: ExpectationsStore
    store_backend:
      class_name: TupleFilesystemStoreBackend
      base_directory: expectations/

  validation_results_store:
    class_name: ValidationResultsStore
    store_backend:
      class_name: TupleFilesystemStoreBackend
      base_directory: uncommitted/validations/

  checkpoint_store:
    class_name: CheckpointStore
    store_backend:
      class_name: TupleFilesystemStoreBackend
      suppress_store_backend_id: true
      base_directory: checkpoints/

  validation_definition_store:
    class_name: ValidationDefinitionStore
    store_backend:
      class_name: TupleFilesystemStoreBackend
      base_directory: validation_definitions/

expectations_store_name: expectations_store
validation_results_store_name: validation_results_store
checkpoint_store_name: checkpoint_store

data_docs_sites:
  # Data Docs make it simple to visualize data quality in your project. These
  # include Expectations, Validations & Profiles. The are built for all
  # Datasources from JSON artifacts in the local repo including validations &
  # profiles from the uncommitted directory. Read more at https://docs.greatexpectations.io/docs/terms/data_docs
  local_site:
    class_name: SiteBuilder
    # set to false to hide how-to buttons in Data Docs
    show_how_to_buttons: true
    store_backend:
        class_name: TupleFilesystemStoreBackend
        base_directory: uncommitted/data_docs/local_site/
    site_index_builder:
        class_name: DefaultSiteIndexBuilder

analytics_enabled: True

The code is a copy of the tutorial available in the GX Core docs here, with context.convert_to_file_context() appended to the end.

# Import required modules from GX library.
import great_expectations as gx

# Create Data Context.
context = gx.get_context()

# Connect to data.
# Create Data Source, Data Asset, Batch Definition, and Batch.
connection_string = "postgresql+psycopg2://try_gx:try_gx@postgres.workshops.greatexpectations.io/gx_example_db"

data_source = context.data_sources.add_postgres(
    "postgres db", connection_string=connection_string
)
data_asset = data_source.add_table_asset(name="taxi data", table_name="nyc_taxi_data")

batch_definition = data_asset.add_batch_definition_whole_table("batch definition")
batch = batch_definition.get_batch()

# Create Expectation Suite containing two Expectations.
suite = context.suites.add(
    gx.core.expectation_suite.ExpectationSuite(name="expectations")
)
suite.add_expectation(
    gx.expectations.ExpectColumnValuesToBeBetween(
        column="passenger_count", min_value=1, max_value=6
    )
)
suite.add_expectation(
    gx.expectations.ExpectColumnValuesToBeBetween(column="fare_amount", min_value=0)
)

# Create Validation Definition.
validation_definition = context.validation_definitions.add(
    gx.core.validation_definition.ValidationDefinition(
        name="validation definition",
        data=batch_definition,
        suite=suite,
    )
)

# Create Checkpoint, run Checkpoint, and capture result.
checkpoint = context.checkpoints.add(
    gx.checkpoint.checkpoint.Checkpoint(
        name="checkpoint", validation_definitions=[validation_definition]
    )
)

checkpoint_result = checkpoint.run()
print(checkpoint_result.describe())

context.convert_to_file_context()

Traceback

Traceback (most recent call last):
  File "c:\Users\siddh\Code\Neptune\adhoc\test.py", line 51, in <module>
    context.convert_to_file_context()
  File "C:\Users\siddh\python_envs\py310\lib\site-packages\great_expectations\data_context\data_context\ephemeral_data_context.py", line 94, in convert_to_file_context
    return migrator.migrate()
  File "C:\Users\siddh\python_envs\py310\lib\site-packages\great_expectations\data_context\migrator\file_migrator.py", line 63, in migrate
    self._migrate_primary_stores(
  File "C:\Users\siddh\python_envs\py310\lib\site-packages\great_expectations\data_context\migrator\file_migrator.py", line 89, in _migrate_primary_stores
    self._migrate_store(
  File "C:\Users\siddh\python_envs\py310\lib\site-packages\great_expectations\data_context\migrator\file_migrator.py", line 110, in _migrate_store
    source_obj = source_store.get(key)
  File "C:\Users\siddh\python_envs\py310\lib\site-packages\great_expectations\data_context\store\store.py", line 235, in get
    return self.deserialize(value)
  File "C:\Users\siddh\python_envs\py310\lib\site-packages\great_expectations\data_context\store\checkpoint_store.py", line 79, in deserialize
    return Checkpoint.parse_raw(value)
  File "C:\Users\siddh\python_envs\py310\lib\site-packages\pydantic\v1\main.py", line 549, in parse_raw
    return cls.parse_obj(obj)
  File "C:\Users\siddh\python_envs\py310\lib\site-packages\pydantic\v1\main.py", line 526, in parse_obj
    return cls(**obj)
  File "C:\Users\siddh\python_envs\py310\lib\site-packages\pydantic\v1\main.py", line 341, in __init__
    raise validation_error
pydantic.v1.error_wrappers.ValidationError: 1 validation error for Checkpoint
validation_definitions
  Unable to retrieve validation definition name='validation definition' id='d7087430-af5f-42e3-a38c-8515e36e9e8e' from store (type=value_error)

Expected behavior Successfully export ephemeral context to file

Environment (please complete the following information):

• Operating System: Windows11
• Great Expectations Version: 1.0.0
• Data Source: PostgreSQL
• Cloud environment: NA

Additional context Add any other context about the problem here.

[adeola-ak]

i think that behavior is usually done in the beginning of the file after fetching the context.

when i first fetch the context, and then use context.convert_to_file_context() to change it, i am not met with the error that you're facing

[SiddhantSadangi]

@adeola-ak This workflow used to run fine in gx<1.0

[adeola-ak]

i can look more into the cause of why it's not working similarly but are you unblocked now?

[SiddhantSadangi]

Hey @adeola-ak , Sorry, I have been a bit occupied.

I'll be unblocked if the workarounds suggested in #10270 work. I'll check and update both the threads.

[SiddhantSadangi]

Sorry for the delay here, but I was finally able to test it. Works for me now ✅