great-expectations / great_expectations

Always know what to expect from your data.
https://docs.greatexpectations.io/
Apache License 2.0
9.86k stars 1.52k forks source link

FIx/Document Plugins with EphermalDataContext #10056

Closed ibbykhazanchi closed 1 month ago

ibbykhazanchi commented 3 months ago

Is your feature request related to a problem? Please describe. I am following the gx, openlineage integration tutorial. This tutorial assumes you are using yaml configuration to define your action list, but I am using an ephemeral data context. The way I instantiate my action list is by doing the following:

        return self._context.add_or_update_checkpoint(name="checkpoint", validator=self._validator, 
            action_list=[
                {
                    "name": "open_lineage",
                    "action": {
                        "class_name": "OpenLineageValidationAction",
                        "module_name": "openlineage.common.provider.great_expectations",
                        "openlineage_host": openlineage_host,
                        "openlineage_namespace": "my-namespace",
                        "job_name": "my-job-name"
                    },
                }
            ],
        )

During runtime, the validation fails due to this error: 'No module namedopenlineage.common.provider.great_expectationscould be found in your plugins directory.\n - Please verify your plugins directory is configured correctly.\n - Please verify you have a module namedopenlineage.common.provider.great_expectationsin your plugins directory.\n

I am using an ephemeral data context, and in no shape or form is using plugins documented, so I would not expect to see this error.

Describe the solution you'd like If using plugins is mandatory to use the openlineage integration, please document how one can do so if they're using an ephemeraldatacontext.

Describe alternatives you've considered I have experimented with creating my own plugin folder in my root, populating the folder with the openlineage integration code, and manually specifying my plugin folder like this:

        defaults = gx.data_context.types.base.InMemoryStoreBackendDefaults()
        config = gx.data_context.types.base.DataContextConfig(
            anonymous_usage_statistics={"enabled": False},
            fluent_datasources={},
            data_docs_sites={},
            store_backend_defaults=defaults,
            plugins_directory=os.path.join(os.getcwd(), "src/expectations_svc/plugins")
        )
        self._context = gx.get_context(config)

But this does not work either.

Additional context If any more context is required, I'm happy to provide!

molliemarie commented 1 month ago

Hello @ibbykhazanchi. With the launch of Great Expectations Core (GX 1.0), we are closing old issues posted regarding previous versions. Moving forward, we will focus our resources on supporting and improving GX Core (version 1.0 and beyond). If you find that an issue you previously reported still exists in GX Core, we encourage you to resubmit it against the new version. With more resources dedicated to community support, we aim to tackle new issues swiftly. For specific details on what is GX-supported vs community-supported, you can reference our integration and support policy.

To get started on your transition to GX Core, check out the GX Core quickstart (click “Full example code” tab to see a code example).

You can also join our upcoming community meeting on August 28th at 9am PT (noon ET / 4pm UTC) for a comprehensive rundown of everything GX Core, plus Q&A as time permits. Go to https://greatexpectations.io/meetup and click “follow calendar” to follow the GX community calendar.

Thank you for being part of the GX community and thank you for submitting this issue. We're excited about this new chapter and look forward to your feedback on GX Core. 🤗