PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.82k stars 1.55k forks source link

Add GCP firestore #13051

Open adeebdvlpr opened 5 months ago

adeebdvlpr commented 5 months ago

There is currently no prefex integrations to connect and process data to Google firestore.

Expectation / Proposal

Add to the prefex-gcp integration components to interact with firestore. Tasks to create a collection and document, read collections and documents, update documents, delete documents, and query a collection.  

Traceback / Example

@task async def firestore_create_collection( collection: str, gcp_credentials: GcpCredentials, project: Optional[str] = None, location: Optional[str] = None, ) -> DocumentReference: logger = get_run_logger() logger.info("Creating collection: %s", collection)

if not project:
    raise ValueError("Project ID must be provided.")

firestore_client = gcp_credentials.get_firestore_client(project=project, location=location)

collection_ref = firestore_client.collection(collection)

# Attempt to create the collection (collections in Firestore are created implicitly)
return collection_ref
desertaxle commented 5 months ago

I'm very excited to see what you build @adeebdvlpr! Let us know if you have any questions while working on this integration.