PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.96k stars 1.57k forks source link

Clarify block concepts to address the common misconception that secret storage is a general requirement of using Prefect. #14337

Closed taylor-curran closed 3 weeks ago

taylor-curran commented 3 months ago

First check

Describe the issue

Folks often report the misconception that secret storage is a general requirement of using Prefect or at least a general requirement of using any of our integration libraries. This likely stems from their misunderstanding of general block concepts.

Currently our blocks concept page doesn't point users to our docs for using Third party secrets with Prefect.

Large companies that use Prefect ask about this consistently.

I also think our integrations docs should point to general block concepts more clearly. Its really confusing seeing blocks in our integration libraries without any further explanation of what they are.

Describe the proposed change

We could call out something like:

Prefect will never require secret storage! Per our hybrid model, our API is built to operate using only metadata. For more information on how blocks can be compatible with third party secret storage solutions (like those of your cloud providers) check out this guide.

Concepts explained in Third party secrets with Prefect include important information for how to reason about blocks and creds such as:

The concept that values can be left blank for cloud provider creds since cloud providers do a great job of checking multiple places for creds.

From our third part docs:

Values for Access Key ID and Secret Access Key are read from the compute environment. Your AWS Access Key ID and Secret Access Key values with permissions to read the AWS Secret are stored locally in your ~/.aws/credentials file, so leave those fields blank. By leaving those attributes blank, Prefect knows to look to the compute environment.

Specify a region in your AWSCredentials block instead of your local AWS config file. The AwsCredentials block takes precedence and is more portable. Under the hood, Prefect uses the AWS boto3 client to create a session. In the AwsCredentials section of the form, click Add + and create an AWS Credentials block by entering the necessary values. If the compute environment contains the necessary credentials, Prefect will use them to authenticate in the order shown in the Boto3 docs. Follow the same order to resolve the AWS region. Specify the region in your AWSCredentials block so your connection works regardless of the contents of your local AWS config file—or whether you run your code on AWS compute located in anther region than your secret.

Also the concept that blocks can be used without saving them can be useful:

Use Prefect's blocks for convenient access to Snowflake. Don't save the blocks to ensure the credentials are not stored in Prefect Cloud.

Additional context

No response

taylor-curran commented 3 months ago

Another thought, we should link to this doc for Read and Write Data to and from Cloud Provider Storage at the part of the concept doc where the load method is introduced so people get a sense of how they could use blocks.