One-click GCP stack deployments

stefannica commented 1 month ago

Describe changes

Adding GCP to the list of cloud providers for which ZenML supports "one-click" full stack deployments - ZenML stacks with attached infrastructure provisioned through automated in-browser cloud-specific mechanisms, in this case GCP Cloud Shell and GCP Deployment Manager.

This allows users to easily deploy a full GCP ZenML stack with all the associated infrastructure and credentials:

from the CLI with the zenml stack deploy -p gcp command
from the dashboard, using the "Smart stack setup" workflows

The GCP ZenML stack includes:

S3 bucket as artifact store
Artifact Registry as container registry
Vertex AI as orchestrator
GCP Cloud Build as image builder

Example CLI output:

$ zenml stack deploy -p gcp --set --name gcp-stefan --location europe-west2

GCP ZenML Cloud Stack Deployment                                                                                                                                                                                   
================================

Provision and register a basic GCP ZenML stack authenticated and connected to all the necessary cloud infrastructure resources required to run pipelines in GCP.                                                   

Instructions                                                                                                                                                                                                       

You will be redirected to a GCP Cloud Shell console in your browser where you'll be asked to log into your GCP project and then create a Deployment Manager deployment to provision the necessary cloud resources  
for ZenML.                                                                                                                                                                                                         

NOTE: The Deployment Manager deployment will create the following new resources in your GCP project. Please ensure you have the necessary permissions and are aware of any potential costs:                        

 • A GCS bucket registered as a ZenML artifact store.                                                                                                                                                              
 • A Google Artifact Registry registered as a ZenML container registry.                                                                                                                                            
 • Vertex AI registered as a ZenML orchestrator.                                                                                                                                                                   
 • GCP Cloud Build registered as a ZenML image builder.                                                                                                                                                            
 • A GCP Service Account with the minimum necessary permissions to access the above resources.                                                                                                                     
 • An GCP Service Account access key used to give access to ZenML to connect to the above resources through a ZenML service connector.                                                                             

The Deployment Manager deployment will automatically create a GCP Service Account secret key and will share it with ZenML to give it permission to access the resources created by the stack. You can revoke these 
permissions at any time by deleting the Deployment Manager deployment in the GCP Cloud Console.                                                                                                                    

Estimated costs                                                                                                                                                                                                    

A small training job would cost around: $0.60                                                                                                                                                                      

These are rough estimates and actual costs may vary based on your usage and specific GCP pricing. Some services may be eligible for the GCP Free Tier. Use the GCP Pricing Calculator for a detailed estimate based
on your usage.                                                                                                                                                                                                     

⚠️ The Cloud Shell session will warn you that the ZenML GitHub repository is untrusted. We recommend that you review the contents of the repository and then check the Trust repo checkbox to proceed with the      
deployment, otherwise the Cloud Shell session will not be authenticated to access your GCP projects.                                                                                                               

💡 After the Deployment Manager deployment is complete, you can close the Cloud Shell session and return to the CLI to view details about the associated ZenML stack automatically registered with ZenML.          

Configuration                                                                                                                                                                                                      

You will be asked to provide the following configuration values during the deployment process:                                                                                                                     

### BEGIN CONFIGURATION ###
ZENML_STACK_NAME=my-stack
ZENML_STACK_REGION=europe-west2
ZENML_SERVER_URL=https://...-zenml.cloudinfra.zenml.io
ZENML_SERVER_API_TOKEN=....
### END CONFIGURATION ###

Proceed to continue with the deployment. You will be automatically redirected to GCP in your browser. [y/n]: y
If your browser did not open automatically, please open the following URL into your browser to deploy the stack to GCP: GCP Cloud Shell Console.                                                                   

Waiting for the deployment to complete and the stack to be registered. Press CTRL+C to abort...

Implementation Details

This PR mainly uses the existing code structure already put in place with the introduction of AWS full stack deployments in the previous release, with the following minor modifications:

the stack deployment API endpoint is modified to also return an optional configuration string in addition to the deployment URL. Users are responsible for copy-pasting this configuration from the CLI/dashboard into the cloud provider console in cases which don't allow transferring this information automatically through URL query params (GCP is one of those cases).
the stack deployment API endpoint also returns the list of integrations required by the stack
the GCP service connector had to be adjusted to allow credentials to be passed in base64-encoded format

The GCP stack is provisioned through a GCP Cloud Shell session with an included MarkDown tutorial and bash script that the user needs to update and execute manually to deploy a GCP Deployment Manager template. This complication was required due to the following factors:

no way to trigger a GCP Deployment Manager template creation directly through a URL
no way to include custom parameters in the GCP Cloud Shell URL

Side-changes

fix the "publish AWS stack deployment templates" github action
introduce a similar github action step for the GCP related artifacts
add local image builder to the full AWS ZenML stack registered by the AWS cloud formation template (for completion sake)
include the AWS account name in the name of the S3 bucket provisioned by the AWS cloud formation template. This is done to avoid S3 namespace clashing (note: all S3 bucket names share the same global namespace regardless of AWS account, region or owner).
include more instruction in the zenml stack deploy CLI after the full stack is deployed: how to activate the stack, which integrations to install
include cost estimation in the zenml stack deploy CLI
add support to allow GCP service connector credentials to be provided as base64 encoded JSON strings. This is necessary because this is the ONLY format supported by the GCP Deployment Manager service account key resource.
increase the lifetime of the ZenML API token generated for cloud stack deployments from 1 hour to 6 hours, to give users more time to go through the stack deployment process.
fix a minor bug in the zenml DB upgrade logic that caused DB upgrade failures in development
validate that the stack component flavors mentioned in the "full stack registration" endpoint are valid

Pre-requisites

Please ensure you have done the following:

[x] I have read the CONTRIBUTING.md document.
[x] If my change requires a change to docs, I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[x] I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.
[x] If my changes require changes to the dashboard, these changes are communicated/requested.

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Other (add details above)

coderabbitai[bot] commented 1 month ago

[!IMPORTANT]

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

## Walkthrough This update significantly enhances the ZenML stack deployment process across AWS and GCP, introducing detailed configuration steps, new deployment scripts, and updated documentation for streamlined cloud deployments. It also incorporates error handling improvements, deployment support for GCP, and enhancements to integration and validation mechanisms. ## Changes | File Path | Change Summary | |------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | .github/workflows/publish_stack_templates.yml | Added permissions for `contents` to read and `id-token` to write in the workflow job. | | docs/book/.../image-builders/gcp.md | Added a hint block to streamline the deployment process using cloud deployment and registration wizards. | | docs/book/.../deploy-a-cloud-stack.md | Enhanced deployment instructions for ZenML stacks on GCP, detailing steps, warnings, and provisioning resources via Deployment Manager. | | infra/README.md | Added detailed configuration steps and parameterized templates for AWS and GCP deployment scripts. | | infra/aws/aws-ecr-s3-sagemaker.yaml | Updated resource naming conventions and added new configurations, including an "image_builder" configuration with a "local" flavor attribute. | | infra/gcp/... | Introduced a deployment tutorial, script updates, and new YAML configurations for GCP. | | src/zenml/cli/stack.py | Improved error handling for invalid deployment locations, updated messaging, and streamlined stack deployment instructions. | | src/zenml/constants.py | Added new constants for configuration paths and token expiration. | | src/zenml/enums.py | Added `GCP` to the `StackDeploymentProvider` enum. | | src/zenml/.../gcp_service_connector.py | Updated credential classes to support base64 encoding for JSON fields, enhancing validation functions accordingly. | | src/zenml/models/__init__.py | Added imports and exported entities for `StackDeploymentConfig`. | | src/zenml/models/.../stack_deployment.py | Added new fields to `StackDeploymentInfo` and `StackDeploymentConfig` classes for better deployment details and configuration. | | src/zenml/.../aws_stack_deployment.py | Updated descriptions, instructions, and method signatures for AWS stack deployment, including integration requirements and cost estimates. | | src/zenml/.../gcp_stack_deployment.py | Introduced functionality for deploying ZenML stacks to GCP with methods for managing deployment processes and configurations. | | src/zenml/.../stack_deployment.py | Enhanced `ZenMLCloudStackDeployment` class with new variables and methods for deployment configuration, integration requirements, and stack retrieval. | | src/zenml/.../utils.py | Included `GCPZenMLCloudStackDeployment` in stack deployment providers. | | src/zenml/.../stack_deployment_endpoints.py | Refactored endpoint logic for stack deployment config retrieval and token expiration management. | | src/zenml/.../rest_zen_store.py | Added `StackDeploymentConfig` to return types and updated methods for improved stack deployment handling and error management. | | src/zenml/.../sql_zen_store.py | Modified database migration logic, error handling, and configuration management in stack deployment methods. | | src/zenml/.../zen_store_interface.py | Updated `get_stack_deployment_config` method signature to return `StackDeploymentConfig`. | ## Sequence Diagram(s) ```mermaid sequenceDiagram participant User participant CLI participant ZenMLServer participant CloudProvider User ->> CLI: Initiates Deployment CLI ->> ZenMLServer: Request Deployment Configuration ZenMLServer ->> CLI: Returns DeploymentConfig CLI ->> CloudProvider: Provision Resources CloudProvider -->> CLI: Resources Provisioned CLI ->> ZenMLServer: Register Deployment ZenMLServer -->> CLI: Deployment Registered CLI ->> User: Deployment Complete ``` ## Poem > 🐰 In the cloud where data thrives, > With scripts and stacks, ZenML arrives. > AWS and GCP, both in stride, > A seamless journey, a smoother ride. > Errors handled, configs enhanced, > To new deployments, let’s advance! 🚀

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai) - [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai) - [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai) - [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

Tips

### Chat There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai): - Review comments: Directly reply to a review comment made by CodeRabbit. Example: - `I pushed a fix in commit .` - `Generate unit testing code for this file.` - `Open a follow-up GitHub issue for this discussion.` - Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples: - `@coderabbitai generate unit testing code for this file.` - `@coderabbitai modularize this function.` - PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples: - `@coderabbitai generate interesting stats about this repository and render them as a table.` - `@coderabbitai show all the console.log statements in this repository.` - `@coderabbitai read src/utils.ts and generate unit testing code.` - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.` - `@coderabbitai help me debug CodeRabbit configuration file.` Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. ### CodeRabbit Commands (invoked as PR comments) - `@coderabbitai pause` to pause the reviews on a PR. - `@coderabbitai resume` to resume the paused reviews. - `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository. - `@coderabbitai full review` to do a full review from scratch and review all the files again. - `@coderabbitai summary` to regenerate the summary of the PR. - `@coderabbitai resolve` resolve all the CodeRabbit review comments. - `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository. - `@coderabbitai help` to get help. Additionally, you can add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed. ### CodeRabbit Configration File (`.coderabbit.yaml`) - You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository. - Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information. - If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json` ### Documentation and Community - Visit our [Documentation](https://coderabbit.ai/docs) for detailed information on how to use CodeRabbit. - Join our [Discord Community](https://discord.com/invite/GsXnASn26c) to get help, request features, and share feedback. - Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.

stefannica commented 1 month ago

@coderabbitai review

coderabbitai[bot] commented 1 month ago

Actions performed

Review triggered. > Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

avishniakov commented 1 month ago

There are some issue detected by the CI, like docstrings. Please, make sure to sort them out before merging.

zenml-io / zenml