Feat(Terraform): Baseline and Genai Scenarios

[nidhirw]

What's included in this PR?

This pull request introduces several significant changes to the project, primarily focusing on improving the development environment and adding comprehensive Terraform-based deployment guides and configurations for the 'Baseline' and 'GenAI' scenarios. Below is a summary of the most important changes:

Features

1. Terraform Templates for Baseline Scenario:

• Azure Resource Configuration for APIM:

  • Introduced a new networking.tf file to manage Azure Virtual Networks, subnets, and Network Security Groups (NSGs).
  • Added service delegation for the deployment subnet.
  • Added new variables and outputs for easier reference to subnet and VNet IDs.

• Monitoring Resources:

  • Created azmon.tf for setting up azurerm_log_analytics_workspace and azurerm_application_insights.
  • Included lifecycle rules to prevent accidental deletion of critical monitoring resources.
  • Added outputs for workspaceId, instrumentationKey, and keyVaultId.

• Private DNS Zone Module:

  • Added a new module for azurerm_private_dns_zone and azurerm_private_dns_zone_virtual_network_link.
  • Provided flexibility with new variables for DNS zones and VNet linkage.

• Private Endpoint Configuration:

  • Created a module for azurerm_private_endpoint including private service connection and DNS zone group setup.
  • Introduced outputs for id, private_dns_zone_group, and private_dns_zone_configs.

• Key Vault Setup:

  • Added a Terraform configuration for creating and managing Key Vault, including access policies and private endpoints.

• Provider and Backend Configuration:

  • Introduced provider.tf for setting up Terraform providers and backend storage configurations.

2. Terraform Templates for GenAI Scenario

• OpenAI Module:

  • Added new modules for azurerm_cognitive_account and azurerm_cognitive_deployment for three openai instances and multiple openai model deployments.
  • Provided flexibility with new variables for DNS zones and VNet linkage.

• Private DNS Zone Module:

  • Added a new module for azurerm_private_dns_zone and azurerm_private_dns_zone_virtual_network_link, that are utilised in openai private endpoints per openai instance.
  • Provided flexibility with new variables for DNS zones and VNet linkage.

• Private Endpoint Configuration:

  • Created a module for azurerm_private_endpoint including private service connection and DNS zone group setup, utilised for three openai instances.
  • Introduced outputs for id, private_dns_zone_group, and private_dns_zone_configs.

• API OpenAI Policies:

  • Created a module for azurerm_api_management_product, azurerm_api_management_product_api and three azurerm_api_management_backend deployments for each openai instance.
  • Created azurerm_api_management_policy_fragment for each openai policy.

• EventHub:

  • Created a module for azurerm_eventhub and azurerm_eventhub_namespace including a role assignment for API management.

3. Deployment Scripts:

• Refactored and relocated deployment scripts under scenarios/scripts/bicep.
• Created deployment scripts for terraform templates, under scenarios/scripts/terraform.
• Updated paths in scripts to match the new directory structure.

Docs

1. Documentation Updates:
   • Updated README files with correct paths and deployment instructions reflecting the new structure.
   • Added a Terraform-based deployment guide for both apim-baseline and workload-genai scenarios, including detailed steps and prerequisites.

Chore

1. Dev Container

   • Added use of dev container.

2. Terraform Sample Script for Terraform Backend

   • Added the script scenarios/scripts/terraform/azure-backend-sample.sh to create the backend required for terraform.

3. Code Formatting:

   • Updated main.bicep and other related files to improve code readability and consistency.
   • Made minor formatting changes across scripts for better readability and maintainability.

How to review the PR

How to deploy the baseline scenario:

• Follow the READ.me here: scenarios/apim-baseline/terraform/README.md

• After following the steps, you will have a .env and your terraform backend configuration in ${ENVIRONMENT_TAG}-backend.hcl e.g. dev-backend.hcl.

• Run the following command to deploy the APIM baseline

  ./scripts/terraform/deploy-apim-baseline.sh -y

• At the end of the deployment, run the CURL command that is generated in the command line.

How to deploy the GenAI scenario:

After running the baseline deployment script, run the following command to deploy the genai scenario:

```bash
./scripts/terraform/deploy-workload-genai.sh
```

• At the end of the deployment, run the CURL command that is generated in the command line.