Support for CosmosDB Continuous Backup Policy 7 Day Retention

bamarch commented 2 years ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Please support the 7 Day tier in the cosmosdb account resource.

It currently supports only "Continuous" on the 30 day tier.

ARM templates support backupPolicy.continuousModeProperties.tier = Continuous7Days.

New or Affected Resource(s)/Data Source(s)

azurerm_cosmosdb_account

Potential Terraform Configuration

resource "azurerm_cosmosdb_account" "example" {
  # ...
  backup {
    type = "Continuous"
    continuous_mode_retention_days = 7
  }
  # ...
}

References

https://docs.microsoft.com/en-us/azure/cosmos-db/continuous-backup-restore-introduction#continuous-30-day-tier-vs-continuous-7-day-tier

neil-yechenwei commented 2 years ago

@bamarch ,thanks for raising this issue. After checked, seems service API hasn't supported this feature. So I filed an issue on https://github.com/Azure/azure-rest-api-specs/issues/20253 for tracking.

pacorreia commented 2 years ago

@neil-yechenwei after digging into Azure portal I've found that there's an API for this, the same used by Azure Portal, in preview, but exists, I'm not sure why even the preview is not documented, but it's being used in Azure Portal GA, so not mcuh a preview, but stable enough to be there and documented:

E.g.:

PATCH https://management.azure.com/subscriptions/<subscriptionId>/resourceGroups/<resource-group-name>/providers/Microsoft.DocumentDB/databaseAccounts/<cosmosdb-account-name>/?api-version=2022-08-15-preview
Content-Type: application/json
DNT: 1
x-ms-command-name: Microsoft_Azure_DocumentDB.
Authorization: Bearer <access token>

{
    "properties": {
        "backupPolicy": {
            "type": "Continuous",
            "continuousModeProperties": {
                "tier": "Continuous7Days"
            }
        }
    }
}

or for 30 days:

PATCH https://management.azure.com/subscriptions/<subscriptionId>/resourceGroups/<resource-group-name>/providers/Microsoft.DocumentDB/databaseAccounts/<cosmosdb-account-name>/?api-version=2022-08-15-preview
Content-Type: application/json
DNT: 1
x-ms-command-name: Microsoft_Azure_DocumentDB.
Authorization: Bearer <access token>

{
    "properties": {
        "backupPolicy": {
            "type": "Continuous",
            "continuousModeProperties": {
                "tier": "Continuous30Days"
            }
        }
    }
}

The response is resource defintion in JSON:

HTTP/1.1 200 Ok
Cache-Control: no-store, no-cache
Pragma: no-cache
Transfer-Encoding: chunked
Content-Type: application/json
Content-Encoding: gzip
Location: https://management.azure.com/subscriptions/xxxxxxxxxxxxxxxxxxxxx/resourceGroups/xxxxxxxxxxxxxxxx/providers/Microsoft.DocumentDB/databaseAccounts/xxxxxxxxxxx/operationResults/59e4aee3-e9f1-4501-97e7-6112c0f1a77e?api-version=2022-08-15-preview
Vary: Accept-Encoding
x-ms-request-id: xxxxxxxxxxxxxxxxxxx
Azure-AsyncOperation: https://management.azure.com/subscriptions/xxxxxxxxxxxxxx/providers/Microsoft.DocumentDB/locations/uksouth/operationsStatus/xxxxxxxxxxxx?api-version=2022-08-15-preview
Strict-Transport-Security: max-age=31536000; includeSubDomains
x-ms-gatewayversion: version=2.14.0
Server: Microsoft-HTTPAPI/2.0
x-ms-ratelimit-remaining-subscription-writes: 1199
x-ms-correlation-request-id: xxxxxxxxxxxxxxxxxx
x-ms-routing-request-id: xxxxxxxxxxxxxxxxxxxxxxxxxxx
X-Content-Type-Options: nosniff
Date: Tue, 27 Sep 2022 15:58:22 GMT
Connection: close

{
  "id": "/subscriptions/xxxxxxxxxx/resourceGroups/xxxxxxxxxxxxxxx/providers/Microsoft.DocumentDB/databaseAccounts/xxxxxxxxxxxxxxxx",
  "name": "xxxxxxxxxxxx",
  "location": "UK South",
  "type": "Microsoft.DocumentDB/databaseAccounts",
  "kind": "GlobalDocumentDB",
  "tags": {
    "hidden-cosmos-mmspecial": ""
  },
  "systemData": {
    "createdAt": "2022-06-28T15:12:31.8295795Z"
  },
  "properties": {
    "provisioningState": "Updating",
    "documentEndpoint": "[https://xxxxxxxxx.documents.azure.com:443/](https://xxxxxxxxxxxxxxx.documents.azure.com/)",
    "publicNetworkAccess": "Enabled",
    "enableAutomaticFailover": true,
    "enableMultipleWriteLocations": false,
    "enablePartitionKeyMonitor": true,
    "isVirtualNetworkFilterEnabled": true,
    "virtualNetworkRules": [
      {
        "id": "/subscriptions/xxxxxxxxxxxxxxxxxx/resourceGroups/xxxxxxxxxx/providers/Microsoft.Network/virtualNetworks/xxxxxxxxxxxxxxx/subnets/xxxxxxxxxxxxx",
        "ignoreMissingVNetServiceEndpoint": false
      },
      {
        "id": "/subscriptions/xxxxxxxxxxx/resourceGroups/xxxxxxxxxx/providers/Microsoft.Network/virtualNetworks/xxxxxxxxxxxxxxxxx/subnets/xxxxxxxxxxxxxx",
        "ignoreMissingVNetServiceEndpoint": false
      }
    ],
    "EnabledApiTypes": "Sql",
    "disableKeyBasedMetadataWriteAccess": false,
    "enableFreeTier": false,
    "enableAnalyticalStorage": false,
    "analyticalStorageConfiguration": {
      "schemaType": "WellDefined"
    },
    "instanceId": "xxxxxxxxxxxxxxxxxxxxxxxx",
    "createMode": "Default",
    "databaseAccountOfferType": "Standard",
    "defaultIdentity": "FirstPartyIdentity",
    "networkAclBypass": "None",
    "disableLocalAuth": false,
    "enablePartitionMerge": false,
    "consistencyPolicy": {
      "defaultConsistencyLevel": "Session",
      "maxIntervalInSeconds": 5,
      "maxStalenessPrefix": 100
    },
    "configurationOverrides": {},
    "writeLocations": [
      {
        "id": "xxxxxxxxxxxx",
        "locationName": "UK South",
        "documentEndpoint": "[https://xxxxxxxxx.documents.azure.com:443/](xxxxxxxxxxxxxxxxxxxx.documents.azure.com/)",
        "provisioningState": "Succeeded",
        "failoverPriority": 0,
        "isZoneRedundant": false
      }
    ],
    "readLocations": [
      {
        "id": "xxxxxxxxxxxxxxxxxxx",
        "locationName": "UK South",
        "documentEndpoint": "[https://xxxxxxxxxxxxxx.documents.azure.com:443/](https://xxxxxxxxxxxxxx.documents.azure.com/)",
        "provisioningState": "Succeeded",
        "failoverPriority": 0,
        "isZoneRedundant": false
      },
      {
        "id": "xxxxxxxxxxxxxxxxxxx",
        "locationName": "UK West",
        "documentEndpoint": "[https://xxxxxxxxxxxxxxxxxx.documents.azure.com:443/](https://xxxxxxxxxxxxxxxxx.documents.azure.com/)",
        "provisioningState": "Succeeded",
        "failoverPriority": 1,
        "isZoneRedundant": false
      }
    ],
    "locations": [
      {
        "id": "xxxxxxxxxxxxxxxxxxxxxx",
        "locationName": "UK South",
        "documentEndpoint": "[https://xxxxxxxxxxxxxx.documents.azure.com:443/](https://xxxxxxxxxxxxxx.documents.azure.com/)",
        "provisioningState": "Succeeded",
        "failoverPriority": 0,
        "isZoneRedundant": false
      },
      {
        "id": "xxxxxxxxxxxxxxxxxx",
        "locationName": "UK West",
        "documentEndpoint": "[https://xxxxxxxxxxxxxx.documents.azure.com:443/](https://xxxxxxxxxxxxxx.documents.azure.com/)",
        "provisioningState": "Succeeded",
        "failoverPriority": 1,
        "isZoneRedundant": false
      }
    ],
    "failoverPolicies": [
      {
        "id": "xxxxxxxxxxxxxxxxxxxxxxxxx",
        "locationName": "UK South",
        "failoverPriority": 0
      },
      {
        "id": "xxxxxxxxxxxxxx",
        "locationName": "UK West",
        "failoverPriority": 1
      }
    ],
    "cors": [],
    "capabilities": [],
    "ipRules": [],
    "backupPolicy": {
      "type": "Continuous",
      "continuousModeProperties": {
        "tier": "Continuous30Days"
      }
    },
    "networkAclBypassResourceIds": [],
    "diagnosticLogSettings": {
      "enableFullTextQuery": "None"
    },
    "capacity": {
      "totalThroughputLimit": 4000
    },
    "keysMetadata": {
      "primaryMasterKey": {
        "generationTime": "2022-06-28T15:12:31.8295795Z"
      },
      "secondaryMasterKey": {
        "generationTime": "2022-06-28T15:12:31.8295795Z"
      },
      "primaryReadonlyMasterKey": {
        "generationTime": "2022-06-28T15:12:31.8295795Z"
      },
      "secondaryReadonlyMasterKey": {
        "generationTime": "2022-06-28T15:12:31.8295795Z"
      }
    }
  },
  "identity": {
    "type": "None"
  }
}

Hope this helps :)

nimmyarjith commented 2 years ago

Any update on this please

RajanikanthGitHub commented 1 year ago

@pacorreia Thank you very much for your help. it worked for me.

aaroncommify commented 1 year ago

Can we get an update on this please? 7 day continuous backups have been available for a while and it doesn't look like you can configure it using the azurerm provider still.

apraveen25 commented 1 year ago

Is this still unsupported?

pacorreia commented 1 year ago

well.... I've provided a solution and docs in how to implement 🙄

A segunda, 10/07/2023, 13:14, apraveen25 @.***> escreveu:

Is this still unsupported?

— Reply to this email directly, view it on GitHub https://github.com/hashicorp/terraform-provider-azurerm/issues/17997#issuecomment-1628841082, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM6OPVYJ4PNZU72N4XXPGKLXPPW3JANCNFSM56RROOOA . You are receiving this because you were mentioned.Message ID: @.***>

apraveen25 commented 1 year ago

@pacorreia Thank you for your solution, appreciate it. But I am looking for a native solution with Terraform provider.

pacorreia commented 1 year ago

I know, I've provided it to be integrated in the provider... until now

A terça, 11/07/2023, 14:13, apraveen25 @.***> escreveu:

@pacorreia https://github.com/pacorreia Thank you for your solution, appreciate it. But I am looking for a native solution with Terraform provider.

— Reply to this email directly, view it on GitHub https://github.com/hashicorp/terraform-provider-azurerm/issues/17997#issuecomment-1630807661, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM6OPV6KPYBO3XXWG6JKSTTXPVGNZANCNFSM56RROOOA . You are receiving this because you were mentioned.Message ID: @.***>

bamarch commented 1 year ago

Anyone after a terraform workaround can use this snippet

resource "azapi_update_resource" "cosmos_example_continuous_7_days" {
  type        = "Microsoft.DocumentDB/databaseAccounts@2023-04-15"
  resource_id = azurerm_cosmosdb_account.example.id

  body = jsonencode({
    properties = {
      backupPolicy = {
        type = "Continuous"
        continuousModeProperties = {
          tier : "Continuous7Days"
        }
      }
    }
  })
}

See more here https://registry.terraform.io/providers/azure/azapi/latest/docs/resources/azapi_update_resource

Note you'll probably then want to use ignore_changes on the parent azurerm_cosmosdb_account resource

  lifecycle {
    ignore_changes = [
      backup, # managing via azapi_update_resource to workaround lack of 7 day retention in azurerm terraform provider
    ]
  }

bamarch commented 1 year ago

@bamarch ,thanks for raising this issue. After checked, seems service API hasn't supported this feature. So I filed an issue on Azure/azure-rest-api-specs#20253 for tracking.

I think it's supported now

https://github.com/Azure/azure-rest-api-specs/blob/23d88533ddfde4d1565a897fe95d42fb0d9333e5/specification/cosmos-db/resource-manager/Microsoft.DocumentDB/stable/2023-04-15/cosmos-db.json#L9895

Hopefully that means this issue is unblocked and Continuous7Days can be implemented in the azurerm terraform provider? 🙏🏻

ivan-zaitsev commented 1 year ago

Any news about this?

pacorreia commented 1 year ago

See the previous comment, seems this was already implemented, but no one came to confirm and close this issue

A quinta, 28/09/2023, 11:14, Ivan Zaitsev @.***> escreveu:

Any news about this?

— Reply to this email directly, view it on GitHub https://github.com/hashicorp/terraform-provider-azurerm/issues/17997#issuecomment-1738866869, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM6OPV4ZM5BBBGCPGQRRWTDX4VEXJANCNFSM56RROOOA . You are receiving this because you were mentioned.Message ID: @.***>

ivan-zaitsev commented 1 year ago

@pacorreia I think it was implemented only in azure-rest-api spec, but not in azurerm provider.

https://github.com/hashicorp/terraform-provider-azurerm/blob/7d44f7eb3980020f778cdd270c1d53f94468e4bf/internal/services/cosmos/cosmosdb_account_resource.go#L518

neilmca-inc commented 9 months ago

Looked at this today as Continuous backup mode appears flagged in Azure Advisor now as best practice

No surprise then to see it's not supported in azurerm_cosmosdb_account - is it going to be?

jeffreyrubi commented 6 months ago

The doc mentioned a "backup" block, is this what your are looking for? https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/cosmosdb_account.html

neilmca-inc commented 6 months ago

The doc mentioned a "backup" block, is this what your are looking for? https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/cosmosdb_account.html

No - see the top description - needs something like a continuous_mode_retention_days setting

hashicorp / terraform-provider-azurerm