Error creating ConsumerQuotaOverride: googleapi: Error 404: No quota limit with limitId

duxbuse commented 6 months ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

❯ terraform -v      
Terraform v1.6.6
on darwin_arm64

Affected Resource(s)

google_service_usage_consumer_quota_override

Terraform Configuration Files

resource "google_service_usage_consumer_quota_override" "gemini" {
  provider       = google-beta
  project        = module.project-factory.project_id
  service        = "aiplatform.googleapis.com"
  metric         = urlencode("aiplatform.googleapis.com/generate_content_requests_per_minute_per_project_per_base_model")
  limit          = urlencode("/min/project/region/base_model")
  override_value = "0"
  force          = true
}

And also getting the same with

resource "google_service_usage_consumer_quota_override" "palm2" {
  provider       = google-beta
  project        = module.project-factory.project_id
  service        = "aiplatform.googleapis.com"
  metric         = urlencode("aiplatform.googleapis.com/online_prediction_requests_per_base_model")
  limit          = urlencode("/min/project/region/base_model")
  override_value = "0"
  force          = true
}

Terraform Plan

Terraform will perform the following actions:

  # module.project.google_service_usage_consumer_quota_override.gemini[0] will be created
  + resource "google_service_usage_consumer_quota_override" "gemini" {
      + force          = true
      + id             = (known after apply)
      + limit          = "%2Fmin%2Fproject%2Fregion%2Fbase_model"
      + metric         = "aiplatform.googleapis.com%2Fgenerate_content_requests_per_minute_per_project_per_base_model"
      + name           = (known after apply)
      + override_value = "0"
      + project        = "tester-6b8d"
      + service        = "aiplatform.googleapis.com"
    }

  # module.project.google_service_usage_consumer_quota_override.palm2[0] will be created
  + resource "google_service_usage_consumer_quota_override" "palm2" {
      + force          = true
      + id             = (known after apply)
      + limit          = "%2Fmin%2Fproject%2Fregion%2Fbase_model"
      + metric         = "aiplatform.googleapis.com%2Fonline_prediction_requests_per_base_model"
      + name           = (known after apply)
      + override_value = "0"
      + project        = "tester-6b8d"
      + service        = "aiplatform.googleapis.com"
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Output

Error: Error creating ConsumerQuotaOverride: googleapi: Error 404: No quota limit with limitId '/min/project/region/base_model' for metric 'aiplatform.googleapis.com/generate_content_requests_per_minute_per_project_per_base_model' found in service 'aiplatform.googleapis.com'.
│ Help Token: ARD_zUbMmtK6H4w-Dg51Mw3b0PoRGIFyXS56hew3CaYQqPNG8Rvjl04btRSRgMrXCbphTjlhKpFYjKK7MEZR_bPkyVZfigE15G6ceN7fnFQW6Xur
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.PreconditionFailure",
│     "violations": [
│       {
│         "subject": "?error_code=101005\u0026limit_id=/min/project/region/base_model\u0026metric=aiplatform.googleapis.com/generate_content_requests_per_minute_per_project_per_base_model\u0026service_name=aiplatform.googleapis.com",
│         "type": "googleapis.com"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "serviceusage.googleapis.com",
│     "metadata": {
│       "limit_id": "/min/project/region/base_model",
│       "metric": "aiplatform.googleapis.com/generate_content_requests_per_minute_per_project_per_base_model",
│       "service_name": "aiplatform.googleapis.com"
│     },
│     "reason": "COMMON_QUOTA_LIMIT_NOT_FOUND"
│   }
│ ]
│ 
│   with module.project.google_service_usage_consumer_quota_override.gemini,
│   on .terraform/modules/project/quota-limits.tf line 1, in resource "google_service_usage_consumer_quota_override" "gemini":
│    1: resource "google_service_usage_consumer_quota_override" "gemini" {
│ 
╵
╷
│ Error: Error creating ConsumerQuotaOverride: googleapi: Error 404: No quota limit with limitId '/min/project/region/base_model' for metric 'aiplatform.googleapis.com/online_prediction_requests_per_base_model' found in service 'aiplatform.googleapis.com'.
│ Help Token: ARD_zUZVfleUAi3MXH6Nz8Npf5PyKtdQO28mxbNMKNGK-TVBYAD-dpbOB6-PtjWByiqMlugu3gId6iPIHqq-hL1wCwM8oS7F2uAA9VSGAFJgM-Vo
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.PreconditionFailure",
│     "violations": [
│       {
│         "subject": "?error_code=101005\u0026limit_id=/min/project/region/base_model\u0026metric=aiplatform.googleapis.com/online_prediction_requests_per_base_model\u0026service_name=aiplatform.googleapis.com",
│         "type": "googleapis.com"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "serviceusage.googleapis.com",
│     "metadata": {
│       "limit_id": "/min/project/region/base_model",
│       "metric": "aiplatform.googleapis.com/online_prediction_requests_per_base_model",
│       "service_name": "aiplatform.googleapis.com"
│     },
│     "reason": "COMMON_QUOTA_LIMIT_NOT_FOUND"
│   }
│ ]
│ 
│   with module.project.google_service_usage_consumer_quota_override.palm2,
│   on .terraform/modules/project/quota-limits.tf line 13, in resource "google_service_usage_consumer_quota_override" "palm2":
│   13: resource "google_service_usage_consumer_quota_override" "palm2" {
│ 
╵

Expected Behavior

Should have created the quota override

running:

gcloud alpha services quota list \  
   --service=aiplatform.googleapis.com \
   --consumer=projects/tester-6b8d

I can see that unit: 1/min/{project}/{region}/{base_model} for both metric: aiplatform.googleapis.com/generate_content_requests_per_minute_per_project_per_base_model and metric: aiplatform.googleapis.com/online_prediction_requests_per_base_model

Steps to Reproduce

terraform apply

Important Factoids

It also doesn't work with any of the below

limit = urlencode("1/min/project/region/base_model")
limit = urlencode("/project/region/base_model")
limit = urlencode("1/min/{project}/{region}/{base_model}")

References

Its likley quite similar to this issue https://github.com/hashicorp/terraform-provider-google/issues/12253 As aiplatform.googleapis.com is quite a new api so might behave differently.

b/320650559

edwardmedia commented 6 months ago

@duxbuse it works fine for me with below config. Can you try it?

resource "google_service_usage_consumer_quota_override" "override" {
  provider       = google-beta
  project        = "myproject"
  service        = "servicemanagement.googleapis.com"
  metric         = urlencode("servicemanagement.googleapis.com/default_requests")
  limit          = urlencode("/min/project")
  override_value = "0"
  force          = true
}

duxbuse commented 6 months ago

@edwardmedia That snippet you gave runs fine. but the vertex ai quotas do not.

edwardmedia commented 6 months ago

https://github.com/hashicorp/terraform-provider-google/issues/12253

@duxbuse do you have a working script for create / update?

gcloud alpha services quota create / update

duxbuse commented 6 months ago

Yup this works fine

gcloud alpha services quota create \
    --service=aiplatform.googleapis.com --consumer=projects/myproject \
    --metric=aiplatform.googleapis.com/online_prediction_requests_per_base_model \
    --unit=1/min/{project}/{region}/{base_model} \
    --value=0 \
    --force

edwardmedia commented 6 months ago

@duxbuse what is the value for {base_model} in your case?

Using below script, I receive the same error COMMON_QUOTA_LIMIT_NOT_FOUND as from running the terraform script. Until we get the right config, it is hard to say if there is a code issue in the terraform resource.

gcloud alpha services quota create \
    --service=aiplatform.googleapis.com --consumer=projects/myproject \
    --metric=aiplatform.googleapis.com/online_prediction_requests_per_base_model \
    --unit=1/min/myproject/us-central1/gemini-pro-vision \
    --value=0 \
    --force

duxbuse commented 6 months ago

its a dimension field. so can be lots of values???

In my use case I want them all, cause I essentially want to disable llm's

edwardmedia commented 6 months ago

@duxbuse I receive the same error buy running example here. Do you have the working gcloud script?

duxbuse commented 6 months ago

Ohh no the script i sent above is exactly what I ran.

this line was a literal no substitution --unit=1/min/{project}/{region}/{base_model} \

edwardmedia commented 6 months ago

I see. Thank you @duxbuse Below api call is working.

https://serviceusage.googleapis.com/v1beta1/projects/1234567890/services/aiplatform.googleapis.com/consumerQuotaMetrics:importConsumerOverrides?alt=json

{
    "force": true,
    "inlineSource": {
        "overrides": [
            {
                "metric": "aiplatform.googleapis.com/online_prediction_requests_per_base_model",
                "overrideValue": "0",
                "unit": "1/min/{project}/{region}/{base_model}"
            }
        ]
    }
}

Currently the provider builds this format for the url. The POST body is different too.. We may want to review both.

duxbuse commented 5 months ago

Hi @edwardmedia any update on this one?

danninov commented 1 month ago

Waiting for update. I got same error as above.

vishnups1 commented 5 days ago

Any update on this one? Waiting for a fix :-)

hashicorp / terraform-provider-google