search

hashicorp / terraform-provider-google

Terraform Provider for Google Cloud Platform
https://registry.terraform.io/providers/hashicorp/google/latest/docs
Mozilla Public License 2.0
2.36k stars 1.75k forks source link

Add new resources for Vertex AI Vector Search (formally known as, Vertex AI Matching Engine) #12818

Closed shotarok closed 3 months ago

shotarok commented 2 years ago

Community Note

Description

Add new resources for Vertex AI Matching Engine. The index and indexEndpoint API endpoints have create, patch, get, and delete methods. Therefore, they would be straightforward to implement.

On the other hand, the following API endpoints correspond to CRUD operations to deploy an index to an endpoint. Therefore, I put another new resource as an implementation idea in "New or Affected Resource(s)".

As a side note, Vertex AI Endpoints (projects.locations.endpoints) has similar methods: deployModel, undeployModel. That'd be great if both Matching Engine and Endpoint resources could have similar resources to deploy index or model to an endpoint.

New or Affected Resource(s)

Potential Terraform Configuration

# Propose what you think the configuration to take advantage of this feature should look like.
# We may not use it verbatim, but it's helpful in understanding your intent.

References

b/309602461

mjpolak commented 1 year ago

Hey, Is there any schedule on when it will be available? I can create indexes and endpoints now by google_vertex_ai_index and google_vertex_ai_index_endpoint But deploying index require now to implement custom code.

shotarok commented 1 year ago

I'm afraid I don't have any ongoing PR right now. I've been using the following gcloud commands for the deploy and undeploy operations.

mikeedjones commented 1 year ago

Is it possible to add support for the Private service connect options, as documented https://cloud.google.com/vertex-ai/docs/matching-engine/match-eng-setup/private-service-connect

Understand its still in pre-GA.

Thanks!

shotarok commented 1 year ago

Thanks for letting me know about the new feature. It looks like both v1 and v1beta1 support privateServiceConnectConfig in a request body like below. After figuring out how to test a private service connect in an acceptance test of magic-modules, I can work on it.

mikeedjones commented 1 year ago

Thank you! Yes that's what we're using at the moment. The feature isn't supported in the python client library - at either creation or query time.

It looks like privateServiceConnect is deprecated on model Endpoints - is that likely to happen to index endpoints as well?

shotarok commented 1 year ago

It looks like enablePrivateServiceConnect is deprecated in the endpoint. The index endpoint also has the deprecated field. I believe privateServiceConnectConfig is the new alternative field to the deprecated field, although I can't know about any internal roadmap 🙂

ruanspies commented 1 year ago

I'm afraid I don't have any ongoing PR right now. I've been using the following gcloud commands for the deploy and undeploy operations.

Is the deploy and undeploy endpoint functionality within scope for resources associated with Vertex AI Matching Engine? Would be keen to collaborate on it.

shotarok commented 1 year ago

@ruanspies Yes, I put the endpoints for CRUD on the issue description. Because of the API endpoint design, I anticipate a lot of custom resource codes are necessary for implementing a resource like google_vertex_ai_index_endpoint_deployed_index. Please take a look at the description for the details.

As a side note, an index-related operations can take a few hours as their timeouts are set to 180. I'd recommend using the data resource: google_vertex_ai_index when running a test locally.

BenLiyanage commented 1 year ago

Definitely interested in support for modifying the deployed_index.

melinath commented 1 year ago

Remaining work here: Adding support for google_vertex_ai_index_endpoint_deployed_index or google_vertex_ai_deployed_index (or determining that they don't make sense in Terraform)

Ludovic-Emo-Pyl-Tech commented 1 year ago

Need definitely the capability to deploy an index on an endpoint with terraform.

EduardJoy commented 1 year ago

any updates for this?

shotarok commented 1 year ago

@mikeedjones The PR to support PSC was merged yesterday. The PSC will be available after a new version is released!

EduardJoy commented 1 year ago

Great, then the only issue that remains is the google_vertex_ai_index_endpoint_deployed_index resource

davidcavazos commented 10 months ago

Are there any updates on when the google_vertex_ai_index_endpoint_deployed_index is planned to be ready?

davidcavazos commented 9 months ago

In the meantime, here's a workaround using gcloud to deploy and undeploy the index.

resource "google_vertex_ai_index" "my_index" {
  ...
}

resource "google_vertex_ai_index_endpoint" "my_endpoint" {
  ...
}

module "gcloud_ai_index_endpoints_deploy_index" {
  source  = "terraform-google-modules/gcloud/google"
  version = "~> 3.0"

  create_cmd_body  = "ai index-endpoints deploy-index ${google_vertex_ai_index_endpoint.my_endpoint.id} --deployed-index-id=deployed_index --display-name=deployed_index --index=${google_vertex_ai_index.my_index.id} --project=${module.project_services.project_id} --region=${var.region}"
  destroy_cmd_body = "ai index-endpoints undeploy-index ${google_vertex_ai_index_endpoint.my_endpoint.id} --deployed-index-id=deployed_index --project=${module.project_services.project_id} --region=${var.region}"
}
EduardJoy commented 7 months ago

Any luck that someone might pick this up?

Freezaa9 commented 6 months ago

Thanks @davidcavazos Any news about including this natively?

SaschaHeyer commented 5 months ago

Is there any update?

NicolaSpreafico commented 5 months ago

Any update on this?

melinath commented 5 months ago

Hey folks, this ticket has been forwarded to engineering for resolution. Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

nicola-spreafico commented 5 months ago

In the meantime, here's a workaround using gcloud to deploy and undeploy the index.

resource "google_vertex_ai_index" "my_index" {
  ...
}

resource "google_vertex_ai_index_endpoint" "my_endpoint" {
  ...
}

module "gcloud_ai_index_endpoints_deploy_index" {
  source  = "terraform-google-modules/gcloud/google"
  version = "~> 3.0"

  create_cmd_body  = "ai index-endpoints deploy-index ${google_vertex_ai_index_endpoint.my_endpoint.id} --deployed-index-id=deployed_index --display-name=deployed_index --index=${google_vertex_ai_index.my_index.id} --project=${module.project_services.project_id} --region=${var.region}"
  destroy_cmd_body = "ai index-endpoints undeploy-index ${google_vertex_ai_index_endpoint.my_endpoint.id} --deployed-index-id=deployed_index --project=${module.project_services.project_id} --region=${var.region}"
}

Thank you, Your workaround works in order to deploy the index, but I'm having an hard time retrieving the id of the deployed index, because being an asynchronous I should wait in any case after the gcloud command and then retrieve the final Id.

I hope the resource will be supported soon on the terraform provider so it will go smooth in terms of integrations

slocoro commented 5 months ago

Would also be interested in gcloud_ai_index_endpoints_deploy_index

github-actions[bot] commented 2 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.