Add new resources for Vertex AI Vector Search (formally known as, Vertex AI Matching Engine)

shotarok commented 1 year ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

Add new resources for Vertex AI Matching Engine. The index and indexEndpoint API endpoints have create, patch, get, and delete methods. Therefore, they would be straightforward to implement.

On the other hand, the following API endpoints correspond to CRUD operations to deploy an index to an endpoint. Therefore, I put another new resource as an implementation idea in "New or Affected Resource(s)".

Create: projects.locations.indexEndpoints/deployIndex
Read: projects.locations.indexEndpoints/get
- Note: The response object: IndexEndpoint has a list of DeployedIndex as deployedIndexes
Update: projects.locations.indexEndpoints/mutateDeployedIndex
Delete: projects.locations.indexEndpoints/undeployIndex

As a side note, Vertex AI Endpoints (projects.locations.endpoints) has similar methods: deployModel, undeployModel. That'd be great if both Matching Engine and Endpoint resources could have similar resources to deploy index or model to an endpoint.

New or Affected Resource(s)

~google_vertex_ai_index~
~google_vertex_ai_index_endpoint~
google_vertex_ai_index_endpoint_deployed_index or google_vertex_ai_deployed_index

Potential Terraform Configuration

# Propose what you think the configuration to take advantage of this feature should look like.
# We may not use it verbatim, but it's helpful in understanding your intent.

References

b/309602461

mjpolak commented 10 months ago

Hey, Is there any schedule on when it will be available? I can create indexes and endpoints now by google_vertex_ai_index and google_vertex_ai_index_endpoint But deploying index require now to implement custom code.

shotarok commented 10 months ago

I'm afraid I don't have any ongoing PR right now. I've been using the following gcloud commands for the deploy and undeploy operations.

mikeedjones commented 10 months ago

Is it possible to add support for the Private service connect options, as documented https://cloud.google.com/vertex-ai/docs/matching-engine/match-eng-setup/private-service-connect

Understand its still in pre-GA.

Thanks!

shotarok commented 10 months ago

Thanks for letting me know about the new feature. It looks like both v1 and v1beta1 support privateServiceConnectConfig in a request body like below. After figuring out how to test a private service connect in an acceptance test of magic-modules, I can work on it.

mikeedjones commented 10 months ago

Thank you! Yes that's what we're using at the moment. The feature isn't supported in the python client library - at either creation or query time.

It looks like privateServiceConnect is deprecated on model Endpoints - is that likely to happen to index endpoints as well?

shotarok commented 10 months ago

It looks like enablePrivateServiceConnect is deprecated in the endpoint. The index endpoint also has the deprecated field. I believe privateServiceConnectConfig is the new alternative field to the deprecated field, although I can't know about any internal roadmap 🙂

ruanspies commented 9 months ago

I'm afraid I don't have any ongoing PR right now. I've been using the following gcloud commands for the deploy and undeploy operations.

Is the deploy and undeploy endpoint functionality within scope for resources associated with Vertex AI Matching Engine? Would be keen to collaborate on it.

shotarok commented 9 months ago

@ruanspies Yes, I put the endpoints for CRUD on the issue description. Because of the API endpoint design, I anticipate a lot of custom resource codes are necessary for implementing a resource like google_vertex_ai_index_endpoint_deployed_index. Please take a look at the description for the details.

As a side note, an index-related operations can take a few hours as their timeouts are set to 180. I'd recommend using the data resource: google_vertex_ai_index when running a test locally.

BenLiyanage commented 9 months ago

Definitely interested in support for modifying the deployed_index.

melinath commented 8 months ago

Remaining work here: Adding support for google_vertex_ai_index_endpoint_deployed_index or google_vertex_ai_deployed_index (or determining that they don't make sense in Terraform)

Ludovic-Emo-Pyl-Tech commented 8 months ago

Need definitely the capability to deploy an index on an endpoint with terraform.

EduardJoy commented 8 months ago

any updates for this?

shotarok commented 7 months ago

@mikeedjones The PR to support PSC was merged yesterday. The PSC will be available after a new version is released!

EduardJoy commented 7 months ago

Great, then the only issue that remains is the google_vertex_ai_index_endpoint_deployed_index resource

davidcavazos commented 5 months ago

Are there any updates on when the google_vertex_ai_index_endpoint_deployed_index is planned to be ready?

davidcavazos commented 4 months ago

In the meantime, here's a workaround using gcloud to deploy and undeploy the index.

resource "google_vertex_ai_index" "my_index" {
  ...
}

resource "google_vertex_ai_index_endpoint" "my_endpoint" {
  ...
}

module "gcloud_ai_index_endpoints_deploy_index" {
  source  = "terraform-google-modules/gcloud/google"
  version = "~> 3.0"

  create_cmd_body  = "ai index-endpoints deploy-index ${google_vertex_ai_index_endpoint.my_endpoint.id} --deployed-index-id=deployed_index --display-name=deployed_index --index=${google_vertex_ai_index.my_index.id} --project=${module.project_services.project_id} --region=${var.region}"
  destroy_cmd_body = "ai index-endpoints undeploy-index ${google_vertex_ai_index_endpoint.my_endpoint.id} --deployed-index-id=deployed_index --project=${module.project_services.project_id} --region=${var.region}"
}

EduardJoy commented 2 months ago

Any luck that someone might pick this up?

Freezaa9 commented 1 month ago

Thanks @davidcavazos Any news about including this natively?

SaschaHeyer commented 1 week ago

Is there any update?

NicolaSpreafico commented 1 week ago

Any update on this?

melinath commented 1 week ago

Hey folks, this ticket has been forwarded to engineering for resolution. Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

nicola-spreafico commented 1 week ago

In the meantime, here's a workaround using gcloud to deploy and undeploy the index.

resource "google_vertex_ai_index" "my_index" {
  ...
}

resource "google_vertex_ai_index_endpoint" "my_endpoint" {
  ...
}

module "gcloud_ai_index_endpoints_deploy_index" {
  source  = "terraform-google-modules/gcloud/google"
  version = "~> 3.0"

  create_cmd_body  = "ai index-endpoints deploy-index ${google_vertex_ai_index_endpoint.my_endpoint.id} --deployed-index-id=deployed_index --display-name=deployed_index --index=${google_vertex_ai_index.my_index.id} --project=${module.project_services.project_id} --region=${var.region}"
  destroy_cmd_body = "ai index-endpoints undeploy-index ${google_vertex_ai_index_endpoint.my_endpoint.id} --deployed-index-id=deployed_index --project=${module.project_services.project_id} --region=${var.region}"
}

Thank you, Your workaround works in order to deploy the index, but I'm having an hard time retrieving the id of the deployed index, because being an asynchronous I should wait in any case after the gcloud command and then retrieve the final Id.

I hope the resource will be supported soon on the terraform provider so it will go smooth in terms of integrations

slocoro commented 2 days ago

Would also be interested in gcloud_ai_index_endpoints_deploy_index

hashicorp / terraform-provider-google