search

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.85k stars 1.83k forks source link

[Feature Request] Upgrade google-cloud-storage to v2 #16156

Open cwperks opened 1 month ago

cwperks commented 1 month ago

Is your feature request related to a problem? Please describe

Coming from conversation on https://github.com/opensearch-project/OpenSearch/pull/16072

The repository-gcs plugin currently depends on v1 of the google-cloud-storage dependencies. Dependabot has been creating automatic updates which consistently fail precommit and tests and cannot be merged without further modifications.

I have tried to upgrade this dependency, but consistently receive errors on GoogleCloudStorageBlobContainerRetriesTests.testWriteLargeBlob post-upgrade which uses the resumable upload feature of google cloud storage.

The test can be run using ./gradlew :plugins:repository-gcs:test --tests GoogleCloudStorageBlobContainerRetriesTests.testWriteLargeBlob -i. When running that test, I have to add an image: tag to test/fixtures/gcs-fixture/docker-compose.yml in all services in order to bring up the containers successfully.

Describe the solution you'd like

Update to v2 for com.google.cloud:google-cloud-storage dependency.

Related component

Plugins

Describe alternatives you've considered

No response

Additional context

There were lines added to dependabot.yml to prevent dependabot from making PRs on a major version update for this dependency. After upgrading to v2, make sure to remove these lines from dependabot.yml.

Yashovardhan08 commented 1 month ago

@cwperks @sandeshkr419, I am interested, looking if someone could help me get started and guide me if I get stuck.

cwperks commented 1 month ago

@Yashovardhan08 Certainly! When I tried to make an update it didn't seem like a straightforward change, but I didn't have time to dig deeper into the tests within the repository-gcs plugin and any changes to resumable upload.

There's another outdated dependency that may be more straightforward which I would suggest starting with. In the discovery-gce plugin, google-api-client is currently using v1 and can be upgraded to the latest v2 version of the dependency. I would recommend starting with the update in discovery-gce plugin and then moving to this issue which is known to require test updates.

You can find me on the public slack or use Github for communication.

dblock commented 1 month ago

[Catch All Triage - 1, 2]

Yashovardhan08 commented 1 month ago

@sandeshkr419, @cwperks Please review the PR here for com.google.api-client upgrade from v1 to v2.