Add support for spring.config.import with S3 backend

celcius112 commented 3 years ago

Type: Feature

First of all, as a disclaimer, I am not sure whether this feature request should be in this repo or in https://github.com/spring-cloud/spring-cloud-config, as there is already a S3 implementation for the legacy bootstrap.

Similar to the parameter store and secrets manager configuration, it would be great if AWS S3 could be used as external configuration backend, such as:

spring:
  config:
    import: aws-s3:my-bucket

At the moment, spring-cloud-config does not seem to propose Spring Boot's spring.config.import property for S3. Since using spring.config.import with parameter-store-config and secrets-manager-config simultaneously with the legacy bootstrap with s3 does not seem to work out of the box (AwsSecretsManagerPropertySourceLocator and AwsSecretsManagerConfigDataLocationResolver are both created simultaneously), it is at the moment a blocking point for us for migrating to the new spring.config.import and correcting https://github.com/awspring/spring-cloud-aws/issues/159.

MatejNedic commented 3 years ago

Hey @celcius112 , S3 module currently doesn't support spring.config.import . Could you please explain in greater detail what would you expect from this feature to support?

celcius112 commented 3 years ago

Hello @MatejNedic,

At the moment, spring-cloud-config-server proposes a S3 module that retrieves .properties, .yml or .json files from a S3 bucket. Using the legacy bootstrap the properties are imported into the application environment.

I have implemented what spring-cloud-config-server proposes, but using Spring Boot's spring.config.import:

spring:
  config:
    import:
      - aws-s3:bucket-name-1;file1.yml
      - aws-s3:bucket-name-1;file2.yml

I don't think this module should have much more functionality than that, except maybe having a way to override the S3Client using the BootstrapContext.

I guess the usage would be similar to spring-cloud-aws's parameter-store and secrets-manager starters.

kunalvarpe commented 1 year ago

@maciejwalkowiak @MatejNedic, I would like to contribute here, I have done some changes in my local. Before pushing that I have some queries/doubts.

As per above comment the expectation is spring confing import should load properties from file.yml and file2.yml from the given buckets.

What if file has profile mentioned with it, in such situation should we respet the profile as well. e.g.

spring:
config:
 import:
  - aws-s3:bucket-name-1;file1-test.yml
  - aws-s3:bucket-name-1;file2.yml
  - aws-s3:bucket-name-2;file3-integration.yml

What would be other usecases here?

maciejwalkowiak commented 1 year ago

@kunalvarpe good question. It should be consistent with other property sources, for example - if you use spring.config.import with a filesystem location, does it respect profiles? It should work the same with S3 location.

kunalvarpe commented 1 year ago

Hi @maciejwalkowiak I have push the initial commit to the PR #849. Could you please let me know if I am doing the changes in correct direction, also I would be needed some more guidence on respecting the profiles support.

vishal94136 commented 6 months ago

Hello @MatejNedic,

At the moment, spring-cloud-config-server proposes a S3 module that retrieves .properties, .yml or .json files from a S3 bucket. Using the legacy bootstrap the properties are imported into the application environment.

I have implemented what spring-cloud-config-server proposes, but using Spring Boot's spring.config.import:
spring:
  config:
    import:
      - aws-s3:bucket-name-1;file1.yml
      - aws-s3:bucket-name-1;file2.yml
I don't think this module should have much more functionality than that, except maybe having a way to override the S3Client using the BootstrapContext.

I guess the usage would be similar to spring-cloud-aws's parameter-store and secrets-manager starters.

is it possible to have aws s3 config server without legacy bootstrap?

maciejwalkowiak commented 6 months ago

@vishal94136 we have S3 spring.config.import in progress here: https://github.com/awspring/spring-cloud-aws/pull/849. It will be included in 3.2.0. Spring Boot's "bootstrap" phase is not legacy - this is the new way of handling such things. What's legacy is using Spring Cloud Bootstrap 🙃

vishal94136 commented 6 months ago

@maciejwalkowiak thanks a lot for replying and sorry I sentenced it wrong I meant the spring cloud bootstrap.

awspring / spring-cloud-aws

Add support for spring.config.import with S3 backend #161