Please allow us to omit the .json extension

martinthurn commented 5 years ago

The Spring Cloud Config standard (https://cloud.spring.io/spring-cloud-static/spring-cloud-config/2.2.0.M3/reference/html/) does NOT require the .json extension. I ask that your module be modified to allow a url format that has no dot-extension.

amenezes commented 5 years ago

@martinthurn for now can you try fix this issue setting, for example:

from config import spring

c = spring.ConfigClient(
        app_name='myapp',
        url="{address}/{branch}/{profile}-{app_name}.json"
    )
c.url
# output: 'http://localhost:8888/master/development-myapp.json'
c.url = c.url[:-5] # << split .json extension
print(c.url)
c.get_config()

rodrigorodrigues commented 4 years ago

Hi @amenezes,

I'm trying to use your solution to remove .json and worked but I have different profiles and the get field using config_client.config['configuration']['jwt']['base64-secret'] is not working.

Could you please take a look when have a chance.

Following piece of code

    config_client = spring.ConfigClient(
    app_name=app_name,
    url="{address}/{app_name}/{profile}.json",
    profile=profile,
    branch=None,
    address=address
    )
    config_client.url = config_client.url[:-5]
    config_client.get_config(headers={'X-Encrypt-Key': app.config['X_ENCRYPT_KEY']})

    log.debug(config_client.config)

    app.config['JWT_SECRET_KEY'] = base64.b64decode(config_client.config['configuration']['jwt']['base64-secret']) #Error in this line

Response Config Client

2020-01-01 23:56:47,974 - __main__ - DEBUG - {'name': 'python-service', 'profiles': ['prod'], 'label': None, 'version': None, 'state': None, 'propertySources': [{'name': 'file:config/application.yml (document #2)', 'source': {'spring.profiles': 'prod', 'spring.sleuth.sampler.probability': 0.1, 'configuration.jwt.base64-secret': '', 'configuration.jwt.keystore': '${KEYSTORE:}', 'configuration.jwt.keystoreAlias': '${KEYSTORE_ALIAS:}', 'configuration.jwt.keystorePassword': '${KEYSTORE_PASSWORD:}', 'security.oauth2.client.accessTokenUri': '${ACCESS_TOKEN_URI:https://spendingbetter.com/auth/oauth/token}', 'security.oauth2.client.userAuthorizationUri': '${AUTHORIZATION_URL:https://spendingbetter.com/auth/oauth/authorize}', 'security.oauth2.client.clientId': '${OAUTH_CLIENT_ID:client}', 'security.oauth2.client.clientSecret': '${OAUTH_CLIENT_SECRET:secret}', 'security.oauth2.resource.jwt.key-store': '${configuration.jwt.keystore:}', 'security.oauth2.resource.jwt.key-store-password': '${configuration.jwt.keystorePassword:}', 'security.oauth2.resource.jwt.key-alias': '${configuration.jwt.keystoreAlias:}', 'security.oauth2.resource.user-info-uri': '${USER_INFO_URI:https://spendingbetter.com/auth/api/authenticatedUser}'}}]}

Thanks

amenezes commented 4 years ago

@rodrigorodrigues,

Based on code snippet shared could you share the output from config_client.config or the config_client.url. The output it's a json or a python dictionary {}.

rodrigorodrigues commented 4 years ago

Hi @amenezes the output for config_client.config is in the bottom of my comment, thanks.

amenezes commented 4 years ago

@rodrigorodrigues,

I adjusted the log output to the format of a dict.

{
    "name": "python-service",
    "profiles": ["prod"],
    "label": None,
    "version": None,
    "state": None,
    "propertySources": [{
        "name": "file:config/application.yml (document #2)",
        "source": {
            "spring.profiles": "prod",
            "spring.sleuth.sampler.probability": 0.1,
            "configuration.jwt.base64-secret": "",
            "configuration.jwt.keystore": "${KEYSTORE:}",
            "configuration.jwt.keystoreAlias": "${KEYSTORE_ALIAS:}",
            "configuration.jwt.keystorePassword": "${KEYSTORE_PASSWORD:}",
            "security.oauth2.client.accessTokenUri": "${ACCESS_TOKEN_URI:https://spendingbetter.com/auth/oauth/token}",
            "security.oauth2.client.userAuthorizationUri": "${AUTHORIZATION_URL:https://spendingbetter.com/auth/oauth/authorize}",
            "security.oauth2.client.clientId": "${OAUTH_CLIENT_ID:client}",
            "security.oauth2.client.clientSecret": "${OAUTH_CLIENT_SECRET:secret}",
            "security.oauth2.resource.jwt.key-store": "${configuration.jwt.keystore:}",
            "security.oauth2.resource.jwt.key-store-password": "${configuration.jwt.keystorePassword:}",
            "security.oauth2.resource.jwt.key-alias": "${configuration.jwt.keystoreAlias:}",
            "security.oauth2.resource.user-info-uri": "${USER_INFO_URI:https://spendingbetter.com/auth/api/authenticatedUser}"
        }
    }]
}

Can you access the configuration.jwt.base64-secret as example below:

option 1: using config property

config_client.config['propertySources'][0]['source']['configuration.jwt.base64-secret']

option 2: using get_attribute method.

config_client.get_attribute('propertySources.0.source')['configuration.jwt.base64-secret']
# or
config_client.get_attribute('propertySources.0.source').get('configuration.jwt.base64-secret')

amenezes commented 4 years ago

@rodrigorodrigues and @martinthurn,

Seems Spring Cloud ConfigServer not load configuration profile properly without the .json extension in the URL.

This could be easily verified if you do a request to server with the extension and make another without the extension.

I built a simple docker container with the cloud server that can you use to test, for example:

# up spring-cloud-configserver instance
docker run -d --rm -p 8888:8888 --name config-server amenezes/spring-cloud-configserver

Making a request with .json extension:

curl http://localhost:8888/master/simpleweb000-prod.json
# Formatted Output
{
    "configuration": {
        "jwt": {
            "base64-secret": "base64_secret_test",
            "keystore": "keystore_test",
            "keystoreAlias": "keystore_alias_test",
            "keystorePassword\"": "keystore_password_test"
        }
    },
    "health": {
        "config": {
            "enabled": false
        }
    },
    "info": {
        "app": {
            "description": "pws simpleweb000",
            "name": "simpleweb000"
        }
    },
    "security": {
        "oauth2": {
            "client": {
                "accessTokenUri\"": "https://spendingbetter.com/auth/oauth/token",
                "clientId\"": "client",
                "clientSecret\"": "secret",
                "userAuthorizationUri\"": "https://spendingbetter.com/auth/oauth/authorize"
            },
            "resource": {
                "jwt": {
                    "key-alias": "keystore_alias_test",
                    "key-store": "keystore_test",
                    "key-store-password": "jwt_keystorePassword_test"
                },
                "user-info-uri": "https://spendingbetter.com/auth/api/authenticatedUser"
            }
        }
    },
    "server": {
        "port": 8080
    },
    "spring": {
        "cloud": {
            "consul": {
                "host": "discovery",
                "port": 8500
            }
        },
        "sleuth": {
            "sampler": {
                "probability": 0.1
            }
        }
    }
}

Making a request without .json extension:

curl http://localhost:8888/master/simpleweb000-prod
# Formatted Output
{
    "name": "master",
    "profiles": ["simpleweb000-prod"],
    "label": null,
    "version": "15aa98080eab966f2a2efe6287af69edd5eee8f2",
    "state": null,
    "propertySources": [{
        "name": "https://github.com/amenezes/spring_config.git/application.yml",
        "source": {
            "health.config.enabled": false,
            "spring.cloud.consul.host": "discovery",
            "spring.cloud.consul.port": 8500
        }
    }]
}

So in this way Rodrigo, could you access your configuration in on the options below: (for example to access configuration.jwt.base64-secret:

# option 1
config_client.get_attribute('configuration.jwt.base64-secret')

# option 2
config_client.config['configuration']['jwt']['base64-secret']

# option 3
config_client.config.get('configuration').get('jwt').get('base64-secret')

Hi @amenezes,

I'm trying to use your solution to remove .json and worked but I have different profiles and the get field using config_client.config['configuration']['jwt']['base64-secret'] is not working.

Could you please take a look when have a chance.

Following piece of code
    config_client = spring.ConfigClient(
  app_name=app_name,
  url="{address}/{app_name}/{profile}.json",
  profile=profile,
  branch=None,
  address=address
    )
    config_client.url = config_client.url[:-5]
    config_client.get_config(headers={'X-Encrypt-Key': app.config['X_ENCRYPT_KEY']})

    log.debug(config_client.config)

    app.config['JWT_SECRET_KEY'] = base64.b64decode(config_client.config['configuration']['jwt']['base64-secret']) #Error in this line

amenezes commented 4 years ago

Issue resolved in version 0.6.0.

cdbennett commented 2 years ago

I'm not sure what version of the Spring Cloud Config Server was previously tested, but in my experience, you do NOT need to provide the .json extension in the requests to it.

Here is a simple test case:

Open in one shell:

$ docker run -it --rm -p 8888:8888 hyness/spring-cloud-config-server:3.1.0-jre17 --spring.cloud.config.server.git.uri=https://github.com/spring-cloud-samples/config-repo

Then in another shell:

$ curl -s localhost:8888/foo/dev | python3 -m json.tool
{
    "name": "foo",
    "profiles": [
        "dev"
    ],
    "label": null,
    "version": "bb51f4173258ae3481c61b95b503c13862ccfba7",
...
}

I suggest we omit the .json extension from the request path. It actually causes confusion and incorrect data in the response since the ".json" is included in the profile name in the response ("dev.json" is not the profile name, only "dev"):

$ curl -s localhost:8888/foo/dev.json | python3 -m json.tool
{
    "name": "foo",
    "profiles": [
        "dev.json"
    ],
...}

cdbennett commented 2 years ago

If we did need to tell the configuration server what content type to return the response in (e.g. XML or Java properties? but Spring Cloud Config Server does not seem to support anything other than JSON), then the right way would be to add a request header like "Accept: application/xml", instead of appending an extension suffix to the URL path.

amenezes commented 2 years ago

@cdbennett,

I used this reference for the implementation: https://docs.spring.io/spring-cloud-config/docs/current/reference/html/#_quick_start

I tried use config-client with the docker command that you shared and indeed the extension did not modify the answer.

But the behavior can be observed using the image present in the docker-compose file in the root directory or running the following command:

docker run --rm -p 8888:8888 --name config-server amenezes/spring-cloud-configserver

# curl -s http://localhost:8888/master/simpleweb000-development | python -m json.tool
{
    "name": "master",
    "profiles": [
        "simpleweb000-development"
    ],
    "label": null,
    "version": "b478bb5c9784bb2285c461892fab22361007e0c9",
    "state": null,
    "propertySources": [
        {
            "name": "https://github.com/amenezes/spring_config.git/application.yml",
            "source": {
                "health.config.enabled": false,
                "spring.cloud.consul.host": "discovery",
                "spring.cloud.consul.port": 8500
            }
        }
    ]
}

# curl -s http://localhost:8888/master/simpleweb000-development.json | python -m json.tool
{
    "health": {
        "config": {
            "enabled": false
        }
    },
    "info": {
        "app": {
            "description": "pws simpleweb000 - development profile",
            "name": "simpleweb000",
            "password": "123"
        }
    },
    "python": {
        "cache": {
            "timeout": 10,
            "type": "simple"
        }
    },
    "server": {
        "port": 8080
    },
    "spring": {
        "cloud": {
            "consul": {
                "host": "discovery",
                "port": 8500
            }
        }
    }
}

Can you try to reproduce this or use this repo - https://github.com/amenezes/spring_config - as source?

amenezes commented 2 years ago

If we did need to tell the configuration server what content type to return the response in (e.g. XML or Java properties? but Spring Cloud Config Server does not seem to support anything other than JSON), then the right way would be to add a request header like "Accept: application/xml", instead of appending an extension suffix to the URL path.

@cdbennett, the following link show how server use alternative formats like yaml/yml or properties

https://docs.spring.io/spring-cloud-config/docs/current/reference/html/#_serving_alternative_formats

rodrigorodrigues commented 2 years ago

Hi @amenezes,

If you follow the samples from the official documentation you can see the .json extension is not needed.

docker run -it -p 8888:8888 \ -e SPRING_CLOUD_CONFIG_SERVER_GIT_URI=https://github.com/spring-cloud-samples/config-repo \ hyness/spring-cloud-config-server

curl -s http://localhost:8889/foo-development,db/main | python -m json.tool
{
    "label": null,
    "name": "foo-development,db",
    "profiles": [
        "main"
    ],
    "propertySources": [
        {
            "name": "https://github.com/spring-cloud-samples/config-repo/foo-development.properties",
            "source": {
                "bar": "spam",
                "foo": "from foo development"
            }
        },
        {
            "name": "https://github.com/spring-cloud-samples/config-repo/Config resource 'file [/tmp/config-repo-1875980609665838541/application.yml' via location '' (document #0)",
            "source": {
                "eureka.client.serviceUrl.defaultZone": "http://localhost:8761/eureka/",
                "foo": "baz",
                "info.description": "Spring Cloud Samples",
                "info.url": "https://github.com/spring-cloud-samples"
            }
        }
    ],
    "state": null,
    "version": "bb51f4173258ae3481c61b95b503c13862ccfba7"
}

amenezes commented 2 years ago

@rodrigorodrigues and @cdbennett,

The official documenation really left me confused 😵‍💫

The HTTP service has resources in the following form:

/{application}/{profile}[/{label}]
/{application}-{profile}.yml
/{label}/{application}-{profile}.yml
/{application}-{profile}.properties
/{label}/{application}-{profile}.properties

For example:

curl localhost:8888/foo/development
curl localhost:8888/foo/development/master
curl localhost:8888/foo/development,db/master
curl localhost:8888/foo-development.yml
curl localhost:8888/foo-db.properties
curl localhost:8888/master/foo-db.properties

Let's run some tests:

/{application}/{profile}[/{label}]

curl -s http://localhost:8888/foo/development # OK
# but with config in the propertySources
# configuraiton apart in the list: application.yml, foo.properties and foo-development.properties

curl -s http://localhost:8888/foo/development/main # OK
# but with config in the propertySources
# configuraiton apart in the list: application.yml, foo.properties and foo-development.properties

/{application}-{profile}.json

curl -s http://localhost:8888/foo-development # Not Found

curl -s http://localhost:8888/foo-development.json # OK
# final_config = application.yml + foo.properties + foo-development.properties
# all configuraiton was merged in the response
# this is the currently expected configuration, i.e. the configserver merges the files and returns the configuration

/{label}/{application}-{profile}.json

curl -s http://localhost:8888/main/foo-development.json # "OK" - but the response have only the application.yml data.

/{application}-{profile}.json

curl -s http://localhost:8888/foo-development.json # OK
# final_config = application.yml + foo.properties + foo-development.properties
# all configuraiton was merged in the response
# this is the currently expected configuration, i.e. the configserver merges the files and returns the configuration

/{label}/{application}-{profile}.json

curl -s http://localhost:8888/main/foo-development.json # "OK" - but the response have only the application.yml data.

So considering the output in which the data was returned in a list of the property propertySources would be sufficient or correct for your use case?

Do you really want the configuration to be separate so that the use of a config is done at run time? For example:

curl -s http://localhost:8888/foo/development
{
    "label": null,
    "name": "foo",
    "profiles": [
        "development"
    ],
    "propertySources": [
        {
            "name": "https://github.com/spring-cloud-samples/config-repo/foo-development.properties",
            "source": {
                "bar": "spam",
                "foo": "from foo development"
            }
        },
        {
            "name": "https://github.com/spring-cloud-samples/config-repo/foo.properties",
            "source": {
                "democonfigclient.message": "hello spring io",
                "foo": "from foo props"
            }
        },
        {
            "name": "https://github.com/spring-cloud-samples/config-repo/Config resource 'file [/tmp/config-repo-15758515442355975669/application.yml' via location '' (document #0)",
            "source": {
                "eureka.client.serviceUrl.defaultZone": "http://localhost:8761/eureka/",
                "foo": "baz",
                "info.description": "Spring Cloud Samples",
                "info.url": "https://github.com/spring-cloud-samples"
            }
        }
    ],
    "state": null,
    "version": "bb51f4173258ae3481c61b95b503c13862ccfba7"
}

foo property it's available with values: foo=from foo development, foo=from foo props and foo=baz

For my use cases I consider only the value of the property specified in the profile, so if the profile=development foo will be from foo development. As you can see:

curl -s http://localhost:8888/foo-development.json
# application.yml + foo.properties + foo-development.properties
{
    "bar": "spam",
    "democonfigclient": {
        "message": "hello spring io"
    },
    "eureka": {
        "client": {
            "serviceUrl": {
                "defaultZone": "http://localhost:8761/eureka/"
            }
        }
    },
    "foo": "from foo development",
    "info": {
        "description": "Spring Cloud Samples",
        "url": "https://github.com/spring-cloud-samples"
    }
}

# profile=default
curl -s http://localhost:8888/foo-default.json
# application.yml + foo.properties
{
    "democonfigclient": {
        "message": "hello spring io"
    },
    "eureka": {
        "client": {
            "serviceUrl": {
                "defaultZone": "http://localhost:8761/eureka/"
            }
        }
    },
    "foo": "from foo props",
    "info": {
        "description": "Spring Cloud Samples",
        "url": "https://github.com/spring-cloud-samples"
    }
}

amenezes commented 2 years ago

@rodrigorodrigues, @cdbennett and @rayanth,

I think this confusion it's a known by the spring-cloud-config server. See that issue for more details.

Anyway I will follow up on this issue and if there is any resolution or outline guidance I inform you.

Maybe, for now I suggest adapt your configuration files to pattern:

application.yml # common configuration
myapp-development.yml # development profile
myapp-test.yml # test profile

from config import ConfigClient

c  = ConfigClient(app_name='myapp')
c.get_config() # by default use development profile
print(c)

c  = ConfigClient(app_name='myapp', profile='test')
c.get_config() # use test profile
print(c)

Or use the method get_file to retrieve raw file.

https://config-client.amenezes.net/docs/client/1.-standard-client/#retrieving-plain-files

amenezes commented 2 years ago

@rodrigorodrigues, @cdbennett and @rayanth,

Could you check if config-client version 1.0.0b1 fixes the behavior related in that issue?

I changed the request method to get configuration files from config-server and now merge files occur in the client. I made some tests using config-server available in the project as well as spring-cloud-config-server and I succeeded.

For install this beta version could you use:

pip install 'config-client[cli]==1.0.0b1'

This option provides a CLI to make requests from server as CURL, but showing only relevant parts. For example:

# foo # application name
# -b label/branch
# -p profiles active
# --all # show all configuration
# -v # verbose mode
python -m config client foo -b main -p dev,docker --all -v

If you prefer could you use REPL or even a test application but checking final configuration stored in the property config.

Can any of you verify that this doesn't break anything when used in CloudFoundry?

amenezes / config-client

Please allow us to omit the .json extension #18