p-config-server is now p.config-server

[rayanth]

in Spring Cloud Services (SCS v3.0 and later, the p-config-server entry in VCAP_SERVICES has changed to p.config-server. This causes numerous failures in config-client, as it is explicitly looking for p-config-server.

The documentation does not mention this, but the value can be overridden by setting os.environ["VCAP_SERVICE_PREFIX"] = "p.config-server".

However, in function __attrs_post_init__ of cfenv.py, there is still a literal "p-config-server" that does not get overridden by the environment variable.

[amenezes]

Hello @rayanth,

In fact, I did not document the options available for configuration with environment variables for the CF module. I'll update that.

There is an error when you try to use the CF module, with this excerpt?

Example:

from config.cf import CF
from config.cfenv import CFenv

cf = CF(cfenv=CFenv(vcap_service_prefix="p.config-server"))
cf.get_config()

notice: that this option only work on version >= 0.11.1, as related in the issue #37

Updated documentation for CF module :heavy_check_mark:

[rayanth]

My use-case was actually with the Flask extension module... it does not throw an error in the __attrs_post_init__ section that I mentioned, it was just something I noticed while debugging the original issue. It may be that this is not executed, i don't really know how it works there :) (attrs is not a library i'm familiar with)

Prior to writing the issue (and prior to your documentation fix), the following code is what I ran to get everything to work, on a non-cloud foundry machine in python shell. it does not present any errors in the logger.

import os
os.environ["VCAP_SERVICE_PREFIX"] = "p.config-server"
os.environ["BRANCH"] = 'master'
os.environ["PROFILE"] = 'test'
os.environ["APP_NAME"] = 'config-test'
os.environ["CONFIGSERVER_ADDRESS"] = '(redacted)'  # copy-pasted from actual cloud foundry config
os.environ["VCAP_SERVICES"] = '(redacted)'  # copy-pasted from actual cloud foundry config
os.environ["VCAP_APPLICATION"] = '(redacted)'  # copy-pasted from actual cloud foundry config
import logging
from flask import Flask, jsonify
from config.spring import ConfigClient
from config.ext.flask import FlaskConfig
from config.auth import OAuth2
import json
logging.basicConfig(level=logging.INFO)
app = Flask(__name__)
FlaskConfig(app, ConfigClient(
    app_name='lscc',
    oauth2=OAuth2(
        access_token_uri=json.loads(os.environ["VCAP_SERVICES"])["p.config-server"][0]["credentials"]["access_token_uri"],
        client_id=json.loads(os.environ["VCAP_SERVICES"])["p.config-server"][0]["credentials"]["client_id"],
        client_secret=json.loads(os.environ["VCAP_SERVICES"])["p.config-server"][0]["credentials"]["client_secret"]
        )
    )
)

(at this point, app.config contains the values from the git repo)

I am now facing a different issue - when I deploy to Cloud Foundry, the code does not work. my organization uses custom, protected certificates for the gitlab servers, not the default certs; i believe the failure stems from not being able to pass in this custom cert to config-client's requests.get to the Git server in http.py, so it cannot verify the SSL cert and fails on SSL validation.

This request originates from flask.py in class FlaskConfig, self._client.get_config() which does not seem to provide a method for passing in a custom cert path. Am I overlooking something?

[amenezes]

@rayanth thanks for share more details.

Did you right. I forgot to put someway to pass *args and **kwargs to get_config method for extensions Flask and AIOHTTP 😅.

For now use the standard client or CF and in the call of method get_config use the parameters as if you were making an HTTP with requests. For example:

from config.spring import ConfigClient

c = ConfigClient(app_name='simpleweb000', address='https://localhost:2016', fail_fast=False)
c.get_config(verify=False)

from config.cf import CF
from config.cfenv import CFenv

cf = CF(cfenv=CFenv(vcap_service_prefix="p.config-server"))
cf.get_config(verify=False)

For the example that you shared, did you try use CF class?

• http://config-client.amenezes.net/docs/integrations/1.-flask/#using-the-cloudfoundry-client

[rayanth]

No, it's still failing after I deploy to my CF server with just this code, though it works fine on my local machine:

cf = CF(cfenv=CFenv(vcap_service_prefix="p.config-server"))
cf.get_config(verify=False)

FlaskConfig(app, cf)

Traceback (provided below with PII redacted) shows it failing in cf.get_config().

Doing a visual trace in the code via the traceback, I see a couple of possible trouble spots:

• in auth.py's OAuth2.configure() it does not pass the kwargs to OAuth2.request_token() ... and request_token also does not include them in the http.post request.
• in spring.py, which is where the above OAuth2.configure is called, it doesn't pass anything at all to the self.oauth2.configure() call, so even if OAuth2.configure accepted the kwargs, they aren't being passed in.

This explains the failure of verify=False as well as specifically telling it to use /etc/ssl/certs/ca-certificates.crt, which I did verify contains my organization's G3 certificate (and many many others).

I'll try to implement a workaround by re-implementing the library as a custom module and passing along the kwargs as mentioned above and report back on how it goes.

Traceback follows:

   2021-12-16T13:24:18.90-0800 [APP/PROC/WEB/0] ERR Traceback (most recent call last):
   2021-12-16T13:24:18.90-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 699, in urlopen
   2021-12-16T13:24:18.90-0800 [APP/PROC/WEB/0] ERR     httplib_response = self._make_request(
   2021-12-16T13:24:18.90-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 382, in _make_request
   2021-12-16T13:24:18.90-0800 [APP/PROC/WEB/0] ERR     self._validate_conn(conn)
   2021-12-16T13:24:18.90-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR     conn.connect()
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/site-packages/urllib3/connection.py", line 416, in connect
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR     self.sock = ssl_wrap_socket(
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR     ssl_sock = _ssl_wrap_socket_impl(
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR     return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/ssl.py", line 512, in wrap_socket
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR     return self.sslsocket_class._create(
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/ssl.py", line 1070, in _create
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR     self.do_handshake()
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/ssl.py", line 1341, in do_handshake
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR     self._sslobj.do_handshake()
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR During handling of the above exception, another exception occurred:
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR Traceback (most recent call last):
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/site-packages/requests/adapters.py", line 439, in send
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR     resp = conn.urlopen(
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 755, in urlopen
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR     retries = retries.increment(
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 574, in increment
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR     raise MaxRetryError(_pool, url, error or ResponseError(cause))
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='[REDACTED]', port=443): Max retries exceeded with url: /oauth/token (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR During handling of the above exception, another exception occurred:
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR Traceback (most recent call last):
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/app/runserver.py", line 24, in <module>
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR     cf.get_config(verify=False)
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/site-packages/config/cf.py", line 45, in get_config
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR     self.client.get_config(**kwargs)
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/site-packages/config/spring.py", line 97, in get_config
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR     kwargs = self._configure_oauth2(**kwargs)
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/site-packages/config/spring.py", line 111, in _configure_oauth2
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR     self.oauth2.configure()
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/site-packages/config/auth.py", line 49, in configure
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR     self.request_token(client_auth, data)
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/site-packages/config/auth.py", line 38, in request_token
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR     response = http.post(self.access_token_uri, auth=client_auth, data=data)
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/site-packages/config/http.py", line 8, in _req
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR     response: requests.Response = fnc(uri, **kwargs)
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/site-packages/requests/api.py", line 117, in post
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR     return request('post', url, data=data, json=json, **kwargs)
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/site-packages/requests/api.py", line 61, in request
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR     return session.request(method=method, url=url, **kwargs)
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/site-packages/requests/sessions.py", line 542, in request
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR     resp = self.send(prep, **send_kwargs)
   2021-12-16T13:24:18.91-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/site-packages/requests/sessions.py", line 655, in send
   2021-12-16T13:24:18.92-0800 [APP/PROC/WEB/0] ERR     r = adapter.send(request, **kwargs)
   2021-12-16T13:24:18.92-0800 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.10/site-packages/requests/adapters.py", line 514, in send
   2021-12-16T13:24:18.92-0800 [APP/PROC/WEB/0] ERR     raise SSLError(e, request=request)
   2021-12-16T13:24:18.92-0800 [APP/PROC/WEB/0] ERR requests.exceptions.SSLError: HTTPSConnectionPool(host='[REDACTED]', port=443): Max retries exceeded with url: /oauth/token (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))

[rayanth]

forked the library to my own module, added the passing of kwargs in the locations noted below, and it now works both with verify=False and with verify='/etc/ssl/certs/ca-certificates.crt'

spring.py, line 111 -- self.oauth2.configure(**kwargs) auth.py, line 36 -- def request_token(self, client_auth: HTTPBasicAuth, data: dict, **kwargs) -> None: auth.py, line 38 -- response = http.post(self.access_token_uri, auth=client_auth, data=data, **kwargs) auth.py, line 46 -- def configure(self, **kwargs) auth.py, line 49 -- self.request_token(client_auth, data, **kwargs)

Final code that worked on-server (bypassing FlaskConfig as that would need even more fixes):

cf = CF(cfenv=CFenv(vcap_service_prefix="p.config-server"))
cf.get_config(verify='/etc/ssl/certs/ca-certificates.crt')

app.config.update(cf.config)

[amenezes]

Great job @rayanth :clap:

I'm will made the adjustments and release a new version with the fix.

[amenezes]

@rayanth,

Could you test config-client alpha release with the possible fix?

pip install config-client==0.13.1a1

I hope the excerpt below works smoothly or with minimal adjustments in the parameters only.

CF version

import logging

from config import CF, CFenv

logging.basicConfig(level=logging.DEBUG)

cf = CF(cfenv=CFenv(vcap_service_prefix="p.config-server")) cf.get_config(verify='/etc/ssl/certs/ca-certificates.crt')

> FlaskConfig version
```python
import logging

from flask import Flask
from config.cf import CF
from config.ext.flask import FlaskConfig

logging.basicConfig(level=logging.DEBUG)

app = Flask('test app')
FlaskConfig(
    app,
    CF(
        cfenv=CFenv(vcap_service_prefix="p.config-server")
    ),
    verify='/etc/ssl/certs/ca-certificates.crt'
)

[rayanth]

unfortunately, I will not be able to test, I had to drop usage of the library entirely due to the enforced .json file endings; .json is not one of the supported filetypes by config-server by default, or in my org's setup.

[amenezes]

Sad to hear that.

@rayanth, what format is available?

I had already worked on a way not to force the .json extension, issue #18, but this led to strange behavior when requesting the settings. In the root directory there's a docker-compose.yml file that provides an instance of config-server. For example:

If you made a request with json extension will get:

• http://localhost:8888/master/simpleweb000-development.json

{
    "health": {
        "config": {
            "enabled": false
        }
    },
    "info": {
        "app": {
            "description": "pws simpleweb000 - development profile",
            "name": "simpleweb000",
            "password": "123"
        }
    },
    "python": {
        "cache": {
            "timeout": 10,
            "type": "simple"
        }
    },
    "server": {
        "port": 8080
    },
    "spring": {
        "cloud": {
            "consul": {
                "host": "discovery",
                "port": 8500
            }
        }
    }
}

And if you made same request without .json extension will get:

• http://localhost:8888/master/simpleweb000-development

{
    "name": "master",
    "profiles": ["simpleweb000-development"],
    "label": null,
    "version": "b478bb5c9784bb2285c461892fab22361007e0c9",
    "state": null,
    "propertySources": [{
        "name": "https://github.com/amenezes/spring_config.git/application.yml",
        "source": {
            "health.config.enabled": false,
            "spring.cloud.consul.host": "discovery",
            "spring.cloud.consul.port": 8500
        }
    }]
}

Another simple option that might not require much effort would be to use the yaml format, like:

• http://localhost:8888/master/simpleweb000-development.yaml
• http://localhost:8888/master/simpleweb000-development.yml

health:
  config:
    enabled: false
info:
  app:
    description: pws simpleweb000 - development profile
    name: simpleweb000
    password: '123'
python:
  cache:
    timeout: 10
    type: simple
server:
  port: 8080
spring:
  cloud:
    consul:
      host: discovery
      port: 8500

At the time, if I'm not mistaken, I observed this documenation: https://cloud.spring.io/spring-cloud-static/spring-cloud-config/2.2.0.M3/reference/html/#_serving_alternative_formats

[amenezes]

@rayanth,

I released the version 0.13.1 with fixes to pass **kwargs to oauth2, FlaskConfig and AioHttpConfig.

If you want, we can continue to discuss more about another formats options to request configurtion from server, but for now I will close this issue.

Thanks for you help 😉