[DE200]: Need ability to set publicURL endpoints

[rpawlik]

As the title states, we need to ability to set different IPs (public, etc) on the various API endpoints.

[odyssey4me]

This is possible through the use of external-vips, although I haven't personally figured out how to use that yet. Perhaps someone can share an example?

This may also relate to https://github.com/rcbops/chef-cookbooks/issues/470 or https://github.com/rcbops/chef-cookbooks/issues/395

[DavidWittman]

@odyssey4me Setting external-vips in the environment works, but it also overrides the internal URLs as well because when you define external-vips it no longer load balances using haproxy/keepalived on the controllers. Plus it only works for HA configs.

[odyssey4me]

We're hoping to be able to override with a DNS entry instead - it provides more flexibility with the external access.

[mancdaz]

@rpawlik I need confirmation on exactly what you want to do here?

For example, in the glance cookbook we have the following attributes that are used to build the glance-api endpoint details that are used to register the endpoint:

default["glance"]["services"]["api"]["scheme"] = "http"
default["glance"]["services"]["api"]["network"] = "public"
default["glance"]["services"]["api"]["port"] = 9292
default["glance"]["services"]["api"]["path"] = "/v1"

This will discover the IP address for the machine interface on the relevant network (in this case the public network). This is then used to register all 3 of the glance endpoint types (admin/internal/public) with the URI that gets built.

Note: It's possible to override the IP that is used so that instead of discovering based on network, the provided IP gets used, by specifying a host value for the service attribute in your environment like this:

default["glance"]["services"]["api"]["host"] = "1.2.3.4"

Also note (@odyssey4me), this can take a hostname rather than an IP address, so you can use DNS entries rather than IP's. This is functionality that exists today, but it still would use the same details for all 3 endpoint types. I think this is what the problem is? You want to be able to specify each of the 3 separately?

So, we could add something like:

default["glance"]["services"]["admin-api"]["scheme"] = "http"
default["glance"]["services"]["admin-api"]["network"] = "management"
default["glance"]["services"]["admin-api"]["port"] = 9292
default["glance"]["services"]["admin-api"]["path"] = "/v1"

default["glance"]["services"]["internal-api"]["scheme"] = "http"
default["glance"]["services"]["internal-api"]["network"] = "management"
default["glance"]["services"]["internal-api"]["port"] = 9292
default["glance"]["services"]["internal-api"]["path"] = "/v1"

leaving the bare "api" as public, and edit the recipes to set the different endpoint types based on this info.

In the case of HA (again for eg glance), you currently have to set:

"override_attributes": {
    "vips": {
      "glance-api": "1.2.3.4",
    },

In the new case, you would set:

"override_attributes": {
    "vips": {
      "glance-api": "1.2.3.4",
      "glance-internal-api": "5.6.7.8",
      "glance-admin-api": "9.10.11.12",
    }, 

I think this would fulfil the requirements of this request, but please do confirm at your earliest opportunity.

[mancdaz]

Actually for this to work, you'd need to make your api services listen on all IP's so that it can receive requests on different endpoint IPs. At the moment, the cookbooks by default make the api services listen on a specific IP but this is overridable.

[odyssey4me]

Hmm, I think the default["glance"]["services"]["api"]["host"] override with a DNS name would work for us. The only issue I see if that our public endpoints are HTTPS, whereas the internal & admin endpoints are not. This does imply that perhaps there should be a separation of the default["glance"]["services"]["api"] into ["public-api"] and ["internal-api"] at least, with the admin-api being served on the same address as the internal-api.

Something else to consider is that we may want to serve all public endpoints via HTTPS on port 443. This makes access from behind corporate firewalls and the like much simpler. The public endpoints would therefore have to be bound to specific public IP's and would not be able to be listening on all bound IP's. It's better for security as well that the services are only listening on specific addresses where possible.

[DavidWittman]

@mancdaz: There are a number of different issues here, so I'll begin by describing the most common use case which we are seeing so that you get a better idea of what we're trying to do. Apologies in advance for the wall of text. :horse:

Story

We have a growing number of customers that want to be able to access OpenStack services externally to leverage tools such as ServiceMesh, RightScale, etc. For this to work, we need to redefine specific service endpoints (usually the publicURLs) while leaving the rest of the endpoints (internal and admin urls) untouched. There are currently no easy ways to do this in the cookbooks.

Implementation

HA Controllers

In the case of HA controllers, external-vips can be set, which allow you to define an external network for your OpenStack endpoints. However, from what I have gathered, this setting works under the assumption that these external VIPs will be handling all of the load balancing for that particular service, regardless of the endpoint type. While that may be a fair assumption for the public endpoints, the internal and admin endpoints are also configured without heartbeat/haproxy, and as a result end up using the same external VIP for the endpoint. So now all internal communications between OpenStack services must go through the external endpoint, which is suboptimal (and may be impossible depending on the network configuration).

Single Controller

With a single controller deployment, it follows an almost entirely different code path for figuring out the service endpoints. Most notably, the endpoints are not removed and re-created during each chef-client run on the controller. This makes it easy to just add new endpoints in a different region for those services without worry of them being unwittingly removed later. While this isn't necessarily the best way to go about it, it works.

Also, as you mentioned earlier, the search mechanism in single controller deployments can also be modified by overriding the node[service]['services'][type] hash. With the recent changes to get_bind_endpoint in rcbops-cookbooks/osops-utils#107, we should be able to easily override the URI to for these services whatever we please. So your suggestion to split out the API services hashes into api, admin-api, and internal-api would probably help in this case.

Proposal

There are a few problems with the way this works right now:

1. Depending on whether the configuration is HA or not, two entirely different attribute sets need to be changed in order to achieve the same result
2. It's difficult to override only one endpoint type while leaving the others alone
3. It doesn't make sense. If I want to define a public endpoint, I have to override four different things and just cross my fingers and hope it works.

I would like to suggest the introduction of attributes which allow the user to short-circuit the whole endpoint search process. For example, setting node['glance']['services']['publicURL'] = 'https://api.example.com' sets the public endpoint for Glance. No looking for networks local to the box, no mucking around with VIPs, just set the endpoint and be done. I have a patch which implements some of this functionality, but I'm hoping one of you guys with more experience on the internals of these cookbooks can step in and make it happen. Or just tell me I'm delusional.

[odyssey4me]

+1 on this proposal - providing a mechanism to override the Public URL with a string value, completely bypassing the search mechanism, would be perfect.

On 6 July 2013 01:55, David Wittman notifications@github.com wrote:

@mancdaz https://github.com/mancdaz: There are a number of different issues here, so I'll begin by describing the most common use case which we are seeing so that you get a better idea of what we're trying to do. Apologies in advance for the wall of text. [image: :horse:] Story

We have a growing number of customers that want to be able to access OpenStack services externally to leverage tools such as ServiceMesh, RightScale, etc. For this to work, we need to redefine specific service endpoints (usually the publicURLs) while leaving the rest of the endpoints (internal and admin urls) untouched. There are currently no easy ways to do this in the cookbooks. Implementation HA Controllers

In the case of HA controllers, external-vips can be set, which allow you to define an external network for your OpenStack endpoints. However, from what I have gathered, this setting works under the assumption that these external VIPs will be handling all of the load balancing for that particular service, regardless of the endpoint type. While that may be a fair assumption for the public endpoints, the internal and admin endpoints are also configured without heartbeat/haproxy, and as a result end up using the same external VIP for the endpoint. So now all internal communications between OpenStack services must go through the external endpoint, which is suboptimal (and may be impossible depending on the network configuration). Single Controller

With a single controller deployment, it follows an almost entirely different code path for figuring out the service endpoints. Most notably, the endpoints are not removed and re-created during each chef-client run on the controller. This makes it easy to just add new endpoints in a different region for those services without worry of them being unwittingly removed later. While this isn't necessarily the best way to go about it, it works.

Also, as you mentioned earlier, the search mechanism in single controller deployments can also be modified by overriding the node[service]['services'][type] hash. With the recent changes to get_bind_endpoint in rcbops-cookbooks/osops-utils#107, we should be able to easily override the URI to for these services whatever we please. So your suggestion to split out the API services hashes into api, admin-api, and internal-api would probably help in this case. Proposal

There are a few problems with the way this works right now:

1. Depending on whether the configuration is HA or not, two entirely different attributes need to be changed in order to achieve the same result
2. It's difficult to override only one endpoint type while leaving the others alone
3. It doesn't make sense. If I want to define a public endpoint, I have to override four different things and just cross my fingers and hope it works.

I would like to suggest the introduction of attributes which allow the user to short-circuit the whole endpoint search process. For example, setting node['glance']['services']['publicURL'] = 'https://api.example.com ' sets the public endpoint for Glance. No looking for networks local to the box, no mucking around with VIPs, just set the endpoint and be done. I have a patch which implements some of this functionality, but I'm hoping one of you guys with more experience on the internals of these cookbooks can step in and make it happen. Or just tell me I'm delusional.

— Reply to this email directly or view it on GitHubhttps://github.com/rcbops/chef-cookbooks/issues/463#issuecomment-20544286 .

Jesse Pretorius mobile: +27 83 680 5492 email: jesse.pretorius@gmail.com skype: jesse.pretorius

[mancdaz]

@DavidWittman

OK so I added the ability to separately specify the admin/internal/public endpoint types to the glance cookbook, thus:

https://github.com/rcbops-cookbooks/glance/blob/grizzly/attributes/default.rb#L24-L37

In order to forcefully override an entire uri for any of those endpoint types, you simply need to pass a value for the ['uri'] for that endpoint type. Eg:

default["glance"]["services"]["internal-api"]["uri"] = "http://blahblah:9292/v1"

This will effectively allow the recipes to still use the search functions, but if a value is present here, it will not actually perform any searches and will use the uri value supplied. See here for how that works in the search libraries:

https://github.com/rcbops-cookbooks/osops-utils/blob/grizzly/libraries/ip_location.rb#L118-L123

At the moment this will work properly for non-ha setups, but as you point out the ha endpoints are derived in a different manner so this will need some additional work.

If this approach works for you, I can add the same functionality to nova, ceilometer, cinder

Thanks

@mancdaz

[DavidWittman]

Thanks for the follow-up, @mancdaz. Looks like a good start to me!

[mancdaz]

@DavidWittman

I working on getting this pushed to the openstack-ha cookbook:

https://github.com/rcbops-cookbooks/openstack-ha/pull/50

The idea is to basically do 2 things:

• if an arbitrary uri is passed for a service/endpoint-type, then it will not be clobbered by recreate_endpoint when the ha stuff is being set up
• if an arbitrary endpoint is not provided, default to using the public endpoint for all endpoint types - this maintains current behaviour.

Please note that it's not actually possible, with the way our ha-controller role currently work (ie - forcing a service to listen on a specific IP so that haproxy can coexist on the box listening on a different IP for the same port) to separately HA each of the 3 different endpoint types for a service. So it's perfectly possible to break all of the HA stuff by passing bad values in...

Once this gates, please check and test and let me know if it fulfils your original requirements.

Cheers

[DavidWittman]

@mancdaz Thanks again. Here are some notes from my attempts at utilizing your changes from rcbops-cookbooks/openstack-ha#50:

• No support for overriding the Keystone endpoints, as they fall under the "identity" case, not the else block
• Overriding URIs doesn't work when using "dynamic" endpoints which include the %(tenant_id)s strings. This fails when the original endpoints are created (before HA recreates them) in get_config_endpoint. It looks like it just barfed parsing the URI on line 120 because it's technically invalid.
• internal_endpoint and admin_endpoint incorrectly clone public_endpoint if a URI is set for public and not the other endpoints. Instead, these endpoints should probably call get_access_endpoint if a URI is not set. e.g.

# ensure we use uri values for each endpoint type if they have been provided.  Else look them up
# NOTE:(mancdaz) right now, unless you provide an override value for an endpoint type, it will use the
# public endpoint.  This maintains current HA functionality.                 
public_endpoint = rcb_safe_deref(node, "#{ns}.services.#{svc}.uri") ? node[ns]["services"][svc] : get_access_endpoint(role, ns, svc)
internal_endpoint = rcb_safe_deref(node, "#{ns}.services.internal-#{svc}.uri") ? node[ns]["services"]["internal-#{svc}"] : get_access_endpoint(role, ns, svc)
admin_endpoint = rcb_safe_deref(node, "#{ns}.services.admin-#{svc}.uri") ? node[ns]["services"]["admin-#{svc}"] : get_access_endpoint(role, ns, svc)

Let me know if you need clarification on any of these points.

[mancdaz]

@DavidWittman

Thanks for the feedback. I'll address each point in random order :-):

• _internal_endpoint and admin_endpoint incorrectly clone public_endpoint if a URI is set for public and not the other endpoints. Instead, these endpoints should probably call get_accessendpoint if a URI is not set.

This is as intended. As stated previously:

- if an arbitrary endpoint is not provided, default to using the public endpoint for all endpoint types - this maintains current behaviour

Previously all endpoint types would get created with the details of what is now the public endpoint type (used to be just 'the endpoint'). To maintain consistency, if no overrides are provided then using the public endpoint is what we will do for all endpoint types within an endpoint, hence the cloning.

Trying to use get_access_endpoint triggers a further chain of dependencies, requiring that a separate VIP is provided not only for each endpoint, but for each endpoint type within each endpoint. As we've discussed, we can't treat each endpoint type as a completely discrete service that can be ha-ed, simply because it isn't.

What we're trying to provide here is a way to provide an arbitrary value for an endpoint type that won't get clobbered by any other part of the recipes. I don't want to (can't) solve the HA-ing of those bits, and so want the recipes to behave as they currently do if we are not providing overrides.

• No support for overriding the Keystone endpoints, as they fall under the "identity" case, not the else block

I had initially intentionally not looked at keystone as it already has the ability to provide a 'service' and 'admin' endpoint type (due to the fact these services run on separate ports unlike all the other api servers), but we perhaps need to try and make it look/behave like all the others do now.

• _Overriding URIs doesn't work when using "dynamic" endpoints which include the %(tenantid)s strings

Hopefully this is just a case of adding some escaping but I'll take a look

[DavidWittman]

Whoops... I need to work on my reading comprehension. It seems cumbersome to set all of the endpoints just to override the public, but I do understand the logic behind it.

What we're trying to provide here is a way to provide an arbitrary value for an endpoint type that won't get clobbered by any other part of the recipes.

So why not make arbitrary attributes like node[ns]['services'][service]['publicURL']) to represent the endpoint URLs? This way it's easy to override the endpoint without interrupting any of the HA stuff or other behaviors. That would likely solve the dynamic endpoint parsing issue as well, as these values would no longer be retrieved in get_access_endpoint.

[mancdaz]

Hi @DavidWittman

We could do that, but we'd still need to decide what to do with internal/admin endpoint types when no override value is provided. Likely we'd do what we're doing here, which is just to clone the 'public' endpoint type to keep current behaviour. If we're doing that, then actually we may as well do what we've done above (which has the added benefit of fitting in to the way all the service attributes are set up now without additional arbitrary endpoints, as well as the ability to override only certain portions of the endpoint like port/host/path etc) since we'd not be calling get_access_endpoint anyway!

...and breathe...

Does that make sense?

[DavidWittman]

@mancdaz It does. Whatever you think is best. As long as we can easily override the publicURLs, I don't see a problem with either. Personally, I'm partial to the behavior which leaves the other endpoints untouched when you override the publicURL, because it prevents the user from having to specify three endpoints just to change one, but I don't want to break any existing behaviors.

Thanks again!

[mancdaz]

Hi @DavidWittman

You only have to override any one of the endpoint types for that value to be used. You don't have to provide three endpoint values just to override one. Any endpoint type where an override value is not provided will simply clone the public endpoint type url.

As to the three outstanding issues with the previous patch, see below:

No support for overriding the Keystone endpoints, as they fall under the "identity" case, not the else block

Addressed by https://github.com/rcbops-cookbooks/openstack-ha/pull/51

Overriding URIs doesn't work when using "dynamic" endpoints which include the %(tenant_id)s strings. This fails when the original endpoints are created (before HA recreates them) in get_config_endpoint. It looks like it just barfed parsing the URI on line 120 because it's technically invalid.

Addressed by https://github.com/rcbops-cookbooks/osops-utils/pull/116

internal_endpoint and admin_endpoint incorrectly clone public_endpoint if a URI is set for public and not the other endpoints. Instead, these endpoints should probably call get_access_endpoint if a URI is not set

Intentional to preserve current behaviour.

LMK what you think after testing.

@mancdaz

[mancdaz]

Are we happy with this? I'd like to close this issue off if possible.

Cheers @mancdaz

[DavidWittman]

LGTM, but I haven't had a chance to test yet. Going to try to find some time this week.

[git-harry]

I'm closing this due to fixes put in place and no additional feedback.