Receiving Malformed Request URL error in 1.0.2 when attempting to read Tenant Limits

[smcallister]

We are attempting to read the limits for each tenant, but we are running across an issue. It seems that the call works correctly for the first tenant, but after that we get the following error:

Malformed request URL: URL's project_id '7a03ba3cad8446fd8d8ba285756bed44' doesn't match Context's project_id '3a52cb057dbc4e898eb650e32e3d9e09'

Here is the sample code we are using:

     OSClient os = OSFactory.builder()
              .endpoint("http://<IP>:5000/v2.0")
              .credentials("admin", "<password>")
              .tenantName("admin")
              .authenticate();
     List<? extends Tenant> tenants = os.identity().tenants().list();
     for (Tenant tenant : tenants) {
        OSClient osTenant = OSFactory.builder()
                 .endpoint("http://<IP>:5000/v2.0")
                 .credentials("admin", "<password>")
                 .tenantName(tenant.getName())
                 .authenticate();
        Limits quotaLimits = osTenant.compute().quotaSets().limits(); // Throws exception on second tenant
     }

I've enabled the logging filter, and this is the output from the above code (for just the last call):

INFO: 1 * LoggingFilter - Request received on thread main 1 > GET http://:8774/v2/7a03ba3cad8446fd8d8ba285756bed44/limits 1 > Accept: application/json 1 > X-Auth-Token:

Sep 23, 2014 11:35:35 AM org.openstack4j.core.transport.internal.HttpLoggingFilter log INFO: 2 * LoggingFilter - Response received on thread main 2 < 400 2 < X-Compute-Request-Id: req-037a5de2-7685-4a13-ac83-c6aabc5ae000 2 < Connection: keep-alive 2 < Content-Length: 188 2 < Date: Tue, 23 Sep 2014 15:35:30 GMT 2 < Content-Type: application/json; charset=UTF-8 {"badRequest": {"message": "Malformed request URL: URL's project_id '7a03ba3cad8446fd8d8ba285756bed44' doesn't match Context's project_id '3a52cb057dbc4e898eb650e32e3d9e09'", "code": 400}}

The strange thing is that this works fine in the 1.0.1 released package, but for a local (unmodified) build of 1.0.2, we are getting this error.

Any ideas?

[gondor]

So I know the default tenant of admin typically always works if logging in as admin. Have you tried doing what you did in the first example (outside of the iteration) and manually specifying some other tenant?

ex:

OSClient osTenant = OSFactory.builder()
                 .endpoint("http://<IP>:5000/v2.0")
                 .credentials("admin", "<password>")
                 .tenantName("<some other tenant name>")
                 .authenticate();

If above works then we must be holding onto the previous session somewhere and I can further look into that.

Also, could you provide me with what the X-Auth-Token: is when you authenticate initially (it also should be the access id in the ison object returned, and what it is in the first iteration of your loop and also provide me with the "tenant { id: " found in the access object. You'll see it in the logging filter.

[gondor]

Disregard my previous comment. I added an extra tenant on mine and could reproduce. I have confirmed that we are sending the URL as defined in the access and the correct URL is being used within the iteration.

This seems to be a bug in OpenStack and I found many people with the same issues by google searching. Here's one example with some comments that may solve it: https://ask.openstack.org/en/question/7942/my-nova-doesnt-work-now-shows-urls-project_id-doesnt-match-contexts-project_id-the-urls-project_id-i-thought-is-the-role-of-admin-added-to-the-nova-at/

[smcallister]

Thanks for taking a look. I spent a bit more time digging into this, and I think I may have found what could be causing it. It looks like DefaultEndpointURLResolver maintains a static cache of URLs that have been resolved, and that map has several criteria it uses for a key. However, the tenant ID is not part of that key. So if I try to call the same API using clients from two different tenants, DefaultEndpointURLResolver will return back the same URL (including the ID of the first tenant) in both cases.

I commented out the lookup code in findURL, and after that everything works as expected. I'm sure there is a better way to solve this, but I'm just not very familiar with the details of the code. :)

[gondor]

Ok, thanks for pointing this out. I'll address it this evening based on your findings! thanks for the investigation :)

[gondor]

@smcallister Just checked in a fix

[smcallister]

Thanks @gondor! I tested the fix and confirmed it works as expected now.

One thing I noticed with the new code is that if we create an OSClient without a tenant name or ID, we get a NullPointerException. Here is the code we tried along with the full stack trace:

     OSClient os = OSFactory.builder()
              .endpoint(host)
              .credentials(user, password)
              .authenticate();

java.lang.NullPointerException at org.openstack4j.openstack.identity.domain.KeystoneAccess.getCacheIdentifier(KeystoneAccess.java:230) at org.openstack4j.openstack.identity.internal.DefaultEndpointURLResolver.findURL(DefaultEndpointURLResolver.java:35) at org.openstack4j.openstack.internal.OSClientSession.getEndpoint(OSClientSession.java:162) at org.openstack4j.core.transport.HttpRequest$RequestBuilder.build(HttpRequest.java:373) at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:135) at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:131) at org.openstack4j.openstack.identity.internal.TenantServiceImpl.list(TenantServiceImpl.java:21)

Since it looks like this may be related to the endpoint resolution fix, I figured it was appropriate to mention it in this issue. However, I can open a separate issue for this if you want.

[gondor]

Thanks, missed that test case. committing fix

[smcallister]

Confirmed, works great. Thanks again!

[gondor]

No problem!