api.NewAPIClient(&cfg) getting back 403 with NSX-T 3.0.2

[rene-at-dell]

Hi,

We've trying to be proactive instead of reactive and put some monitoring in place so that our VM provisioning process allocates IPs from an IP Pool that actually has free IPs.

We've looked at the Terraform Provider to see how it's its doing it's thing and we replicated the client creation with the same exact parameters we're using with Terraform Templates. Ie. the api.NewAPIClient(&cfg) method is receiving the exact same parameters.

Then the weirdness starts. When we to authenticate (username/password basic auth), we always get a 403 with a message in the body stating the the account is locked out (think that might be a bug on the API as it's a false positive). We're able to authenticate successfully with postman, but by using query parameters which this library doesn't use (we're using vSphere 7 by the way).

As I'm sure you know, stepping through code of a Terraform Provider is next to impossible, but I can state with a high degree of certainty after putting log statements all over the place in their code that it's also getting a 403 on the client creation. Only thing, it's ignoring that error and later using the github.com/vmware/vsphere-automation-sdk-go library to reauthenticate. (note that the methods they're using don't even exist in the latest version, they're using 0.5.0)

Now we can't follow that same path as we need to get the info for all IP Pools (nsxtClient.PoolManagementApi.ListIpPools() seems to fit the bill perfect) and not just one (automation sdk doesn't seem to support it). Starting to see my dilemma?

Here's what I'm wondering:

1. Is that 403 a bug, or am I just doing something wrong in the Client initialization? (have tried every imaginable permutation)
2. Can I fetch the IP Pool information for all Pools with just this lib (ie call /api/v1/pools/ip-pools)? Or do I need the automation SDK as well???
3. Was that too much of a mouthful? :)

Note: The fix for the auth is really not a big deal (disclaimer: only have vSphere 7.0 to test against). Are you guys open to a pull request if you don't have the bandwidth to fix this?

[annakhm]

Hi @rene-at-dell, terraform provider https://github.com/vmware/terraform-provider-nsxt is using 2 sets of APIs - old MP APIs (soon to be deprecated) use this sdk, and new Policy API use github.com/vmware/vsphere-automation-sdk-go. You can read more about the policy API here https://blogs.vmware.com/networkvirtualization/2020/06/navigating-nsxt-policy-apis.html/. Corresponding to those two sets of APIs, the provider offers two sets of resources, and each resource would use its corresponding sdk (with its own connection). Both sdks authenticate successfully. I'm a bit confused why you mention vsphere version, since both sdks in question only work against NSXT, which is a separate endpoint with separate auth parameters. Could this be the root of your issue? The vsphere-automation-sdk-go sdk is much more recent and up to date than this repo. IP pools are offered here https://github.com/vmware/vsphere-automation-sdk-go/tree/master/services/nsxt/infra/ip_pools. Note that the old MP API is also offered via https://github.com/vmware/vsphere-automation-sdk-go/tree/master/services/nsxt-mp.

[rene-at-dell]

I'm sorry @annakhm, I actually meant NSX-T Version 3.0.2.0.0.16887200... I wrote that after too many hours of debugging, and I've also been working with the govmomi lib: my wires got crossed :o

I am most definitely using the NSX-T client. Creating it with this: https://github.com/vmware/go-vmware-nsxt/blob/9a3393987775c2611af97b415eec835268b861aa/api_client.go#L234

The 403 is actually very easy to reproduce as it happens within the constructor code. This is the exact spot where you can see it: https://github.com/vmware/go-vmware-nsxt/blob/9a3393987775c2611af97b415eec835268b861aa/api_client.go#L137

Note: You guys are not using the body of the response, but if you read it and convert it to a string, you'll see the "account is locked" error (don't remember the exact wording).

We where able to make the 403 go away by stepping though the code and manually adding query parameters with the debugger here: https://github.com/vmware/go-vmware-nsxt/blob/9a3393987775c2611af97b415eec835268b861aa/api_client.go#L105

Ok, so if I understood correctly, we should switch to the new API? Problem with that is it doesn't seem to have a way of listing all IP Pools, and then for each finding out what the total # of IPs are and how many are used ??? I missing something?

Context info: Our VM provisioning automation process uses whatever assets an admin decides -> clusters, data stores and networks on the vsphere side; IP Pools on the NSX-T side. That monitoring process doesn't actually know what's supposed to be used, it just collects stats on everything and sends it over, a sort of resume on utilization. That's why we need a way to "discover" all existing IP pools.

Make sense?

[annakhm]

Hi @rene-at-dell, not sure this is relevant for you any more, but this issue is now being fixed. While the overall connection was functioning correctly, session create functionality was broken, which resulted in session token being ignored for future requests. This is going to be addressed with this PR https://github.com/vmware/go-vmware-nsxt/pull/48.