SSH certificate Error: template '{{identity.entity.aliases.auth_jwt_1012d0.name}}' could not be rendered -> alias not found

[prbinu]

Describe the bug We are trying to create a Vault SSH-engine role to generate SSH certificate. What we are doing is to allow the user to login using either OIDC/JWT or LDAP and then request SSH certificate through one role. The Vault role spec is provided below:

vault write ssh/roles/test -<<"EOH"
{
    "algorithm_signer": "rsa-sha2-256",
    "allow_user_certificates": true,
    "allow_user_key_ids": false,
    "allowed_users": "{{identity.entity.aliases.auth_jwt_1012d0.name}}, {{identity.entity.aliases.auth_ldap_12345.name}}",
    "allowed_users_template": true,
    "default_critical_options": {},
    "default_user": "",
    "key_type": "ca",
    "max_ttl": 3600,
    "ttl": 3600
}
EOH

To Reproduce The above role spec doesn't work because Vault is throwing template error -- ie when I try fetching ssh certs using LDAP login, it throws JWT alias template error, and same issue is observed when I try the other way round :

URL: PUT https://vault.xxx.com/v1/ssh/sign/test
Code: 400. Errors:

* template '{{identity.entity.aliases.auth_jwt_1012d0.name}}' could not be rendered -> alias not found

Expected behavior The expectation is that Vault shall honor only the valid identity entity alias. For example, when I request certs using JWT login, it shall return the SSH cert, but ignoring LDAP identity entity alias. Likewise if I request certs with LDAP login, the Vault shall ignore the JWT identity entity alias and issue the cert with my LDAP identity entity alias.

The only workaround at the moment is to use two separate Vault roles one for each - which is inconvenient and sub-optimal. We are aware of identity.entity.name, but this doesn't scale because we have to individually map each user from LDAP to Vault identity.entity.name.

Environment: Vault 1.7

[prbinu]

Just checking - any thoughts?

[pmmukh]

Hey @prbinu ! Thanks for submitting this issue! We plan to take a look at this sometime soon, in the meanwhile could you maybe add more explicit instructions for the bug repro ? The problem seems clear enough, but figuring out the exact sequence of instructions to get to there is always a little tricky, so a more detailed repro (as in, a clear sequence of cli commands if possible) would be great! :)

[heatherezell]

Hi @prbinu! Just echoing Prat here, if you could provide more specific repro steps, it'll help our engineers immensely. Thanks in advance!

[prbinu]

No sure what the confusion is. It should be straight forward for Vault team. I'm currently using a Vault instance operated by a different team, and to recreate everything on a localhost is a bit of work (LDAP, JWT setup etc). I will try to do it when i find some time, but i think it you can do it much faster.

Thank you for responding to my request.

[pmmukh]

No worries! I think it's just that a clearly defined set of repro steps helps us figure out whether something is a bug or not quickly, which is really helpful given the volume of issues we work with. However, if you aren't really able to reproduce this easily, that is fine, I still think we have enough in this issue to go on, and will try to get back to you soon as we can. Thanks! :)

[prbinu]

@pmmukh, it would be very helpful if you can address this issue.

[prbinu]

@pmmukh @hsimon-hashicorp , It would be helpful if you can review and merge this PR - #13344

[pmmukh]

Hey @prbinu ! Thanks for submitting the PR, I took a quick look at it today. Firstly, thanks for that great gist with all the testing steps, that's super helpful. However, I don't think your solution is going to be an acceptable one, as it seems to remove an error handling block, I believe whether or not we want to add this functionality, ignoring an error from the PopulateIdentityTemplate function is not going to work. I'll take a closer look at the underlying cause later, and try to make a recommendation on a possible solution.

[pmmukh]

Hey @prbinu! So I repro-d the issue, and was able to figure out why this was being a problem. So the 2 aliases, for the 2 mounts, weren't tied to the same entity at all. In your repro script, after you save the accessor ids into the files, add these 3 lines ( steps 12-14 of https://learn.hashicorp.com/tutorials/vault/identity )

vault write -format=json identity/entity name="bob-smith" policies="default" \
   metadata=organization="ACME Inc." \
   metadata=team="QA" \
   | jq -r ".data.id" > entity_id.txt
vault write identity/entity-alias name="bob" \
   canonical_id=$(cat entity_id.txt) \
   mount_accessor=$(cat accessor_test.txt) \
   custom_metadata=account="Tester Account"

vault write identity/entity-alias name="bsmith" \
   canonical_id=$(cat entity_id.txt) \
   mount_accessor=$(cat accessor_qa.txt) \
   custom_metadata=account="Tester Account"

After tying the 2 aliases to the same entity, this works fine. I'm going to close out this issue now, as this doesn't appear to be a problem.

[prbinu]

@pmmukh, thanks for reviewing my code. I'm aware of the entity aliasing, and if you see my original description's last section, I have mentioned about this.

excerpt:

The only workaround at the moment is to use two separate Vault roles one for each - which is inconvenient and sub-optimal. We are aware of identity.entity.name, but this doesn't scale because we have to individually map each user from LDAP to Vault identity.entity.name.

Like I mentioned, it doesn't scale well hence I opened this issue in the first place.

ignoring an error from the PopulateIdentityTemplate function is not going to work

Yes, it appears that removing an error check an issue, but I explored other ways, and determined that this would be the better option compared to others. If you can propose a better way, that would be great.

I would like you to reconsider as this issue is hurting us.

[heatherezell]

@prbinu can you please clarify what the issue that you're hitting is? Please avoid telling us what you think the solution would be - what I'm interested in hearing is what the actual problem is, and how it's affecting you. The more we can avoid the idea of expectations and solutions, and the more we can dig into the actual problem itself, the better our chances of finding a robust fix for you and for others who may be running into the same issue. Thanks for understanding!

[prbinu]

@hsimon-hashicorp, the issue is described in the original issue description. To reiterate, we have our users login to Vault through, say JWT/OIDC or through LDAP. Both these users would like to use the same Vault ssh-cert role to fetch SSH certificate. However currently Vault ssh cert role supports only only one identity alias in a role (example: "allowed_users": "{{identity.entity.aliases.auth_jwt_1012d0.name}} OR {{identity.entity.aliases.auth_ldap_12345.name}}", not both).

The request is to allow users that use different auth mechanisms (notably JWT and LDAP) to share the same ssh cert role.

For example:

User-1:
1. Use vault login with oidc/jwt
2. fetch SSH certs from ssh/sign/mycert
3. ssh user-1@target

User-2:
1. Use vault login /auth/ldap/user
2. fetch SSH certs from ssh/sign/mycert
3. ssh user-2@target

Tying the 2 aliases to the same entity practically not helpful because we cannot enumerate all users upfront in LDAP or OIDC and create entity mappings in Vault (this works well if we create users locally in Vault, say userpass mode).

Right now we are creating separate roles for each auth engine. This is creating lot of logistical issues related to communication, documentation, support requests, and additional work. I hope these details help.

[pmmukh]

hey @prbinu ! Ah ok I missed the part in the description around why you don't want to be tying aliases into the same entity for logistical overhead issues, I misunderstood and thought that was overlooked in the repro, my bad. Thanks a ton for the detailed explanation of your use case, the context is helpful and much appreciated. I'm going to reopen this, and mark it as an enhancement instead of a bug, as the current behavior works correctly, what we want in this scenario is to ignore unresolved templates instead of erroring outright I believe. This seems like a decent chunk of work, as the error return of PopulateIdentityTemplate does not right now distinguish between this template doesn't match anything and this template is wrong, and I'm not sure how much effort it would be to make that distinction. So we may not get around to this soon, and would probably look into prioritizing this at a later date, meanwhile do feel free to take a stab at it yourself! Also happy to discuss approaches on this issue in the interim as well! Also, as with all feature requests, for others who are interested in this, please stick a 👍 on this issue.

[pmmukh]

Hi! I just wanted to followup here, and add some clarification around expectations. I was a bit hasty in inviting a PR on this yesterday, this feels like a tricky problem to solve that requires further discussion before active work. The idea of ignoring template matching failures is probably not a great solution, so please bear with us while we figure out

1. If this is something we want to solve for, or if its not feasible and the alias->entity mapping approach is the one to go with fo the foreseeable future
2. If it is, what such a solution would look like.

In the interim, do feel free to suggest solutions for this problem that make sense to you, and would be happy to discuss those here!

[prbinu]

The idea of ignoring template matching failures is probably not a great solution, so please bear with us while we figure out

Yes, I agree with this statement, but, consider this nuance: https://gist.github.com/prbinu/3a2e349eac5d265fd8683fcb01a84028#file-ssh-cert-template-test-sh-L39

Here you can see ubuntu as another principal/user in addition to the templated user (for now consider just one template). Now if I make an ssh cert request with ubuntu as the principal, irrespective of what is in the template (including a bad template), it works. It is because in such cases, the control never reaches template matching loop. BUT even in this flow, I think there is a bug. The bug is that two different users with same permissions/policy attached see two different behavior. Here is how to reproduce:

$ vault auth list
Path              Type        Accessor                  Description
----              ----        --------                  -----------
token/            token       auth_token_3691a7fb       token based credentials
userpass-qa/      userpass    auth_userpass_55f9ceb3    n/a
userpass-test/    userpass    auth_userpass_80317275    n/a

$ vault login -format=json -method=userpass -path=userpass-qa \
    username=bsmith password=training \
    | jq -r ".auth.client_token" > bsmith_token.txt

# this doesn't work!
$ VAULT_TOKEN=$(cat bsmith_token.txt) vault write -field=signed_key ssh-client-signer/sign/my-role \
        valid_principals="ubuntu" \
        public_key=@$HOME/.ssh/id_rsa.pub  | ssh-keygen -L -f -
Error writing data to ssh-client-signer/sign/my-role: Error making API request.
URL: PUT http://127.0.0.1:8200/v1/ssh-client-signer/sign/my-role
Code: 400. Errors:

* template '{{identity.entity.aliases.auth_userpass_80317275.name}}' could not be rendered -> alias not found

The same command with root token (or another user - different auth engine) works.

$ VAULT_TOKEN=<root-token> vault write -field=signed_key ssh-client-signer/sign/my-role  \
        valid_principals="ubuntu"  public_key=@$HOME/.ssh/id_rsa.pub | ssh-keygen -L -f -
(stdin):1:
        Type: ssh-rsa-cert-v01@openssh.com user certificate
        Public key: RSA-CERT SHA256:iePpS5llnO4W81CMEn5mRBuGXFx9yuEo8yE2zBTYHZc
        Signing CA: RSA SHA256:xEzFHGTxj8YeSH/I6DkQevGyyi42b8Hgj7oGKNAOTf8 (using ssh-rsa)
        Key ID: "vault-root-89e3e94b99659cee16f3508c127e66441b865c5c7dcae128f32136cc14d81d97"
        Serial: 12183672459135669608
        Valid: from 2021-12-15T11:14:16 to 2021-12-15T11:44:46
        Principals:
                ubuntu
        Critical Options: (none)
        Extensions:
                permit-pty

I believe this is a bug, because both users are trying to get a certificate with principal ubuntu, and both are authorized as per the policy, but in the former case it works, but in the latter case it fails.

---

Keeping aside this bug, by applying the same logic used for non-templated users on templates, Vault may iterate through multiple templates to find a good match (similar to how to match hardcoded principals), and if no match exists, then return error.

Possible error check point:

1. Do some sanity checks when a user creates role with templated principals to see the syntax is correct (a quick test shows that there are no checks happening during role creating stage - wouldn't that be an issue?)

[prbinu]

@pmmukh did you get a chance to look into this issue?

[heatherezell]

Hi @prbinu - this will actually be more of an @sgmiller thing right now. :) (We have internal schedules for our engineers who help steward particular areas of focus during our sprints. Please always feel free to ping me, as I'm your main point of contact.)

[prbinu]

thanks @hsimon-hashicorp @sgmiller , please let me know if you need any additional details on this.

[jdef]

looking for an update here. any timeline as to when this will be resolved?

[prbinu]

@sgmiller, @hsimon-hashicorp can we get this fixed please? it's a bug.

[heatherezell]

I'll check with our product owners, but unfortunately I cannot guarantee any specific timelines. Thanks for your patience!

[cwchristerw]

I would like to see solution that makes it possible to use {{ identity.entity.aliases.auth_ldap_xxxxxxxx.name }},{{ identity.entity.aliases.auth_oidc_xxxxxxxx.name }} and it would automatically ignore missing alias or add option to make it ignore missing alias. I would use this in SSH (Secret Engine) Allowed users, I have created users role.