[GH-352] kubernetes auth for lookup

Hi @pfeifferj welcome! Thanks for looking to continue the work in #220 .

First, I'd like to ensure that the commits in this branch that came from that PR still retain the original author's info to ensure proper credit. Let me know if you need a hand trying to get that info in the commits.

Also @chris93111 if you are interested in picking this work up again, please let us know, maybe you can collaborate with @pfeifferj if that's the case?

Otherwise, there's a few issues correct from the original commits, a previous rebase in that PR seems to have incorrectly resolved some conflicts, so there are some strange changes in here for example the changes to the hashi_vault lookup (those should be removed).

The version_added will also need to be changed but that's best done closer to the PR's completion since there may be releases between then and now.

Other than that, please look over my comments in #220 carefully, as most of those asks still apply.

For example, we'll want to ensure we have unit and integration tests.

Please also take a look at the Contributor guide.

It would be great to get this completed, and I can help with some aspects like testing, as time permits.

Hi @briantist, Thank you for your comments, sounds good! I'll fix the commit attribution later today. Are you okay with the commits still being squashed to keep the history more readable, though? As for the rest, I will create sub-tasks in the PR description so we can keep track of those :)

I would propose this based on the git log:

pic-selected-230303-1105-59

Hi @briantist i have no problem with this PR , I have trouble finding time to make the working CI (k3d and vault) if @pfeifferj can help no problem, if we can both be contributors, that's cool I use this lookup in production for over a year

I'll fix the commit attribution later today. Are you okay with the commits still being squashed to keep the history more readable, though?

Yes squashing is fine. I think I might have a few commits in that history but I'm only concerned with attribution for @chris93111 , any commits from me were minor suggestions or whatever and I don't care if those get squashed or removed.

As for the rest, I will create sub-tasks in the PR description so we can keep track of those :)

Perfect, thank you!

i have no problem with this PR , I have trouble finding time to make the working CI (k3d and vault) if @pfeifferj can help no problem, if we can both be contributors, that's cool I use this lookup in production for over a year

Great! Thanks for following up. We have more than just the hashi_vault lookup now, and they all share auth methods, so when this merges, you'll also be able to use kubernetes auth with any of the other plugins (and modules!) we have.

Hi @pfeifferj , I might be able to help resolve some of these conflicts and older rebase artifacts, but I noticed several pushes today so I don't want to push up commits that might step on your work if you've still got some thing in flight locally.

Let me know if you'd like me to pull it down and try to resolve this stuff!

Hi @pfeifferj , I might be able to help resolve some of these conflicts and older rebase artifacts, but I noticed several pushes today so I don't want to push up commits that might step on your work if you've still got some thing in flight locally. Let me know if you'd like me to pull it down and try to resolve this stuff!

hi @briantist those pushes were just to fix the commit history. Would be really great if you could help with the conflicts. I should have time to work on the tests next weekend :)

Cheers, Josie

Ok, I've done a rebase against main to bring in those changes, and in the process I think I've resolved all the outstanding conflicts and such, so I think this is ready to be built upon for the tests and such. Thanks Josie!

Codecov Report

Merging #353 (345eef8) into main (fcbfae5) will decrease coverage by 0.44%. The diff coverage is 41.93%.

:exclamation: Current head 345eef8 differs from pull request most recent head dad910a. Consider uploading reports for the commit dad910a to get more accurate results

@@            Coverage Diff             @@
##             main     #353      +/-   ##
==========================================
- Coverage   98.82%   98.39%   -0.44%     
==========================================
  Files          80       81       +1     
  Lines        4095     4112      +17     
  Branches      259      262       +3     
==========================================
- Hits         4047     4046       -1     
- Misses         39       57      +18     
  Partials        9        9

Flag	Coverage Δ
env_docker-default	`98.39% <41.93%> (-0.44%)`	:arrow_down:
integration	`80.51% <41.93%> (-0.58%)`	:arrow_down:
sanity	`39.84% <38.70%> (+0.08%)`	:arrow_up:
target_ansible-doc	`100.00% <ø> (ø)`
target_auth_approle	`89.47% <ø> (ø)`
target_auth_aws_iam	`50.00% <ø> (ø)`
target_auth_azure	`53.84% <ø> (ø)`
target_auth_cert	`86.36% <ø> (ø)`
target_auth_jwt	`91.30% <ø> (ø)`
target_auth_ldap	`89.47% <ø> (ø)`
target_auth_none	`100.00% <ø> (ø)`
target_auth_token	`71.42% <ø> (ø)`
target_auth_userpass	`85.71% <ø> (ø)`
target_connection_options	`74.76% <ø> (ø)`
target_controller	`83.02% <41.93%> (-0.77%)`	:arrow_down:
target_filter_vault_login_token	`77.77% <ø> (ø)`
target_import	`39.84% <38.70%> (+0.08%)`	:arrow_up:
target_lookup_hashi_vault	`81.33% <ø> (ø)`
target_lookup_vault_ansible_settings	`55.75% <41.93%> (-0.26%)`	:arrow_down:
target_lookup_vault_kv1_get	`91.30% <ø> (ø)`
target_lookup_vault_kv2_get	`91.11% <ø> (ø)`
target_lookup_vault_list	`90.00% <ø> (ø)`
target_lookup_vault_login	`88.57% <ø> (ø)`
target_lookup_vault_read	`90.00% <ø> (ø)`
target_lookup_vault_token_create	`79.24% <ø> (ø)`
target_lookup_vault_write	`56.72% <41.93%> (-0.34%)`	:arrow_down:
target_module_utils	`96.22% <41.93%> (-1.14%)`	:arrow_down:
target_module_vault_kv1_get	`87.50% <ø> (ø)`
target_module_vault_kv2_delete	`56.20% <41.93%> (-0.73%)`	:arrow_down:
target_module_vault_kv2_get	`87.23% <ø> (ø)`
target_module_vault_list	`85.71% <ø> (ø)`
target_module_vault_login	`83.72% <ø> (ø)`
target_module_vault_pki_generate_certificate	`78.72% <ø> (ø)`
target_module_vault_read	`85.71% <ø> (ø)`
target_module_vault_token_create	`91.66% <ø> (ø)`
target_module_vault_write	`55.55% <41.93%> (-0.70%)`	:arrow_down:
target_modules	`81.12% <41.93%> (-0.67%)`	:arrow_down:
units	`96.12% <41.93%> (-0.43%)`	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
plugins/doc_fragments/auth.py	`100.00% <ø> (ø)`
plugins/module_utils/_auth_method_k8s.py	`40.00% <40.00%> (ø)`
plugins/module_utils/_authenticator.py	`100.00% <100.00%> (ø)`

... and 2 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

hi @briantist I'm looking into the tests now and wanted to touch base on what you think the best approach is. Should I just extend mmock for integration tests and the fixtures for unit tests?

I'm looking into the tests now and wanted to touch base on what you think the best approach is. Should I just extend mmock for integration tests and the fixtures for unit tests?

Extending MMock would be the easier way, but it will also be less accurate/effective.

In the previous PR I remember looking into it a little bit and it seemed like it'd be possible to set up k3s/k3d during testing, but I didn't get around to trying it, it was more of a suggestion. I also don't have any experience with using those.

So if you have some time to look into that it might be better long term, but it's a bigger ask. I'd definitely accept new MMock endpoints instead.

Above is all for integration; we'll want to do units too but that won't require any external dependencies.

As a possible bonus, the same fixtures (JSON output) you might generate for some of the unit tests could prove useful for new MMock configs.

Take a look and let me know what you think, I'm sure I can help a bit if you get started and need a hand.

Side note, it looks like https://github.com/ansible-collections/community.hashi_vault/pull/353/commits/dfb9ba5dd993f4eef8d77115ef5b249acea439b9 and https://github.com/ansible-collections/community.hashi_vault/pull/353/commits/c80f36d60486fa3610c0aa20a4dc97ceeb5aab9d re-introduced some stuff from the old PR that I had fixed up in the previous commits, and some of that is still lingering. It looks like the second one tried to fix some of that.

There should be no changes in hashi_vault.py (the lookup) for example compared to main.

Extending MMock would be the easier way, but it will also be less accurate/effective.

In the previous PR I remember looking into it a little bit and it seemed like it'd be possible to set up k3s/k3d during testing, but > I didn't get around to trying it, it was more of a suggestion. I also don't have any experience with using those.

So if you have some time to look into that it might be better long term, but it's a bigger ask. I'd definitely accept new MMock endpoints instead.

I haven't used k3s in a testing/ci context before but could look into it. I would suggest doing so in a follow-up ticket, though

Above is all for integration; we'll want to do units too but that won't require any external dependencies.

As a possible bonus, the same fixtures (JSON output) you might generate for some of the unit tests could prove useful for new MMock configs.

Take a look and let me know what you think, I'm sure I can help a bit if you get started and need a hand.

Great, thank you. I'll work on it some more and will get in touch for feedback.

Side note, it looks like https://github.com/ansible-collections/community.hashi_vault/commit/dfb9ba5dd993f4eef8d77115ef5b249acea439b9 and https://github.com/ansible-collections/community.hashi_vault/commit/c80f36d60486fa3610c0aa20a4dc97ceeb5aab9d re-introduced some stuff from the old PR that I had fixed up in the previous commits, and some of that is still lingering. It looks like the second one tried to fix some of that.

There should be no changes in hashi_vault.py (the lookup) for example compared to main.

My bad. I forgot to pull your fix and made a bit of a mess. Will take care of it now :)

any plans on completing this? thanks

SUMMARY

continuation of pr#220

ISSUE TYPE

Feature Pull Request #352

Resolves #352 Closes #220

COMPONENT NAME

community.hashi_vault/plugins/lookup

ADDITIONAL INFORMATION

todo:

[x] Restore commit authors
[x] Fix incorrectly resolved conflicts caused by previous rebase (for example the changes to the hashi_vault lookup -- those should be removed)
[ ] The version_added will also need to be changed but that's best done closer to the PR's completion since there may be releases between then and now.
[ ] Resolve comments in https://github.com/ansible-collections/community.hashi_vault/pull/220 carefully, as most of those asks still apply.
[ ] Add unit and integration tests. https://github.com/ansible-collections/community.hashi_vault/tree/main/tests/unit & https://github.com/ansible-collections/community.hashi_vault/tree/main/tests/integration/targets/setup_localenv_docker

Docs Build 📝

Thank you for contribution!✨

The docs for this PR have been published here: https://ansible-collections.github.io/community.hashi_vault/pr/353

You can compare to the docs for the main branch here: https://ansible-collections.github.io/community.hashi_vault/branch/main

The docsite for this PR is also available for download as an artifact from this run: https://github.com/ansible-collections/community.hashi_vault/actions/runs/4407025680

File changes:

M collections/community/hashi_vault/hashi_vault_lookup.html

M collections/community/hashi_vault/vault_kv1_get_lookup.html

M collections/community/hashi_vault/vault_kv1_get_module.html

M collections/community/hashi_vault/vault_kv2_delete_module.html

M collections/community/hashi_vault/vault_kv2_get_lookup.html

M collections/community/hashi_vault/vault_kv2_get_module.html

M collections/community/hashi_vault/vault_list_lookup.html

M collections/community/hashi_vault/vault_list_module.html

M collections/community/hashi_vault/vault_login_lookup.html

M collections/community/hashi_vault/vault_login_module.html

M collections/community/hashi_vault/vault_pki_generate_certificate_module.html

M collections/community/hashi_vault/vault_read_lookup.html

M collections/community/hashi_vault/vault_read_module.html

M collections/community/hashi_vault/vault_token_create_lookup.html

M collections/community/hashi_vault/vault_token_create_module.html

M collections/community/hashi_vault/vault_write_lookup.html

M collections/community/hashi_vault/vault_write_module.html

M collections/environment_variables.html

Click to see the diff comparison.

**NOTE:** only file modifications are shown here. New and deleted files are excluded. See the file list and check the published docs to see those files. **The diff output was truncated because it exceeded the maximum size.** ```diff diff --git a/home/runner/work/community.hashi_vault/community.hashi_vault/docsbuild/base/collections/community/hashi_vault/hashi_vault_lookup.html b/home/runner/work/community.hashi_vault/community.hashi_vault/docsbuild/head/collections/community/hashi_vault/hashi_vault_lookup.html index 1eabb59..48ec29d 100644 --- a/home/runner/work/community.hashi_vault/community.hashi_vault/docsbuild/base/collections/community/hashi_vault/hashi_vault_lookup.html +++ b/home/runner/work/community.hashi_vault/community.hashi_vault/docsbuild/head/collections/community/hashi_vault/hashi_vault_lookup.html @@ -192,11 +192,19 @@ see "azure"

"jwt"

"cert"

"kubernetes"

"none"

Configuration:

INI entry:

INI entries:
+
```
[lookup_hashi_vault]
+auth_method = token
+
```
+
+
Removed in: version 3.0.0
+
Why: collection-wide config section
+
Alternative: use section [hashi_vault_collection]
```
[hashi_vault_collection]
 auth_method = token
 
```
@@ -235,7 +243,14 @@ see
If specified, sets the value to use for the X-Vault-AWS-IAM-Server-ID header as part of GetCallerIdentity request.

Configuration:
- INI entries:
  +
  [lookup_hashi_vault] +aws_iam_server_id = VALUE +
  +
  +
  Removed in: version 3.0.0
  +
  Why: collection-wide config section
  +
  Alternative: use section [hashi_vault_collection]
  [hashi_vault_collection] aws_iam_server_id = VALUE
  @@ -372,7 +387,15 @@ see
  VAULT_CACERT environment variable will be used.
  
  Configuration:
  - INI entries:
    +
    [lookup_hashi_vault] +ca_cert = VALUE +
    +
    +
    added in community.hashi_vault 1.2.0
    +
    Removed in: version 3.0.0
    +
    Why: collection-wide config section
    +
    Alternative: use section [hashi_vault_collection]
    
    [hashi_vault_collection] ca_cert = VALUE
    @@ -436,6 +459,39 @@ see
    +
    kubernetes_token
    +
    string
    +
    added in community.hashi_vault 2.5.0
    +
    +
    The Kubernetes Token (JWT) to use for Kubernetes authentication to Vault.
    +
    Configuration:
    +
    +
    Environment variable: ANSIBLE_HASHI_VAULT_KUBERNETES_TOKEN
    +
    Variable: ansible_hashi_vault_kubernetes_token
    +
    +
    + +
    +
    kubernetes_token_path
    +
    string
    +
    added in community.hashi_vault 2.5.0
    +
    +
    If no kubernetes_token is specified, will try to read the token from this path.
    +
    Default: "/var/run/secrets/kubernetes.io/serviceaccount/token"
    +
    Configuration:
    +
    +
    INI entry:
    +
    [hashi_vault_collection] +kubernetes_token_path = /var/run/secrets/kubernetes.io/serviceaccount/token +
    +
    +
    +
    Environment variable: ANSIBLE_HASHI_VAULT_KUBERNETES_TOKEN_PATH
    +
    Variable: ansible_hashi_vault_kubernetes_token_path
    +
    +
    + +
    
    mount_point
    
    string
    
    @@ -451,7 +507,7 @@ see added in community.hashi_vault 1.5.0
  - Environment variable: ANSIBLE_HASHI_VAULT_MOUNT_POINT
    +
  - Environment variable: ANSIBLE_HASHI_VAULT_MOUNT_POINT
    
    added in community.hashi_vault 1.5.0
  - Variable: ansible_hashi_vault_mount_point
    @@ -469,14 +525,22 @@ see VAULT_NAMESPACE is set, its value will be used last among all ways to specify namespace.
    
    Configuration:
    
    -
    INI entry:
    +
    INI entries:
    +
    [lookup_hashi_vault] +namespace = VALUE +
    +
    +
    added in community.hashi_vault 0.2.0
    +
    Removed in: version 3.0.0
    +
    Why: collection-wide config section
    +
    Alternative: use section [hashi_vault_collection]
    
    [hashi_vault_collection] namespace = VALUE
    
    added in community.hashi_vault 1.4.0
    
    -
    Environment variable: ANSIBLE_HASHI_VAULT_NAMESPACE
    +
    Environment variable: ANSIBLE_HASHI_VAULT_NAMESPACE
    
    added in community.hashi_vault 0.2.0
    
    Variable: ansible_hashi_vault_namespace
    @@ -492,7 +556,7 @@ see
    Authentication password.
    
    Configuration:
    
    -
    Environment variable: ANSIBLE_HASHI_VAULT_PASSWORD
    +
    Environment variable: ANSIBLE_HASHI_VAULT_PASSWORD
    
    added in community.hashi_vault 1.2.0
    
    Variable: ansible_hashi_vault_password
    @@ -515,14 +579,21 @@ see environment variables from the Requests library are used.
    
    Configuration:
    
    -
    INI entry:
    +
    INI entries:
    +
    [lookup_hashi_vault] +proxies = VALUE +
    +
    +
    Removed in: version 3.0.0
    +
    Why: collection-wide config section
    +
    Alternative: use section [hashi_vault_collection]
    
    [hashi_vault_collection] proxies = VALUE
    
    added in community.hashi_vault 1.4.0
    
    -
    Environment variable: ANSIBLE_HASHI_VAULT_PROXIES
    +
    Environment variable: ANSIBLE_HASHI_VAULT_PROXIES
    
    Variable: ansible_hashi_vault_proxies
    
    added in community.hashi_vault 1.2.0
    
    @@ -536,8 +607,8 @@ see
    The AWS region for which to create the connection.
    
    Configuration:
    
    -
    Environment variable: EC2_REGION
    -
    Environment variable: AWS_REGION
    +
    Environment variable: EC2_REGION
    +
    Environment variable: AWS_REGION
  @@ -555,14 +626,21 @@ see Configuration:
  - Environment variable: ANSIBLE_HASHI_VAULT_RETRIES
  - Environment variable: ANSIBLE_HASHI_VAULT_RETRIES
  - Variable: ansible_hashi_vault_retries
@@ -581,14 +659,21 @@ see Configuration:
- Environment variable: ANSIBLE_HASHI_VAULT_RETRY_ACTION
- Environment variable: ANSIBLE_HASHI_VAULT_RETRY_ACTION
- Variable: ansible_hashi_vault_retry_action

Codecov Report

ansible-collections / community.hashi_vault