Strategy for updating openapi.json in this repo

[maxb]

I wanted to update the openapi.json in this repo, to pick up my changes that have now landed in the Vault repo, and close #180.

I can see there are LOTS of OpenAPI changes to come, and I wonder how the HashiCorp reviewers would prefer to handle them. Here are some options I could imagine taking:

1) "Big bang" update to OpenAPI from current Vault main branch. One PR, but it would be immense, and challenging to changelog and review. From the perspective of a user of this library, it would make the changes in the next version be a scary, difficult to understand, single lump. Maybe that's just about tolerable this early in the library's lifecycle, but it would be increasingly less palatable the more people depend on the library.

2) "Slightly smaller but still pretty big bang" update to OpenAPI from Vault 1.14.0 release. Holds back some of the feature changes, but still, turbulent. (And then update to 1.15.0-dev in a separate PR.)

3) Trickle the updates in through a series of targeted PRs grouped around specific feature changes. This would make it more practical to carefully consider the changes to the generated code being triggered by the OpenAPI changes. This would be done either by generating OpenAPI from various builds of Vault at different points in history, or by manually curating which OpenAPI changes to bring in in intermediate PRs, by selecting chunks in a side-by-side diff viewer.

Please let me know in which direction you'd like to go, so I know whether it would be useful for me to create any PRs.

Also, where can I find the canonical script used to generate the openapi.json file in this repo? I tried using scripts/gen_openapi.sh in the Vault repo, however that doesn't seem to handle mounting both kv-v1 and kv-v2 correctly.

[averche]

Hi @maxb!

I think we've been doing the "big bang" style updates so far and we are OK with it since the library is still in beta. The plan in the future is to have the openapi.json spec reside in a centralized repository and updated frequently through a CI pipeline. Each update will trigger an automated PR to each of the OpenAPI-based libraries. This way all updates should hopefully be small and incremental.

Also, where can I find the canonical script used to generate the openapi.json file in this repo? I tried using scripts/gen_openapi.sh in the Vault repo, however that doesn't seem to handle mounting both kv-v1 and kv-v2 correctly.

scripts/gen_openapi.sh -p is the script I've been using and it seems to be generating both kv-v1 and kv-v2 mounts. I used the following to verify:

cat scripts/openapi.json | jq -c '.paths | to_entries[] | .key as $key | .value | to_entries[] | select(.key == "get" or .key == "post" or .key == "list" or .key == "delete") | [$key, .key, .value.operationId]' | grep kv-v1
["/{kv_mount_path}/{path}","get","kv-v1-read"]
["/{kv_mount_path}/{path}","post","kv-v1-write"]
["/{kv_mount_path}/{path}","delete","kv-v1-delete"]

cat scripts/openapi.json | jq -c '.paths | to_entries[] | .key as $key | .value | to_entries[] | select(.key == "get" or .key == "post" or .key == "list" or .key == "delete") | [$key, .key, .value.operationId]' | grep kv-v2
["/{secret_mount_path}/config","get","kv-v2-read-configuration"]
["/{secret_mount_path}/config","post","kv-v2-configure"]
["/{secret_mount_path}/data/{path}","get","kv-v2-read"]
["/{secret_mount_path}/data/{path}","post","kv-v2-write"]
["/{secret_mount_path}/data/{path}","delete","kv-v2-delete"]
["/{secret_mount_path}/delete/{path}","post","kv-v2-delete-versions"]
["/{secret_mount_path}/destroy/{path}","post","kv-v2-destroy-versions"]
["/{secret_mount_path}/metadata/{path}","get","kv-v2-read-metadata"]
["/{secret_mount_path}/metadata/{path}","post","kv-v2-write-metadata"]
["/{secret_mount_path}/metadata/{path}","delete","kv-v2-delete-metadata"]
["/{secret_mount_path}/subkeys/{path}","get","kv-v2-read-subkeys"]
["/{secret_mount_path}/undelete/{path}","post","kv-v2-undelete-versions"]

If you are having issues with the script, let me know and I can put together the PR here :)

[maxb]

Hmm... I agree the script from the Vault repo does generate the output you've quoted above... but this output is different from the openapi.json currently checked into the vault-client-go and -dotnet repositories: (note the path parameters)

$ cat openapi.json | jq -c '.paths | to_entries[] | .key as $key | .value | to_entries[] | select(.key == "get" or .key == "post" or .key == "list" or .key == "delete") | [$key, .key, .value.operationId]' | grep kv-v1
["/{kv-v1_mount_path}/{path}","get","kv-v1-read"]
["/{kv-v1_mount_path}/{path}","post","kv-v1-write"]
["/{kv-v1_mount_path}/{path}","delete","kv-v1-delete"]

$ cat openapi.json | jq -c '.paths | to_entries[] | .key as $key | .value | to_entries[] | select(.key == "get" or .key == "post" or .key == "list" or .key == "delete") | [$key, .key, .value.operationId]' | grep kv-v2
["/{kv-v2_mount_path}/config","get","kv-v2-read-configuration"]
["/{kv-v2_mount_path}/config","post","kv-v2-configure"]
["/{kv-v2_mount_path}/data/{path}","get","kv-v2-read"]
["/{kv-v2_mount_path}/data/{path}","post","kv-v2-write"]
["/{kv-v2_mount_path}/data/{path}","delete","kv-v2-delete"]
["/{kv-v2_mount_path}/delete/{path}","post","kv-v2-delete-versions"]
["/{kv-v2_mount_path}/destroy/{path}","post","kv-v2-destroy-versions"]
["/{kv-v2_mount_path}/metadata/{path}","get","kv-v2-read-metadata"]
["/{kv-v2_mount_path}/metadata/{path}","post","kv-v2-write-metadata"]
["/{kv-v2_mount_path}/metadata/{path}","delete","kv-v2-delete-metadata"]
["/{kv-v2_mount_path}/subkeys/{path}","get","kv-v2-read-subkeys"]
["/{kv-v2_mount_path}/undelete/{path}","post","kv-v2-undelete-versions"]

I guess this difference might be minor enough (I think) to not affect the generated APIs, but it causes a fair amount of noise in the diff between the old and new openapi.json (which then makes it harder to plan a suitable changelog).

I think the explicit names currently in the checked in generated file (kv-v1 and kv-v2) are better than the names from the scripts/gen_openapi.sh script (kv and secret).

Shall I put in a PR to the Vault repo updating gen_openapi.sh to use those explicit names?

[averche]

Shall I put in a PR to the Vault repo updating gen_openapi.sh to use those explicit names?

That would be lovely, thanks!

[averche]

One thing I just remembered, the last time I regenerated openapi.json, I built and loaded the plugins locally since at the time the latest operation ID changes had not been published or imported into vault yet. This might explain the difference in the mount path :)

[averche]

I ended up implementing the simple fix above, let me know if that's not what you had in mind :)

[maxb]

Looks good to me. My version, which was still sitting unsubmitted on my laptop as I ended up having to travel to London yesterday, used hyphens not underscores - I commented on your PR.

I'd also spotted a few other minor bits and pieces in the script too, so I dropped the parts you'd already PRed from mine, and submitted as https://github.com/hashicorp/vault/pull/21568

[maxb]

188 opened to act on the results of this discussion.