Kong / deck

decK: Configuration management and drift detection for Kong
https://docs.konghq.com/deck/overview
Apache License 2.0
444 stars 128 forks source link

Add `kong2tf` command #1391

Closed mheap closed 2 months ago

mheap commented 2 months ago

Adds the kong2tf command to convert a deck file to Terraform.

If --generate-imports-for-control-plane-id is provided, import blocks will be added to Terraform to adopt existing resources.

If --ignore-credential-changes is added, any credentials will be ignored until they are destroyed and recreated.

Tested with the following deck file which contains one of every entity, including plugin scopes.

_format_version: "3.0"
_konnect:
  control_plane_name: CP With Everything
ca_certificates:
  - cert: |-
      -----BEGIN CERTIFICATE-----
      MIIBwDCCAUagAwIBAgIUMDCK/a+EH6a9PvQeyFwHDALYrx0wCgYIKoZIzj0EAwIw
      FzEVMBMGA1UEAwwMa29uZ190ZXN0X2NhMB4XDTI0MDkwODE3NDUxMFoXDTI3MDkw
      ODE3NDUxMFowFzEVMBMGA1UEAwwMa29uZ190ZXN0X2NhMHYwEAYHKoZIzj0CAQYF
      K4EEACIDYgAEYEP63cMf47Xf6QcH9QT8ehjxBAbfF68PdCXOPgKZdSFvRYs+JZ+Y
      dIhWV4M06WiEP8Exn352xuzgZc4mVT9B6SpAsp+hsyJ9ZkuQkyb0lC1XodP5+ZlF
      yP0dZHkPYQ5Po1MwUTAdBgNVHQ4EFgQUYS5tJ4WfzVaNsB2oYrJOEf2WJ/UwHwYD
      VR0jBBgwFoAUYS5tJ4WfzVaNsB2oYrJOEf2WJ/UwDwYDVR0TAQH/BAUwAwEB/zAK
      BggqhkjOPQQDAgNoADBlAjBCIF9CB1CuRkdRpdjHq/x2G1TJkj622L80+Emvkctw
      IdyXs8Vp2gHrLjpbBAKoWAgCMQD0E/fjMNr0FAce66MlpKSxdKbeUGt7Zv3Gauld
      3jeXOUCitQiKp60wg0IhAslBoEM=
      -----END CERTIFICATE-----
    cert_digest: ef407ce6a2f2c0b68c24a31cf7c75dfd6fd850d631775545f17b6d67b4b93c64
    id: c878251f-35b1-403d-987f-89554e98f8bb
certificates:
  - cert: |-
      -----BEGIN CERTIFICATE-----
      MIIBxzCCAUygAwIBAgIUfCAyT/813MlPQFqasFetYcAdPVYwCgYIKoZIzj0EAwIw
      GjEYMBYGA1UEAwwPa29uZ19jbHVzdGVyaW5nMB4XDTI0MDkwODE3NDQzNloXDTI3
      MDkwODE3NDQzNlowGjEYMBYGA1UEAwwPa29uZ19jbHVzdGVyaW5nMHYwEAYHKoZI
      zj0CAQYFK4EEACIDYgAEygTwKjbWvTvj0AlmNfxXFaPFbOctV0LA9D4aUzdG8dPg
      3bZEco1wWKNY1ucs7sFO51T6oPuI77nHv+owCMLbS7Cx2m7Tdkdibp63ejggiogZ
      U6er9tNWI7kF3NJos8mdo1MwUTAdBgNVHQ4EFgQUrpk2dywGsqDcrr1lNPhEQT7n
      3OMwHwYDVR0jBBgwFoAUrpk2dywGsqDcrr1lNPhEQT7n3OMwDwYDVR0TAQH/BAUw
      AwEB/zAKBggqhkjOPQQDAgNpADBmAjEAuotEtPBtrU1loNBwxpWWbeJ3cH8yJVb7
      jtAqcs3KC2nZa+gDbzCQgePjL1nQ7M2yAjEAootOUs9SiFh89TbjEpjYX0JTV1sX
      3aHopOi30aG8yuGRkC8NAn85+YpjVvvk+dJC
      -----END CERTIFICATE-----
    id: f9a53112-2e73-4fc4-bb5f-4be86071e881
    key: |-
      -----BEGIN PRIVATE KEY-----
      MIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDCjxfLtYRbO+4501Zx4
      bwN2BK/yDugQ/p1uA+yz6Z02oOPbPq0eWl5iZTp7LsxpHP+hZANiAATKBPAqNta9
      O+PQCWY1/FcVo8Vs5y1XQsD0PhpTN0bx0+DdtkRyjXBYo1jW5yzuwU7nVPqg+4jv
      uce/6jAIwttLsLHabtN2R2Junrd6OCCKiBlTp6v201YjuQXc0mizyZ0=
      -----END PRIVATE KEY-----
    snis:
      - id: 68bd0d63-6ec7-4e6e-aa03-e7093e272c5b
        name: foo.example.com
    tags:
      - kong-clustering
consumer_groups:
  - id: 56d3cf4b-defc-4f55-a0d9-078b083bfe63
    name: gold
    plugins:
      - config:
          day: null
          error_code: 429
          error_message: API rate limit exceeded
          fault_tolerant: true
          header_name: null
          hide_client_headers: false
          hour: 10000
          limit_by: consumer
          minute: null
          month: null
          path: null
          policy: local
          redis:
            database: 0
            host: null
            password: null
            port: 6379
            server_name: null
            ssl: false
            ssl_verify: false
            timeout: 2000
            username: null
          redis_database: 0
          redis_host: null
          redis_password: null
          redis_port: 6379
          redis_server_name: null
          redis_ssl: false
          redis_ssl_verify: false
          redis_timeout: 2000
          redis_username: null
          second: null
          sync_rate: -1
          year: null
        id: 793f5bd6-8ec4-4fff-afeb-9787be181b49
        name: rate-limiting
    tags:
      - gold
      - alice
consumers:
  - acls:
      - group: group-a
        id: b482b4f7-069a-4ca8-876c-4e9dadbdcecc
        tags:
          - alice
          - group-a
    basicauth_credentials:
      - id: 43d95734-64cd-4c11-929d-c26d9e7f305a
        password: 60cf54018b3a0264fb19517f8214dddaa4d83ab7
        username: alice
    custom_id: alice123
    groups:
      - id: 56d3cf4b-defc-4f55-a0d9-078b083bfe63
        name: gold
        tags:
          - gold
          - alice
    hmacauth_credentials:
      - id: c85a1ce8-37f6-4114-a314-fcedae367b38
        secret: alicesecret
        username: aliceuser
    id: bdd14928-4c9c-4572-a266-29a6e1602663
    jwt_secrets:
      - algorithm: HS256
        id: ce19c10f-4a06-4016-b22f-07216c9a56b2
        key: alicejwtkey
        secret: alicejwtsecret
    keyauth_credentials:
      - id: faf3581a-d131-42d9-a0de-7c6dee9a072a
        key: alicekey
        tags:
          - alice
    plugins:
      - config:
          day: null
          error_code: 429
          error_message: API rate limit exceeded
          fault_tolerant: true
          header_name: null
          hide_client_headers: false
          hour: 10
          limit_by: consumer
          minute: null
          month: null
          path: null
          policy: local
          redis:
            database: 0
            host: null
            password: null
            port: 6379
            server_name: null
            ssl: false
            ssl_verify: false
            timeout: 2000
            username: null
          redis_database: 0
          redis_host: null
          redis_password: null
          redis_port: 6379
          redis_server_name: null
          redis_ssl: false
          redis_ssl_verify: false
          redis_timeout: 2000
          redis_username: null
          second: null
          sync_rate: -1
          year: null
        enabled: true
        id: bf7626c4-8383-4cda-b3f3-80fa75bd2121
        name: rate-limiting
        protocols:
          - grpc
          - grpcs
          - http
          - https
    username: alice
  - custom_id: bob456
    id: b2e8673d-8ed5-49cf-be9c-b3c93640c7a6
    tags:
      - bob
    username: bob
plugins:
  - config:
      anonymous: null
      hide_credentials: false
      key_in_body: false
      key_in_header: true
      key_in_query: true
      key_names:
        - apikey-global
      realm: null
      run_on_preflight: true
    enabled: true
    id: 600c38c5-da4b-4ec5-b3b8-70fc279a6503
    name: key-auth
    protocols:
      - grpc
      - grpcs
      - http
      - https
      - ws
      - wss
  - config:
      anonymous: null
      hide_credentials: false
      key_in_body: false
      key_in_header: true
      key_in_query: true
      key_names:
        - apikey-in-route
      realm: null
      run_on_preflight: true
    enabled: true
    id: efa261d5-7978-4c8f-84e5-c6b0387ba37c
    name: key-auth
    protocols:
      - grpc
      - grpcs
      - http
      - https
      - ws
      - wss
    route: route-in-service
    service: httpbin
routes:
  - https_redirect_status_code: 426
    id: 7a8141cc-aade-49b2-a6c0-a95b624c1b6f
    name: route-standalone
    path_handling: v0
    paths:
      - /
    preserve_host: false
    protocols:
      - http
      - https
    regex_priority: 0
    request_buffering: true
    response_buffering: true
    strip_path: true
services:
  - connect_timeout: 60000
    enabled: true
    host: httpbin.org
    id: 0cc39546-f5c9-4aae-898c-c63ef5b4a33b
    name: httpbin
    plugins:
      - config:
          anonymous: null
          hide_credentials: false
          key_in_body: false
          key_in_header: true
          key_in_query: true
          key_names:
            - apikey-service
          realm: null
          run_on_preflight: true
        enabled: true
        id: 44a6f30a-23d6-4522-9ce8-f56243a054e3
        name: key-auth
        protocols:
          - grpc
          - grpcs
          - http
          - https
          - ws
          - wss
    port: 443
    protocol: https
    read_timeout: 60000
    retries: 5
    routes:
      - https_redirect_status_code: 426
        id: 9ac61ee8-4de7-4d75-82af-b7624ec5fb33
        name: route-in-service
        path_handling: v0
        paths:
          - /
        preserve_host: false
        protocols:
          - http
          - https
        regex_priority: 0
        request_buffering: true
        response_buffering: true
        strip_path: true
    write_timeout: 60000
upstreams:
  - algorithm: round-robin
    hash_fallback: none
    hash_on: none
    hash_on_cookie_path: /
    healthchecks:
      active:
        concurrency: 10
        healthy:
          http_statuses:
            - 200
            - 302
          interval: 0
          successes: 0
        http_path: /
        https_verify_certificate: true
        timeout: 1
        type: http
        unhealthy:
          http_failures: 0
          http_statuses:
            - 429
            - 404
            - 500
            - 501
            - 502
            - 503
            - 504
            - 505
          interval: 0
          tcp_failures: 0
          timeouts: 0
      passive:
        healthy:
          http_statuses:
            - 200
            - 201
            - 202
            - 203
            - 204
            - 205
            - 206
            - 207
            - 208
            - 226
            - 300
            - 301
            - 302
            - 303
            - 304
            - 305
            - 306
            - 307
            - 308
          successes: 0
        type: http
        unhealthy:
          http_failures: 0
          http_statuses:
            - 429
            - 500
            - 503
          tcp_failures: 0
          timeouts: 0
      threshold: 0
    id: d8432ca0-b6c5-4c82-bad8-924c56fb2d04
    name: my-upstream
    slots: 10000
    targets:
      - id: eca40df5-a499-4611-86e8-b377d1b3acdb
        target: a.example.com:443
        weight: 100
      - id: 69f683b9-796f-4117-8461-7bc0a21e5d53
        target: b.example.com:443
        weight: 100
    use_srv_name: false
vaults:
  - config:
      prefix: none
    id: 5370372e-9bab-47c9-a8de-b6af35cfb170
    name: env
    prefix: my-env-vault

Steps to test yourself:

  1. Configure a CP using deck or the UI
  2. Run deck dump --konnect-token $KONNECT_TOKEN --konnect-control-plane-name "YOUR_CP_NAME" --with-id
  3. Convert to Terraform with deck file kong2tf -s ./kong.yaml --generate-imports-for-control-plane-id "0dea9abf-074a-4988-bdd6-c9ea1ea25d4b" --ignore-credential-changes
  4. Run terraform plan. It will show all entities being adopted
CLAassistant commented 2 months ago

CLA assistant check
All committers have signed the CLA.

mheap commented 2 months ago

@Prashansa-K I'm primarily looking for mistakes + potential bugs in this review. I know the code needs a ton of work to be up to standard. I'd like to submit that in a followup as this is a Summit feature.

codecov-commenter commented 2 months ago

Codecov Report

Attention: Patch coverage is 82.83828% with 104 lines in your changes missing coverage. Please review.

Please upload report for BASE (main@fb795ad). Learn more about missing BASE report. Report is 11 commits behind head on main.

Files with missing lines Patch % Lines
cmd/file_kong2tf.go 0.00% 50 Missing :warning:
kong2tf/builder_default_terraform.go 89.14% 32 Missing and 5 partials :warning:
kong2tf/generate_resource.go 91.70% 12 Missing and 4 partials :warning:
cmd/root.go 0.00% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1391 +/- ## ======================================= Coverage ? 29.49% ======================================= Files ? 59 Lines ? 5136 Branches ? 0 ======================================= Hits ? 1515 Misses ? 3512 Partials ? 109 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Prashansa-K commented 2 months ago

@mheap Do we want this command to be in prod before the summit? Or we doing a beta release first?

mheap commented 2 months ago

@Prashansa-K Before summit - the provider is already available as a beta

Prashansa-K commented 2 months ago

@mheap This looks good to me. The presence of exhaustive tests is giving me more confidence on this. Just the lint failure and konnect integration test needs to be checked out.

Prashansa-K commented 2 months ago

Integration test seems like a flake. Could be related to this issue https://github.com/Kong/deck/issues/1365.

mheap commented 2 months ago

@Prashansa-K @battlebyte Can you work together to fix the lint error and release?

Prashansa-K commented 2 months ago

I am working on fixing the integration test errors here: https://github.com/Kong/deck/pull/1396 PR is in review. Once this is merged in main, we can merge it here.

For the release, I will add a changelog with this PR: https://github.com/Kong/deck/pull/1395 which is already there for a pending release item. @battlebyte Could you help me with a changelog entry? Also, I believe now since we have a added a new command, we would have to bump the minor version?

battlebyte commented 2 months ago

@Prashansa-K @mheap PR for the docs here: https://github.com/Kong/docs.konghq.com/pull/7899