Response headers having duplicate keys are getting overriden

[karanpopat]

The values for duplicate keys in the response_headers in the step output are getting overridden.

Example header output using curl -

Key - link

set-cookie: JSESSIONID=DA89ED19D02CDAFD7345931F5E6B1FC4; Path=/; Secure; HttpOnly
link: <https://dev-88719661.okta.com/api/v1/apps?limit=1>; rel="self"
link: <https://dev-88719661.okta.com/api/v1/apps?after=0oa1e5a9p5d7&limit=1>; rel="next"

Flowpipe returns

"Link": "<https://dev-88719661.okta.com/api/v1/apps?limit=1>; rel=\"self\"",

The other link header would be required for pagination

[karanpopat]

pipeline "list_applications" {
  title       = "List Applications"
  description = "Lists all applications."

  param "api_token" {
    type        = string
    description = local.api_token_param_description
    default     = var.api_token
  }

  param "domain" {
    type        = string
    description = local.domain_param_description
    default     = var.domain
  }

  step "http" "list_applications" {
    method = "get"
    url    = "${param.domain}/api/v1/apps?limit=200"
    request_headers = {
      Content-Type  = "application/json"
      Authorization = "SSWS ${param.api_token}"
    }
  }

  output "applications" {
    description = "List of applications."
    value       = step.http.list_applications.response_body
  }
}

[karanpopat]

Original Output

➜  flowpipe-mod-okta git:(initial-mod) curl -i -X GET \
  'https://dev-88719661.okta.com/api/v1/apps?limit=1' \                                                     
  -H 'Authorization: SSWS auth'
HTTP/2 200 
date: Thu, 30 Nov 2023 16:54:00 GMT
content-type: application/json
server: nginx
vary: Accept-Encoding
x-okta-request-id: aaaa
x-xss-protection: 0
p3p: CP="HONK"
set-cookie: sid="";Version=1;Path=/;Max-Age=0
set-cookie: autolaunch_triggered=""; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=/
content-security-policy: default-src 'self' dev-887661.okta.com *.oktacdn.com; connect-src 'self' dev-89661.okta.com dev-88719661-admin.okta.com *.oktacdn.com *.mixpanel.com *.mapbox.com app.pendo.io data.pendo.io pendo-static-5634101do-static-5391521872216064.storage.googleapis.com *.mtls.okta.com https://oinmanager.okta.com data:; script-src 'unsafe-inline' 'unsafe-eval' 'self' okta.com *.oktacdn.com *.tiles.mapbox.com *.mapbox.com app.pendo.io data.pendo.io cdn.pendo.io pendo-static-56341013984.storage.googleapis.com pendo-static-5391521872216064.storage.googleapis.com data: blob:; font-src 'self' dev-887191.okta.com data: *.oktacdn.com fonts.gstatic.com; frame-ancestors 'self'
x-rate-limit-limit: 10
x-rate-limit-remaining: 8
x-rate-limit-reset: 1701363272
cache-control: no-cache, no-store
pragma: no-cache
expires: 0
set-cookie: JSESSIONID=45931F5E6B1FC4; Path=/; Secure; HttpOnly
link: <https://dev-887161.okta.com/api/v1/apps?limit=1>; rel="self"
link: <https://dev-887161.okta.com/api/v1/apps?after=D6p5d7&limit=1>; rel="next"
x-okta-edge-log: rlInfo=PRL_SSWS_TOKEN:PCL_SSWS_TOKEN:CAT_C orgId=00o1e5EdW5d7 dbri=NO DB_TX=0.001 APP=0.364 MOCA=0.045 DB=0.034 clInfo=OC_1:PCL_1
x-content-type-options: nosniff
strict-transport-security: max-age=315360000; includeSubDomains

Flowpipe Output

  "response_headers": {
    "Cache-Control": "no-cache, no-store",
    "Connection": "keep-alive",
    "Content-Security-Policy": "default-src 'self' dev9661.okta.com *.oktacdn.com; connect-src 'self' dev-88719661.okta.googleapis.com pendo-static-5391521816064.storage.googleapis.com data: blob:; font-src 'self' dev-88719661.okta.com data: *.oktacdn.com fonts.gstatic.com; frame-ancestors 'self'",
    "Content-Type": "application/json",
    "Date": "Thu, 30 Nov 2023 16:49:37 GMT",
    "Expires": "0",
    "Link": "<https://dev-887661.okta.com/api/v1/apps?limit=1>; rel=\"self\"",
    "P3p": "CP=\"HONK\"",
    "Pragma": "no-cache",
    "Server": "nginx",
    "Set-Cookie": [
      "sid=\"\";Version=1;Path=/;Max-Age=0",
      "autolaunch_triggered=\"\"; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=/",
      "JSESSIONID=9D33E42018EE; Path=/; Secure; HttpOnly"
    ],
    "Strict-Transport-Security": "max-age=315360000; includeSubDomains",
    "Vary": "Accept-Encoding",
    "X-Content-Type-Options": "nosniff",
    "X-Okta-Edge-Log": "rlInfo=PRL_SSWS_TOKEN:PCL_SSWS_TOKEN:CAT_C orgId=00o19hEdW5d7 dbri=NO MOCA=0.027 APP=0.307 DB=0.032 DB_TX=0.001 clInfo=OC_1:PCL_1",
    "X-Okta-Request-Id": "975447da0882e07ee16f019dc",
    "X-Rate-Limit-Limit": "10",
    "X-Rate-Limit-Remaining": "7",
    "X-Rate-Limit-Reset": "1701362998",
    "X-Xss-Protection": "0"
  },
  "started_at": "2023-11-30T16:49:36.140596Z",
  "status": "200 OK",
  "status_code": 200
}

[vhadianto]

We have a provision for "multi value" fields in response headers. Currently we only treat "set-cookie" as a multi value field. It's easy enough to add "link" here, but I wonder if we should have it configured @johnsmyth

I've added Link in the multi value list with this commit: https://github.com/turbot/flowpipe/commit/2058d5e27a7d195180a7eb2b9e68ce4f1e27df31 so it should work now.

[vhadianto]

We're going to follow https://datatracker.ietf.org/doc/html/rfc2616#section-4.2

Multiple message-header fields with the same field-name MAY be
   present in a message if and only if the entire field-value for that
   header field is defined as a comma-separated list [i.e., #(values)].
   It MUST be possible to combine the multiple header fields into one
   "field-name: field-value" pair, without changing the semantics of the
   message, by appending each subsequent field-value to the first, each
   separated by a comma. 

So for "multi values" we're going to join the values separated with commas.