[BUG] Headers are shown in all lowercase even if they were not

[tremby]

Describe the bug If I send a request to the request bin with an upper or mixed case header it shows up in the interface as all lowercase.

To Reproduce Steps to reproduce the behavior:

1. Send a request to your request bin with a header for example curl -H "MIXEDcase: 1" your-bin-url
2. Look at the headers as shown by the request bin UI
3. See that it appears as mixedcase: 1

Expected behavior It should show the headers exactly as they were sent.

Additional context I do recognize that according to the HTTP spec headers are meant to be case-insensitive. But this does not mean that all systems treat them that way. For example, AWS Lambda Proxy Integration provides the headers in a case-sensitive dictionary at event.headers without normalizing their case. Pipedream can therefore lead me astray when I'm inspecting requests.

[dylburger]

@tremby thanks for the request. Can you describe the specific issue in a little more detail? e.g. are you sending HTTP requests from a Lambda to Pipedream, using the Proxy Integration, and therefore the differences are affecting your tests?

[tremby]

The application I'm in charge of responds to webhooks from one 3rd party site via API Gateway -> proxy integration -> Lambda.

Client recently asked me to have it also respond to webhooks from another 3rd party site. This site has poor documentation. Meanwhile, my client's tech team have locked down my AWS access a little too much so that I can't easily get to the lambda logs. In light of that, to save myself from trial and error debugging I set the 3rd party site's webhook configuration to send to Pipedream rather than to the lambda, so I can see exactly what it's sending, and so that I can make sure I handle the request correctly in the lambda.

Pipedream was reporting that it was sending a signature header, but their docs listed it as Signature. Given that the docs were poor I didn't know which to believe, so I had to trial and error it after all. They were indeed sending Signature but Pipedream was reporting it as signature. Lambda (at least in my configuration) treats headers as case-sensitive, so my code was non-functional first time.

[dylburger]

Very much appreciate the detail!

[dylburger]

@tremby I dug into this more and wanted to share what I found. I believe the requests are being made over HTTP/2, which encodes all headers as lowercase, as per the spec (see this resource). When you send an HTTP request to Pipedream, it's routed through an AWS application load balancer, which converts the headers to lowercase as demanded by HTTP/2.

If you create a new Pipedream workflow, choose the HTTP API trigger, and send an HTTP/1.1 request to its HTTP endpoint, you'll see the case-sensitive, raw headers in steps.trigger.raw_event.headers.

curl -H 'HeLlO: world' -v https://endpoint.m.pipedream.net --http1.1

Overriding the standard behavior of HTTP/2 requests is not something we can easily do, given that intermediate services like AWS ALB and other libs treat HTTP/2 headers case-insensitively. And given that HTTP/1.1 headers are represented in their raw form correctly in workflows, I'm going to go ahead and close this. Let me know if that information helped!

[tremby]

I'm not sure if I'm misunderstanding something.

I'm using requestbin.com -- if I follow your link it looks like I have to sign up, which I have not done.

On the interface I see, it doesn't matter if I send http1.1 or http2, the headers always show up in the UI as lowercase. Via similar requests, the headers show up in Lambda in... well... it seems that Lambda and/or API Gateway is just doing whatever the heck it likes. I modified my lambda to cough up the contents of event.headers and I get this for http2:

{
  ...
  "headers": {
    "Accept": "text/plain", <- I sent this as all-lowercase `accept: text/plain`
    "CloudFront-Forwarded-Proto": "https",
    "CloudFront-Is-Desktop-Viewer": "true",
    "CloudFront-Is-Mobile-Viewer": "false",
    "CloudFront-Is-SmartTV-Viewer": "false",
    "CloudFront-Is-Tablet-Viewer": "false",
    "CloudFront-Viewer-Country": "CA",
    "content-type": "application/json", <- I sent this as CONTENT-TYPE
    "Host": [xxx],
    "mixed": "case", <- I sent this as MiXeD
    "User-Agent": "curl/7.58.0",
    "Via": ...
  }
}

and if http1.1:

{
  ...
  "headers": {
    "Accept": "text/plain",
    "CloudFront-Forwarded-Proto": "https",
    "CloudFront-Is-Desktop-Viewer": "true",
    "CloudFront-Is-Mobile-Viewer": "false",
    "CloudFront-Is-SmartTV-Viewer": "false",
    "CloudFront-Is-Tablet-Viewer": "false",
    "CloudFront-Viewer-Country": "CA",
    "CONTENT-TYPE": "application/json",
    "Host": [xxx],
    "mIxEd": "case",
    "User-Agent": "curl/7.58.0",
    "Via": ...
  }
}

So it seems for http2 requests it's transforming some headers to lowercase but enforcing the case of others, even ones which were sent by me. For http1.1 it keeps the original case for most, but enforces case of others.

But all headers from both types of request sent to Pipedream show up in all lowercase. It still seems like a bug to me.

[dylburger]

@tremby would you be open to using https://pipedream.com for this? This repo tracks requests against that service, which is why I thought you were using that. Free to sign up and use, and the raw headers should be correctly represented there.

[tremby]

I didn't realize they were two different things -- logo on the site I was using says Pipedream. I only heard of it via a link in these docs. (This is the first 3rd-party service, not the one I'm adding support for.)

I'll believe you that the bug doesn't exist in the software you thought I was using. Sorry for posting in the wrong place. It sounds like the two pieces of software are related though -- is there an issue tracker for requestbin.com? I couldn't see any contact information or links to support or feedback or a repo at all on that site.

[dylburger]

Yes, we also operate requestbin.com, but the core of our development work is done on pipedream.com. We don't operate an issue tracker for RequestBin, but I noted it internally.