https:// url in call triggered called to http://

teb510 commented 3 months ago

i've noticed a strange behavior for certain public apis on the internet.

the following config will return a 301 for the CALL block:

_format_version: "1.1"

services:
- name: demo
  url: http://httpbin.org
  routes:
  - name: my-route
    paths:
    - /headers
    strip_path: false
    filter_chains:
    - filters:
      - name: datakit
        config:
          debug: true
          nodes:

          - name: FIRST
            type: call
            url: https://catfact.ninja/fact
          - inputs:
            - FIRST
            - request_headers
            name: JOIN
            output: RESP
            template: |
              {
                 "my_state": {{ FIRST.ip }}
              }
            type: template
          - name: MY_HEADERS
            output: response_headers
            template: |
              {
                "Content-Type": "application/json",
                "X-Hello": "World",
                "X-Foo": "Bar"
              }
            type: template
          - name: RESP
            status: 201
            type: response
            warn_headers_sent: false

however curl works just fine (as does the browser)

curl https://catfact.ninja/fact                              
{"fact":"A cat's smell is their strongest sense, and they rely on this leading sense to identify people and objects; a feline's sense of smell is 14x better than a human's.","length":163}%

teb510 commented 3 months ago

debug is:

{
    "action": "value",
    "name": "FIRST",
    "type": "raw",
    "value": "<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body>\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>nginx/1.24.0</center>\r\n</body>\r\n</html>\r\n"
  },

hishamhm commented 3 months ago

@teb510 The strange behavior you were observing was caused by a combination of two things:

the "call" node was not forwarding the HTTPS scheme correctly. This was a bug, and it's fixed in 0.1.1. If you try again with 0.1.1, https://catfact.ninja/fact now works as expected.
the "call" node does not automatically follow HTTP redirects. This would be a new feature. For a general-purpose API client, one could argue either way. (But then again, the list of potential features in a general-purpose HTTP client is huge).

I'm inclined to close this issue since the example given here is now fixed. But one could consider HTTP 301 auto-redirect support to be a separate feature request.

Kong / datakit

https:// url in call triggered called to http:// #17