search

witnet / witnet-requests-js

Witnet Requests Javascript Library
MIT License
5 stars 11 forks source link

Wrong custom header value #84

Open fewensa opened 1 year ago

fewensa commented 1 year ago

When i try to call post request with custom header follow docs https://docs.witnet.io/smart-contracts/witnet-web-oracle/make-a-post-request

const testPostSource = new Witnet.HttpPostSource(
    "https://httpbin.org/post",
    "This is the request body",
    {
        "Header-Name": "Header-Value"
    }
)
    .parseJsonMap()
    .getMap("headers")
    .getString("Header-Name")

got this response

 |      Execution trace:
 │        [0] HTTP-POST -> String: "{\n  \"args\": {}, \n  \"data\": \"This is the request body\", \n  \"files\": {}, \n  \"form\": {}, \n  \"headers\": {\n    \"Accept\": \"*/*\", \n    \"Accept-Encoding\": \"deflate, gzip\", \n    \"Content-Length\": \"24\", \n    \"Content-Type\": \"[\\\"application/json\\\"]\", \n    \"Header-Name\": \"[\\\"Header-Value\\\"]\", \n    \"Host\": \"httpbin.org\", \n    \"User-Agent\": \"[\\\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36 Edg/92.0.902.78\\\"]\", \n    \"X-Amzn-Trace-Id\": \"Root=1-646dd9b0-20d177407637005c73bab8d1\"\n  }, \n  \"json\": null, \n  \"origin\": \"180.109.166.147\", \n  \"url\": \"https://httpbin.org/post\"\n}\n"
 │        [1] .parseJSONMap() -> Map: {"args":{"RadonMap":{}},"data":{"RadonString":"This is the request body"},"files":{"RadonMap":{}},"form":{"RadonMap":{}},"headers":{"RadonMap":{"Accept":{"RadonString":"*/*"},"Accept-Encoding":{"RadonString":"deflate, gzip"},"Content-Length":{"RadonString":"24"},"Content-Type":{"RadonString":"[\"application/json\"]"},"Header-Name":{"RadonString":"[\"Header-Value\"]"},"Host":{"RadonString":"httpbin.org"},"User-Agent":{"RadonString":"[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36 Edg/92.0.902.78\"]"},"X-Amzn-Trace-Id":{"RadonString":"Root=1-646dd9b0-20d177407637005c73bab8d1"}}},"origin":{"RadonString":"180.109.166.147"},"url":{"RadonString":"https://httpbin.org/post"}}
 |      Result: Map: {"args":{"RadonMap":{}},"data":{"RadonString":"This is the request body"},"files":{"RadonMap":{}},"form":{"RadonMap":{}},"headers":{"RadonMap":{"Accept":{"RadonString":"*/*"},"Accept-Encoding":{"RadonString":"deflate, gzip"},"Content-Length":{"RadonString":"24"},"Content-Type":{"RadonString":"[\"application/json\"]"},"Header-Name":{"RadonString":"[\"Header-Value\"]"},"Host":{"RadonString":"httpbin.org"},"User-Agent":{"RadonString":"[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36 Edg/92.0.902.78\"]"},"X-Amzn-Trace-Id":{"RadonString":"Root=1-646dd9b0-20d177407637005c73bab8d1"}}},"origin":{"RadonString":"180.109.166.147"},"url":{"RadonString":"https://httpbin.org/post"}}
 │
 │  ┌────────────────────────────────────────────────┐
 ├──┤ Aggregation stage                              │
 │  ├────────────────────────────────────────────────┤
 │  │ Execution time: 0.22196 ms                     │
 │  │ Result is Map: {"args":{"RadonMap":{}},"data":{"RadonString":"This is the request body"},"files":{"RadonMap":{}},"form":{"RadonMap":{}},"headers":{"RadonMap":{"Accept":{"RadonString":"*/*"},"Accept-Encoding":{"RadonString":"deflate, gzip"},"Content-Length":{"RadonString":"24"},"Content-Type":{"RadonString":"[\"application/json\"]"},"Header-Name":{"RadonString":"[\"Header-Value\"]"},"Host":{"RadonString":"httpbin.org"},"User-Agent":{"RadonString":"[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36 Edg/92.0.902.78\"]"},"X-Amzn-Trace-Id":{"RadonString":"Root=1-646dd9b0-20d177407637005c73bab8d1"}}},"origin":{"RadonString":"180.109.166.147"},"url":{"RadonString":"https://httpbin.org/post"}}│
 │  └────────────────────────────────────────────────┘
 │  
 │  ┌────────────────────────────────────────────────┐
 └──┤ Tally stage                                    │
    ├────────────────────────────────────────────────┤
    │ Execution time: 0.175298 ms                    │
    │ Result is Map: {"args":{"RadonMap":{}},"data":{"RadonString":"This is the request body"},"files":{"RadonMap":{}},"form":{"RadonMap":{}},"headers":{"RadonMap":{"Accept":{"RadonString":"*/*"},"Accept-Encoding":{"RadonString":"deflate, gzip"},"Content-Length":{"RadonString":"24"},"Content-Type":{"RadonString":"[\"application/json\"]"},"Header-Name":{"RadonString":"[\"Header-Value\"]"},"Host":{"RadonString":"httpbin.org"},"User-Agent":{"RadonString":"[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36 Edg/92.0.902.78\"]"},"X-Amzn-Trace-Id":{"RadonString":"Root=1-646dd9b0-20d177407637005c73bab8d1"}}},"origin":{"RadonString":"180.109.166.147"},"url":{"RadonString":"https://httpbin.org/post"}}│

The reason is some where changed header value

{
    "args": {
        "RadonMap": {}
    },
    "data": {
        "RadonString": "This is the request body"
    },
    "files": {
        "RadonMap": {}
    },
    "form": {
        "RadonMap": {}
    },
    "headers": {
        "RadonMap": {
            "Accept": {
                "RadonString": "*/*"
            },
            "Accept-Encoding": {
                "RadonString": "deflate, gzip"
            },
            "Content-Length": {
                "RadonString": "24"
            },
            "Content-Type": {
                "RadonString": "[\"application/json\"]"
            },
            "Header-Name": {
                "RadonString": "[\"Header-Value\"]"
            },
            "Host": {
                "RadonString": "httpbin.org"
            },
            "User-Agent": {
                "RadonString": "[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36 Edg/92.0.902.78\"]"
            },
            "X-Amzn-Trace-Id": {
                "RadonString": "Root=1-646dd9b0-20d177407637005c73bab8d1"
            }
        }
    },
    "origin": {
        "RadonString": "180.109.166.147"
    },
    "url": {
        "RadonString": "https://httpbin.org/post"
    }
}

The head value changed to ["Header-Value"] and ["application/json"] not raw value.

aesedepece commented 1 year ago

Hi @fewensa thanks for reporting! I'm trying to reproduce this error and will keep you posted of any findings. Stay tuned!

aesedepece commented 1 year ago

For a start, I have updated the documentation so it mentions .parseJSONMap, not .parseJsonMap.

aesedepece commented 1 year ago

@fewensa I can see that header values are being changed to arrays containing the actual value as their only entry.

I believe this is done internally by the library we use for HTTP queries. I'll investigate some more and confirm where is this conversion happening.

I would also love to know more about your specific use case so that we can prototype the best way to deal with this.

aesedepece commented 1 year ago

@fewensa For what it's worth, you can simply add this at the end of the retrieval script to get rid of the extra brackets:

    .parseJSONArray()
    .getString(0)

Here's the full execution trace:


╔════════════════════════════════════════════╗
║ Witnet query local execution report        ║
╚╤═══════════════════════════════════════════╝
 │
 │  ┌────────────────────────────────────────────────┐
 ├──┤ Retrieval stage                                │
 │  ├────────────────────────────────────────────────┤
 │  │ Number of retrieved data sources: 1            │
 │  └┬───────────────────────────────────────────────┘
 │   │
 │   └─[ Source #0 (httpbin.org) ]
 |      Method: HTTP-POST
 |      Complete URL: https://httpbin.org/post
 |      Headers: 
 |        "Header-Name": "Header-Value"
 |      Body: This is the request body
 |      Number of executed operators: 5
 |      Execution time: 0.056541 ms
 |      Execution trace:
 │        [0] HTTP-POST -> String: "{\n  \"args\": {}, \n  \"data\": \"This is the request body\", \n  \"files\": {}, \n  \"form\": {}, \n  \"headers\": {\n    \"Accept\": \"*/*\", \n    \"Accept-Encoding\": \"deflate, gzip\", \n    \"Content-Length\": \"24\", \n    \"Content-Type\": \"[\\\"application/octet-stream\\\"]\", \n    \"Header-Name\": \"[\\\"Header-Value\\\"]\", \n    \"Host\": \"httpbin.org\", \n    \"User-Agent\": \"[\\\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.2 Safari/605.1.15\\\"]\", \n    \"X-Amzn-Trace-Id\": \"Root=1-646e1a89-72de8699175814593f144d0d\"\n  }, \n  \"json\": null, \n  \"origin\": \"139.47.74.45\", \n  \"url\": \"https://httpbin.org/post\"\n}\n"
 │        [1] .parseJSONMap() -> Map: {"args":{"RadonMap":{}},"data":{"RadonString":"This is the request body"},"files":{"RadonMap":{}},"form":{"RadonMap":{}},"headers":{"RadonMap":{"Accept":{"RadonString":"*/*"},"Accept-Encoding":{"RadonString":"deflate, gzip"},"Content-Length":{"RadonString":"24"},"Content-Type":{"RadonString":"[\"application/octet-stream\"]"},"Header-Name":{"RadonString":"[\"Header-Value\"]"},"Host":{"RadonString":"httpbin.org"},"User-Agent":{"RadonString":"[\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.2 Safari/605.1.15\"]"},"X-Amzn-Trace-Id":{"RadonString":"Root=1-646e1a89-72de8699175814593f144d0d"}}},"origin":{"RadonString":"139.47.74.45"},"url":{"RadonString":"https://httpbin.org/post"}}
 │        [2] .getMap(headers) -> Map: {"Accept":{"RadonString":"*/*"},"Accept-Encoding":{"RadonString":"deflate, gzip"},"Content-Length":{"RadonString":"24"},"Content-Type":{"RadonString":"[\"application/octet-stream\"]"},"Header-Name":{"RadonString":"[\"Header-Value\"]"},"Host":{"RadonString":"httpbin.org"},"User-Agent":{"RadonString":"[\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.2 Safari/605.1.15\"]"},"X-Amzn-Trace-Id":{"RadonString":"Root=1-646e1a89-72de8699175814593f144d0d"}}
 │        [3] .getString(Header-Name) -> String: "[\"Header-Value\"]"
 │        [4] .parseJSONArray() -> Array: [{"RadonString":"Header-Value"}]
 │        [5] .getString(0) -> String: "Header-Value"
 |      Result: String: "Header-Value"
 │
 │  ┌────────────────────────────────────────────────┐
 ├──┤ Aggregation stage                              │
 │  ├────────────────────────────────────────────────┤
 │  │ Execution time: 0.00653 ms                     │
 │  │ Result is String: "Header-Value"               │
 │  └────────────────────────────────────────────────┘
 │  
 │  ┌────────────────────────────────────────────────┐
 └──┤ Tally stage                                    │
    ├────────────────────────────────────────────────┤
    │ Execution time: 0.00204 ms                     │
    │ Result is String: "Header-Value"               │
    └────────────────────────────────────────────────┘
fewensa commented 1 year ago

Hi @aesedepece Thanks for your help, I think now don't care how to parse response, because this has no value, this data is the request data structure returned by httpbin.org.

 │        [0] HTTP-POST -> String: "{\n  \"args\": {}, \n  \"data\": \"This is the request body\", \n  \"files\": {}, \n  \"form\": {}, \n  \"headers\": {\n    \"Accept\": \"*/*\", \n    \"Accept-Encoding\": \"deflate, gzip\", \n    \"Content-Length\": \"24\", \n    \"Content-Type\": \"[\\\"application/octet-stream\\\"]\", \n    \"Header-Name\": \"[\\\"Header-Value\\\"]\", \n    \"Host\": \"httpbin.org\", \n    \"User-Agent\": \"[\\\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.2 Safari/605.1.15\\\"]\", \n    \"X-Amzn-Trace-Id\": \"Root=1-646e1a89-72de8699175814593f144d0d\"\n  }, \n  \"json\": null, \n  \"origin\": \"139.47.74.45\", \n  \"url\": \"https://httpbin.org/post\"\n}\n"

\"Content-Type\": \"[\\\"application/octet-stream\\\"]\"

As long as you still see an array here, it means that the request is wrong.

Wrong header

$ curl -vvv -X POST -H 'Content-Type: ["application/json"]' https://pangolin-rpc.darwinia.network -d '{"jsonrpc":"2.0","id":"1","method":"chain_getFinalizedHead"}'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 198.18.2.221:443...
* Connected to pangolin-rpc.darwinia.network (198.18.2.221) port 443 (#0)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
* using HTTP/1.1
> POST / HTTP/1.1
> Host: pangolin-rpc.darwinia.network
> User-Agent: curl/8.0.1
> Accept: */*
> Content-Type: ["application/json"]
> Content-Length: 60
>
< HTTP/1.1 415 Unsupported Media Type
< Date: Wed, 24 May 2023 09:36:20 GMT
< Content-Type: text/plain
< Content-Length: 81
< Connection: keep-alive
< access-control-allow-origin: *
< vary: origin
< vary: access-control-request-method
< vary: access-control-request-headers
< access-control-expose-headers: *
< CF-Cache-Status: DYNAMIC
< Server: cloudflare
< CF-RAY: 7cc48dbc8acd0499-HKG
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
<
Supplied content type is not allowed. Content-Type: application/json is required
* Connection #0 to host pangolin-rpc.darwinia.network left intact

Right header

$ curl -vvv -X POST -H 'Content-Type: application/json' https://pangolin-rpc.darwinia.network -d '{"jsonrpc":"2.0","id":"1","method":"chain_getFinalizedHead"}'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 198.18.2.221:443...
* Connected to pangolin-rpc.darwinia.network (198.18.2.221) port 443 (#0)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
* using HTTP/1.1
> POST / HTTP/1.1
> Host: pangolin-rpc.darwinia.network
> User-Agent: curl/8.0.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 60
>
< HTTP/1.1 200 OK
< Date: Wed, 24 May 2023 09:36:52 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 104
< Connection: keep-alive
< access-control-allow-origin: *
< vary: origin
< vary: access-control-request-method
< vary: access-control-request-headers
< access-control-expose-headers: *
< CF-Cache-Status: DYNAMIC
< Server: cloudflare
< CF-RAY: 7cc48e82cbc50500-HKG
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
<
{"jsonrpc":"2.0","result":"0x18936a14745b23b1aa9205d486db58181715f8b84cd57c2e4dafd49f133c8eba","id":"1"}

I believe this is done internally by the library we use for HTTP queries. I'll investigate some more and confirm where is this conversion happening.

I think so, I tried to investigate the code of witnet-requests-js. I didn't find where the http request was called. https://github.com/witnet/witnet-requests-js/blob/6a03a6678f716ff4fe35bcefd44bd52b6ce20cab/src/lib/radon/stages.js#L26

fewensa commented 1 year ago

I think there changed header value to array. then send data direct use sede_json::to_string, send raw json string to server.

https://github.com/witnet/witnet-rust/blob/7e00113eae71ced5119474f7ebc4e74d23b5b34c/rad/src/lib.rs#L369-L376

https://docs.rs/http-types/2.12.0/http_types/headers/trait.ToHeaderValues.html#tymethod.to_header_values

https://docs.rs/http-types/2.12.0/http_types/headers/struct.HeaderValues.html

fewensa commented 1 year ago

https://cloud.google.com/apigee/docs/api-platform/antipatterns/multi-value-http-headers?hl=en