Rest connector not working for POST method when authorization header and content type are set in additional headers.

[lijithomas88]

I am trying out Data Factory to ingest data from Rest endpoint to DataLake. Tried out Get method after setting authorization Bearer in additional headers and it worked. Post method is not working when I set authorization Bearer token and content type. PFB details. I have set 1) request body contents 2) Content-type as application/json and authorization Bearer token in additional headers.
I get a error response when I click on Preview Data button- Your request was made with invalid credentials. If I click on same button once again, then I get data validation failed error. Same request is working from postman .. Details of data factory - ADF v2. I am using a REST connector.

1) Rest endpoint which I am trying out - https://gorest.co.in/rest-console.html POST - https://gorest.co.in/public-api/users Bearer token gets expired for this endpoint from time to time, so I change the token for rest source in datafactory. Even then this is not working. sample request - { "first_name":"jijo", "last_name":"thomas", "email":"jijovayalumkal28@gmail.com", "gender":"male" }

2) Tried out Get endpoint from datafactory and it worked. GET - https://gorest.co.in/public-api/users

3) Also I tried a different endpoint which does n't require any bearer token for post method. It also worked from data factory.

Please let me know if I am missing anything here.

[fhljys]

Are you still experiencing this?

[travis-hofmeister-quisitive]

I am experiencing the same behavior when having to send a bearer token into the Copy Activity.

[yew]

I am experiencing the same behavior when having to send a bearer token into the Copy Activity.

Hi @travis-hofmeister-quisitive , here is Guosen from ADF team. I have tested that ADF can correctly handle "authorization" header. I made a test HTTP server, which will list all headers in the request, and here is the result.

Please check if the token is valid at that time, and only the "authorization" header is sufficient to complete the auth process.

[yew]

@lijithomas88 , I have done some investigation on the issue, here is the conclusion: First, there is a difference between GET and POST method, when using GET method, ADF will not send request body. And then, ADF will decide whether to include the “content-type” header in the request based on whether the request body is provided.

In your scenario, GET can succeed, but POST failed. I think that ADF included “content-type” header in the POST request, which ultimately caused your server to not work properly.

[travis-hofmeister-quisitive]

@yew It looks like my issue might not be exactly this. The REST service I am trying to connect to requires a separate login activity which presents a cookie that needs to be attached to subsequent requests.

headers = { "cookie": "AZT=~<cookie value received from prior web activity>", "Content-Type": "application/json" }

[yew]

@yew It looks like my issue might not be exactly this. The REST service I am trying to connect to requires a separate login activity which presents a cookie that needs to be attached to subsequent requests.

headers = { "cookie": "AZT=~<cookie value received from prior web activity>", "Content-Type": "application/json" }

Glad to hear that you figured it out.

[travis-hofmeister-quisitive]

@yew oh but the issue still exists. I am not sure if it is a bug in the copy data activity, but it doesn't seem to allow you to pass a cookie into the headers.

[yew]

@travis-hofmeister-quisitive , based on my test, "Cookie" is not allowed in REST connector, but it is allowed in HTTP connector. You can try that if it can meet your requirements.

[slehtola]

@travis-hofmeister-quisitive , based on my test, "Cookie" is not allowed in REST connector, but it is allowed in HTTP connector. You can try that if it can meet your requirements.

I have been banging my head to wall with this and finally I tested both GET and POST methods against Pipedream API and it seems to drop "Cookie" header with both methods. I have no idea how you can make GET method succeed.

Dropping "Cookie" header without mentioning it in documentation seems to be quite silly idea. It causes extra debugging time.