Can't send Text/CSV request body

[JohnKacz]

So I'm honestly not sure if this is an ExForce issue or Tesla.

I'm trying to use ExForce to send some Bulk API 2.0 requests (example). You can see in step 3 "Uploading your CSV data" the content-type header is text/csv. However, the following fails because when the client is built the JSON middleware is set, and I believe is encoding the csv_data.

ExForce.Client.request(client,
  method: :put,
  url: "jobs/ingest/#{job_id}/batches/",
  headers: ["Content-Type": "text/csv", Accept: "application/json"],
  body: csv_data
)

Would solving this just be a matter of writing a CSV Tesla middleware or is this currently outside the scope of what ExForce can handle?

[chulkilee]

I wasn't aware of such cases.

If the body is already binary (string), Tesla.Middleware.Json seems to 1) pass the body as-is and 2) do not set the content-type header. See https://github.com/teamon/tesla/blob/v1.2.1/lib/tesla/middleware/json.ex#L74

Could you pass the csv_data as string? (already encoded as csv)

[JohnKacz]

I see you're right. I'll do some more testing. Why do you say "do not set the content-type header"?

[JohnKacz]

So I tried it again and got the same error. More details:

ExForce.Client.request(client,
  method: :put,
  url: "jobs/ingest/#{job_id}/batches/",
  headers: ["Content-Type": "text/csv", Accept: "application/json"], # commenting out this line yeilds the same results
  body: csv_data # csv_data is a binary
)

Returns a 409 response with the body:

body: [%{
  "errorCode" => "BULK_API_ERROR",
  "message" => "Wrong content-type for batch (), job is of type: text/csv"
}]

However

HTTPoison.put(
  "#{base_ur}/services/data/v#{version}/jobs/ingest/#{job_id}/batches/",
  csv_data,
  ["Content-Type": "text/csv", Accept: "application/json", authorization: token]
)

Works just fine. So my assumption was that Tesla was doing something to the string before sending it on.

Additionally if I pass a Map instead of a String as csv_data the response error is the same except it specifies the type.

body: [%{
  "errorCode" => "BULK_API_ERROR",
  "message" => "Wrong content-type for batch (application/json), job is of type: text/csv"
}]

So the fact that the error message doesn't give a type for the wrong content-type when passing a String must mean something.

[chulkilee]

First, headers are incorrect format. It should be [{binary(), binary()}] (ref). ["Content-Type": "text/csv"] == [{:"Content-Type", "text/csv"}]

Second, Tesla httpc adopter has a bug that only content-type (all downcase) works. I reported to tesla - https://github.com/teamon/tesla/issues/307

Third, If you pass map or list, then the json middleware find it is "encodable" so it will set the content-type header. (see code)

See this code:

csv_data = "a,b,c\ne,f,g"
client = ExForce.build_client("https://httpbin.org")

{:ok, %{body: %{"headers" => headers, "data" => data}} = resp} =
  ExForce.Client.request(client,
    method: :put,
    url: "/put",
    headers: [{"content-type", "text/csv"}, {"accept", "application/json"}],
    body: csv_data
  )

"data:application/octet-stream;base64," <> body_encoded = data
^csv_data = body_encoded |> Base.decode64!() |> :zlib.gunzip()

# The resp shows only one content-type - but I believe tesla sends multiple content-type
# and httpbin just picks up the first value.
# You can see the data is in json
{:ok, resp} =
  ExForce.Client.request(client,
    method: :put,
    url: "/put",
    headers: [{"content-type", "text/csv"}, {"accept", "application/json"}],
    body: [["a", "b", "c"]]
  )

In short: could you test with lower case header in correct type? [{"content-type", "text/csv"}, {"accept", "application/json"}]

[JohnKacz]

That was it. Thanks @chulkilee and sorry for the trouble.