badarg On POST

[pulltab]

Hello! I am exploring using kafboy for a project of mine, however, I have encountered the following error with POSTing JSON to kafboy.

Client Executes:

B = iolist_to_binary(mochijson2:encode({struct,
 [{timestamp, <<"2014-11-4 22:12:21">>},
 {owner, <<"OwnerData">>},
 {peer, <<"PeerData">>},
 {src_lang, <<"en">>},
 {body, <<"Chat test!">>}]})).

Req = {"http://localhost:8000/async/chat", [], "application/json", B}.

httpc:request(post, Req, [], []).

The POST is processed by kafboy, however, kafboy reports the following error:

Error in process <0.1592.0> on node 'kafboy@127.0.0.1' with exit value: {badarg,[{erlang,byte_size,[true],[]},{ekaf_protocol_produce,encode_message,1,[{file,"src/ekaf_protocol_produce.erl"},{line,138}]},{ekaf_protocol_produce,encode_messages,2,[{file,"src/ekaf_protocol_produce.erl"},{line...

Is there something incorrect with how I am submitting this POST request?

[pulltab]

Doing some extra digging. Adding a debug message to kafboy_producer:async revealed the following:

ASYNC: topic=<<"chat">>, data=[{<<"{\"timestamp\":\"2014-11-4 22:12:21\",\"owner\":\"OwnerData\",\"peer\":\"PeerData\",\"src_lang\":\"en\",\"body\":\"Chat test!\"}">>,
                                true}]

At this point int the code, I expected that data would be a binary containing only the body of my HTTP request, however, it instead it is a list containing the tuple {HTTPBody, true}.

[pulltab]

The source of the tuple is cowboy_req:body_qs/1.

From what I can tell, kafboy_http_handler is not properly handling (not designed for?) JSON encoded POST operations where the body of the HTTP request is formatted JSON.

Perhaps kafboy_http_handle desires that the ContentType is application/x-www-form-urlencoded?

[bosky101]

Yes, it was designed for form POSTs, but i can see how application/json will be useful.

i was able to reproduce this, the consumer got {<<"some_json_encoded">>>, true } . i may have to check for content-type to decide how to send data from kafboy to ekaf.

thanks for reporting this. will update here in 2 weeks, but until then i'd advice considering http form post's with one field - say data=json_encoded_value

~B

[pulltab]

Thanks for the reply.

The reason I thought kafboy used JSON is due to the README. It reads:

With 0.8, Kafka clients take greater responsibility of deciding which broker and partition to publish to for a given topic.

kafboy is a http wrapper over the ekafka client, that takes care of routing http requests to the right kafka broker socket. kafboy is self-aware over a cluster, and supoprts nodes routing requests arriving on any node, to the right process in the cluster.

Simply send a POST with the desired JSON, to one of the following paths

I am already preparing a version which makes use of JSON instead. If this is something you would consider using, I wouldn't mind submitting it for your review as a merge request.