search

esl / MongooseIM

MongooseIM is Erlang Solutions' robust, scalable and efficient XMPP server, aimed at large installations. Specifically designed for enterprise purposes, it is fault-tolerant and can utilise the resources of multiple clustered machines.
Other
1.66k stars 426 forks source link

MongooseIM's client REST API #1910

Closed Beisenbek closed 6 years ago

Beisenbek commented 6 years ago

MongooseIM version: 3.0.0 Installed from: source Erlang/OTP version: 18

I found that REST API for One-to-one messages returns partial info about messages. There are no extension part of messages. Only body. Request:

curl -X GET --header 'Accept: application/json' 'http://localhost:8089/api/messages?limit=10'

And there are no archive_chat_markers in result set. But it enabled in config:

{mod_mam_meta, [
    {backend, riak},
    {archive_chat_markers,true},
...
Beisenbek commented 6 years ago

Sample result set

[
    {
        "to": "UserA@{host}",
        "timestamp": 1528283702183,
        "id": "DeGu9-20159",
        "from": "UserB@{host}",
        "body": "test message"
    }
]
michalwski commented 6 years ago

Hi @Beisenbek,

Thanks for trying the HTTP API! We implemented basic functionality over HTTP and are open to extend it. How would you see the extended result? Do you expect the full xml stanzas in the response or more keys in the JSON object?

Beisenbek commented 6 years ago

Hi @michalwski It seems to me that it'll better to get full xml stanzas in the response and more keys in the JSON object. In my case I want to put messages through active tcp connection and get MAM messages through REST API. So, there is problem in getting identical result for json based REST API and standard xml based MAM stanza.

michalwski commented 6 years ago

I think we could start with the full xml added to the JSON object under xml key like below:

[
    {
        "to": "UserA@{host}",
        "timestamp": 1528283702183,
        "id": "DeGu9-20159",
        "from": "UserB@{host}",
        "body": "test message",
        "xml": "<message id='some-id' to='UserA@{host}`><body>test message</body><other/></message>"
    }
]

The place in the code which generates the result is here: https://github.com/esl/MongooseIM/blob/33db205313b412e89fad3a3217b7a67808acfbb6/src/mongoose_client_api_messages.erl#L108-L114 would you be able to send us a PR? We are about to start new sprint so there will probably be no time to make the change in next 2 weeks by us.

Beisenbek commented 6 years ago

ok, I'll try send PR as quick as possible.

Beisenbek commented 6 years ago

Hi @michalwski

I've faced with problem. I got error invalid_ejson when tried to convert xml object to json.

My steps:

  1. got binary X1 from Msg object with exml:to_binary
  2. recursively parsed erlang tuple X2 from X1. So X2 equals to 
    {<<"message">>,
[
[
    {<<"@xml:lang">>,<<"en">>},
    {<<"@to">>,<<"UserA">>},
    {<<"@id">>,<<"1V4jk-18041">>},
    {<<"@type">>,<<"chat">>},
    {<<"body">>,
        [
            [
                {<<"@xml:lang">>,<<"en_US">>},
                {<<"#text">>,<<208,159,209,128,208,184,208,178,208,181,209,130,33,32,92,110,32,208,154,208,176,208,186,32,208,180,208,181,208,187,208,176,63>>}
            ]
        ]
    },
    {<<"properties">>,
        [
            [
                {<<"@xmlns">>,<<"http://www.jivesoftware.com/xmlns/xmpp/properties">>},
                {<<"property">>,[{<<"name">>,[{<<"#text">>,<<"chatVersion">>}]},{<<"value">>,[[{<<"@type">>,<<"string">>},{<<"#text">>,<<"4">>}]]}]},
                {<<"property">>,[{<<"name">>,[{<<"#text">>,<<"prevMessageId">>}]},{<<"value">>,[[{<<"@type">>,<<"string">>},{<<"#text">>,<<"mB6D0-33">>}]]}]}
            ]
        ]
    }
]
]}
  3. Now I can't convert X2 to json with jiffy:encode. Any help?

Parser code taken from here: https://stackoverflow.com/a/43151721:

convert(XML) when erlang:is_binary(XML) ->
    {ok, XMLEl} = exml:parse(XML),
    V = convert2(XMLEl),
    jiffy:encode({[V]}).

convert2(#xmlel{name = Name
              ,attrs = []
              ,children = Children}) ->
    {Name,  convert_children(Children,[])};
convert2(#xmlel{name = Name
              ,attrs = []
              ,children = [{xmlcdata, Data}]}) ->
    {Name, Data};
convert2(#xmlel{name = Name
              ,attrs = Attrs
              ,children = Children}) ->
    {Name,  [convert_attrs(Attrs,[]) ++ convert_children(Children,[])]}.

convert_attrs([V|T], Acc) ->
    convert_attrs(T, Acc ++ [convert_attr(V)]);
convert_attrs([], Acc) -> Acc.

convert_attr({Attr, Value}) ->
    {<<$@, Attr/binary>>, Value}.

convert_children([V|T], Acc) ->
    convert_children(T, Acc ++ [convert_child(V)]);
convert_children([], Acc) -> Acc.

convert_child({xmlcdata, Data}) ->
    {<<"#text">>, Data};
convert_child(#xmlel{}=XMLEl) ->
    convert2(XMLEl).
michalwski commented 6 years ago

Right, mapping XMPP's message stanza to a valid JSON object may not be possible without dropping some parts. That's why I suggested to add the XML String to the response. I know mixing XML with JSON is not very elegant way. On the other hand this gives you what you need in the easiest way.

Regarding you approach, and the parser you took from stack overflow I think you don't have to call exml:to_binary because the first thing the parser does is exml:parse (and you already have that in the Msg var). What's the error returned by jiffy:encode? maybe it gives some hints where is the problem.

Beisenbek commented 6 years ago

I agree that adding XML String to reponse is fastest way. I've implemented it already, but I'm feeling not comfortable with it.

Thanks for advice with exml:to_binary. Is it possible by some way to get from Msg var an erlang map?

Error from jiffy:encode:


2018-06-19 11:37:55.594 [error] emulator Error in process <0.1544.0> on node mongooseim@localhost with exit value:
{{nocatch,[{reason,{error,{invalid_ejson,{<<"@xml:lang">>,<<"en">>}}}},{mfa,{mongoose_client_api_messages,to_json,2}},{stacktrace,[{jiffy,encode,2,[{file,"/opt/xmpp/MongooseIM/_build/default/lib/jiffy/src/jiffy.erl"},{line,97}]},{mongoose_client_api_messages,encode,2,[{file,"/opt/xmpp/MongooseIM/_build/prod/lib/mongooseim/src/mongoose_client_api_messages.erl"},{line,122}]},{mongoose_client_api_messages,'-maybe_to_json_with_jid/4-lc$^0/1-0-',1,[{file,"/opt/xmpp/MongooseIM/_build/prod/lib/mongooseim/src/mongoose_client_api_messages.erl"},{line,76}]},{mongoose_client_api_messages,maybe_to_json_with_jid,4,[{file,"/opt/xmpp/MongooseIM/_build/prod/lib/mongooseim/src/mongoose_client_api_messages.erl"},{line,76}]},{cowboy_rest,call,3,[{file,"/opt/xmpp/MongooseIM/_build/default/lib/cowboy/src/cowboy_rest.erl"},{line,976}]},{cowboy_rest,set_resp_body,2,[{file,"/opt/xmpp/MongooseIM/_build/default/lib/cowboy/src/cowboy_rest.erl"},{line,858}]},{cowboy_protocol,execute,4,[{file,"/opt/xmpp/MongooseIM/_build/default/lib/cowboy/src/cowboy_protocol.erl"},{line,442}]}]},{req,[{socket,#Port<0.18489>},{transport,ranch_tcp},{connection,keepalive},{pid,<0.1544.0>},{method,<<"GET">>},{version,'HTTP/1.1'},{peer,{{46,19,47,34},32832}},{host,<<"{host}">>},{host_info,undefined},{port,8089},{path,<<"/api/messages/UserA@{host}">>},{path_info,undefined},{qs,<<"limit=1">>},{qs_vals,undefined},{bindings,[{with,<<"UserA@{host}">>}]},{headers,[{<<"cache-control">>,<<"no-cache">>},{<<"authorization">>,<<"Basic ABCDE=">>},{<<"user-agent">>,<<"PostmanRuntime/7.1.1">>},{<<"accept">>,<<"*/*">>},{<<"host">>,<<"{host}:8089">>},{<<"accept-encoding">>,<<"gzip, deflate">>},{<<"connection">>,<<"keep-alive">>}]},{p_headers,[{<<"if-modified-since">>,undefined},{<<"if-none-match">>,undefined},{<<"if-unmodified-since">>,undefined},{<<"if-match">>,undefined},{<<"accept">>,[{{<<"*">>,<<"*">>,[]},1000,[]}]},{<<"authorization">>,{<<"basic">>,{<<"UserB@{host}">>,<<"a0818e41-aa42-4b64-b5b9-46784590c01d">>}}},{<<"connection">>,[<<"keep-alive">>]}]},{cookies,undefined},{meta,[{media_type,{<<"application">>,<<"json">>,[]}},{charset,undefined}]},{body_state,waiting},{buffer,<<>>},{multipart,undefined},{resp_compress,true},{resp_state,waiting},{resp_headers,[{<<"content-type">>,[<<"application">>,<<"/">>,<<"json">>,<<>>]}]},{resp_body,<<>>},{onresponse,undefined}]},{state,#{jid => {jid,<<"UserB">>,<<"{host}">>,<<>>,<<"UserB">>,<<"{host}">>,<<>>},user => <<"UserB@{host}">>}}]},[{cowboy_rest,set_resp_body,2,[{file,"/opt/xmpp/MongooseIM/_build/default/lib/cowboy/src/cowboy_rest.erl"},{line,858}]},{cowboy_protocol,execute,4,[{file,"/opt/xmpp/MongooseIM/_build/default/lib/cowboy/src/cowboy_protocol.erl"},{line,442}]}]}
2018-06-19 11:37:55.594 [error] <0.1544.0> Ranch listener 'ejabberd_cowboy_0.0.0.0_8089' terminated with reason: {nocatch,[{reason,{error,{invalid_ejson,{<<"@xml:lang">>,<<"en">>}}}},{mfa,{mongoose_client_api_messages,to_json,2}},{stacktrace,[{jiffy,encode,2,[{file,"/opt/xmpp/MongooseIM/_build/default/lib/jiffy/src/jiffy.erl"},{line,97}]},{mongoose_client_api_messages,encode,2,[{file,"/opt/xmpp/MongooseIM/_build/prod/lib/mongooseim/src/mongoose_client_api_messages.erl"},{line,122}]},{mongoose_client_api_messages,'-maybe_to_json_with_jid/4-lc$^0/1-0-',1,[{file,"/opt/xmpp/MongooseIM/_build/prod/lib/mong..."},...]},...]},...]} in cowboy_rest:set_resp_body/2 line 858```
michalwski commented 6 years ago

Is it possible by some way to get from Msg var an erlang map?

No, currently there is no such an API. I know that be useful in this case.

Regarding the error from jiffy:encode, I think that it's not able to encode the key @xml:lang. I'm not sure but : is probably not a valid character in JSON key.

Beisenbek commented 6 years ago

I checked for key @xml:lang - removing spec char : doesn't affect to error. The propblem was in parser. After little refactoring problem with ejson was solved:

convert(XML) ->
    {ok, XMLEl} = exml:parse(XML),
    V = {[convert2(XMLEl)]},
    jiffy:encode(V).

convert2(#xmlel{name = Name
              ,attrs = []
              ,children = [{xmlcdata, Data}]}) ->
    {Name, Data};
convert2(#xmlel{name = Name
              ,attrs = Attrs
              ,children = Children}) ->
    {Name,  {convert_attrs(Attrs) ++ convert_children(Children)}}.

convert_attrs(Attrs) ->
    convert_attrs(Attrs,[]).

convert_attrs([Attr|Attrs1], Attrs2) ->
    convert_attrs(Attrs1, [convert_attr(Attr)|Attrs2]);
convert_attrs([], Attrs2) ->
    lists:reverse(Attrs2).

convert_attr({Attr, Value}) ->
    {<<$@, Attr/binary>>, Value}.

convert_children(Children) ->
    convert_children(Children, []).

convert_children([Child|Children1], Children2) ->
    convert_children(Children1, [convert_child(Child)|Children2]);
convert_children([], Children2) ->
    lists:reverse(Children2).

convert_child({xmlcdata, Data}) ->
    {<<"#text">>, Data};
convert_child(#xmlel{}=XMLEl) ->
    convert2(XMLEl).

For getting erlang map, i tried to use jiffy:decode with return_maps option

31> K = {[{<<"message">>,{[{<<"@xml:lang">>,<<"en">>},{<<"@to">>,<<"UserA">>},{<<"@id">>,<<"1V4jk-18041">>},{<<"@type">>,<<"chat">>},{<<"body">>,{[{<<"@xml:lang">>,<<"en_US">>},{<<"#text">>,<<208,159,209,128,208,184,208,178,208,181,209,130,33,32,92,110,32,208,154,208,176,208,186,32,208,180,208,181,208,187,208,176,63>>}]}},{<<"properties">>,{[{<<"@xmlns">>,<<"http://www.jivesoftware.com/xmlns/xmpp/properties">>},{<<"property">>,{[{<<"name">>,<<"chatVersion">>},{<<"value">>,{[{<<"@type">>,<<"string">>},{<<"#text">>,<<"4">>}]}}]}},
31> {<<"property">>,{[{<<"name">>,<<"prevMessageId">>},{<<"value">>,{[{<<"@type">>,<<"string">>},{<<"#text">>,<<"mB6D0-33">>}]}}]}}]}}]}}]}.
{[{<<"message">>,
   {[{<<"@xml:lang">>,<<"en">>},
     {<<"@to">>,<<"UserA">>},
     {<<"@id">>,<<"1V4jk-18041">>},
     {<<"@type">>,<<"chat">>},
     {<<"body">>,
      {[{<<"@xml:lang">>,<<"en_US">>},
        {<<"#text">>,
         <<208,159,209,128,208,184,208,178,208,181,209,130,...>>}]}},
     {<<"properties">>,
      {[{<<"@xmlns">>,
         <<"http://www.jivesoftware.com/xmlns/xmpp/propertie"...>>},
        {<<"property">>,
         {[{<<"name">>,<<"chatVersion">>},
           {<<"value">>,
            {[{<<"@type">>,<<"string">>},{<<"#text">>,<<"4">>}]}}]}},
        {<<"property">>,
         {[{<<"name">>,<<"prevMessageId">>},
           {<<"value">>,
            {[{<<"@type">>,<<"stri"...>>},
              {<<"#tex"...>>,<<...>>}]}}]}}]}}]}}]}
34> L = jiffy:encode(K).
<<"{\"message\":{\"@xml:lang\":\"en\",\"@to\":\"UserA\",\"@id\":\"1V4jk-18041\",\"@type\":\"chat\",\"body\":{\"@xml:lang\":\"en_US\",\"#text\":\""...>>
44> M = jiffy:decode(L,[return_maps]).
#{<<"message">> =>
      #{<<"@id">> => <<"1V4jk-18041">>,<<"@to">> => <<"UserA">>,
        <<"@type">> => <<"chat">>,<<"@xml:lang">> => <<"en">>,
        <<"body">> =>
            #{<<"#text">> =>
                  <<208,159,209,128,208,184,208,178,208,181,209,130,33,32,
                    92,110,32,208,154,208,176,208,...>>,
              <<"@xml:lang">> => <<"en_US">>},
        <<"properties">> =>
            #{<<"@xmlns">> =>
                  <<"http://www.jivesoftware.com/xmlns/xmpp/properties">>,
              <<"property">> =>
                  #{<<"name">> => <<"prevMessageId">>,
                    <<"value">> =>
                        #{<<"#text">> => <<"mB6D0-33">>,
                          <<"@type">> => <<"string">>}}}}}

I'm wondering why jiffy:decode dropped second property called chatVersion in the map M.

If call jiffy:decode without any option - it's ok. But result is not erlang map.

45> N = jiffy:decode(L).
{[{<<"message">>,
   {[{<<"@xml:lang">>,<<"en">>},
     {<<"@to">>,<<"UserA">>},
     {<<"@id">>,<<"1V4jk-18041">>},
     {<<"@type">>,<<"chat">>},
     {<<"body">>,
      {[{<<"@xml:lang">>,<<"en_US">>},
        {<<"#text">>,
         <<208,159,209,128,208,184,208,178,208,181,209,130,...>>}]}},
     {<<"properties">>,
      {[{<<"@xmlns">>,
         <<"http://www.jivesoftware.com/xmlns/xmpp/propertie"...>>},
        {<<"property">>,
         {[{<<"name">>,<<"chatVersion">>},
           {<<"value">>,
            {[{<<"@type">>,<<"string">>},{<<"#text">>,<<"4">>}]}}]}},
        {<<"property">>,
         {[{<<"name">>,<<"prevMessageId">>},
           {<<"value">>,
            {[{<<"@type">>,<<"stri"...>>},
              {<<"#tex"...>>,<<...>>}]}}]}}]}}]}}]}
kzemek commented 6 years ago

@Beisenbek in JSON, keys are unique - you cannot have an object that has two "property" keys. This part has to be modeled differently, e.g. "properties" should hold an array.

Beisenbek commented 6 years ago

thanks @kzemek! do you have any idea how to handle it? you mean modeling inside exml:parse or in other place?

kzemek commented 6 years ago

@Beisenbek to losslessly translate XML element into a valid JSON object you could directly translate #xmlel{} structures, which would result in a JSON object like this:

{"name": "...", "params": [{"name": "...", "value": "..."}], "children": [...]}

So basically:

to_json_obj(#xmlel{name = Name, params = Params, children = Children}) ->
    [{"name", Name},
     {"params", Params},
     {"children", [to_json_obj(Child) || Child <- Children]]}].
Beisenbek commented 6 years ago

Hi @michalwski

Can you review my PR 1938. Whats wrong with it?

Thanks.

fenek commented 6 years ago

@Beisenbek Since #1976 is merged, can we close this issue?

Beisenbek commented 6 years ago

@fenek yep, thanks!

Denismih commented 5 years ago

MongooseIM version: 3.1.0 Installed from: source Erlang/OTP version: latest with source

Hello, I have problem with out of band messages and REST API. When I'm trying send oob message via REST API : { "body": "some text", "x": { "xmlns": "jabber:x:oob", "url": "http://XXX.XXX.XXX.XXX/jabber/782ac6570eb7e0f5748ab66d8824d831bcfc8159ffbd0fea4ec13dacb0bc4b56/5F93DCE3-258D-4119-B3A8-9A886D422A85.jpeg" } }

in client xmpp side I receive only /body without oob teg <x xmlns="jabber:x:oob"><url>http://XXX.XXX.XXX.XXX/jabber/782ac6570eb7e0f5748ab66d8824d831bcfc8159ffbd0fea4ec13dacb0bc4b56/5F93DCE3-258D-4119-B3A8-9A886D422A85.jpeg</url></x>

fenek commented 5 years ago

Hi @Denismih

Can you please create a new thread for your issue? :)