json output? - Githubissues

kenniki commented 9 years ago

Would it be possible to format the output of tg-cli as JSON? This would massively help with development of bots.

jonnywilliamson commented 9 years ago

Yes I would think this would be an excellent addition.

vysheng commented 9 years ago

Added (maybe buggy) support via jansson to test branch

jonnywilliamson commented 9 years ago

Wow. That's a big commit - thank you @vysheng! Will test this out.

vysheng commented 9 years ago

FYI I don't expect I will use it, so it's up to you to find bugs

jonnywilliamson commented 9 years ago

@vysheng Understood. It's looking great so far, but I might have a very small bug. About to type it up.

jonnywilliamson commented 9 years ago

OK first bug report.

using the json flag, and asking for the dialog list returns VALID json. The json, is wrapped in [ ]

[
    {
        "first_name": "<removed>",
        "print_name": "<removed>",
        "id": "<removed>",
        "type": "user",
        "phone": "<removed>",
        "flags": 257,
        "last_name": "<removed>",
        "username": ""
    },
    {
        "first_name": "<removed>",
        "print_name": "<removed>",
        "id": "<removed>",
        "type": "user",
        "flags": 256,
        "last_name": "<removed>",
        "username": "<removed>"
    }
]

All good. However when you send a message you get the reponse back in json, but it is missing a couple of brackets and a comma separating each item.

INCORRECT

{
    "service": false,
    "id": 259,
    "text": "Hello",
    "date": 1431346134,
    "out": true,
    "to": {
        "first_name": "<removed>",
        "print_name": "<removed>",
        "id": "<removed>",
        "type": "user",
        "phone": "<removed>",
        "flags": 257,
        "last_name": "<removed>",
        "username": "<removed>"
    },
    "from": {
        "first_name": "<removed>",
        "print_name": "<removed>",
        "id": "<removed>",
        "type": "user",
        "phone": "<removed>",
        "flags": 264,
        "last_name": "<removed>",
        "username": "<removed>"
    },
    "unread": true
}{
    "result": "SUCCESS"
}

It should be this:

[
    {
        "service": false,
        "id": 259,
        "text": "Hello",
        "date": 1431346134,
        "out": true,
        "to": {
            "first_name": "<removed>",
            "print_name": "<removed>",
            "id": "<removed>",
            "type": "user",
            "phone": "<removed>",
            "flags": 257,
            "last_name": "<removed>",
            "username": "<removed>"
        },
        "from": {
            "first_name": "<removed>",
            "print_name": "<removed>",
            "id": "<removed>",
            "type": "user",
            "phone": "<removed>",
            "flags": 264,
            "last_name": "<removed>",
            "username": "<removed>"
        },
        "unread": true
    },
    {
        "result": "SUCCESS"
    }
]

Unless the response is supposed to come in 2 separate json replies....

vysheng commented 9 years ago

it is so, because it is two independent messages. First is SUCCESS for send message query, second is update containing message. It is rather hard to unite this messages to one

kenniki commented 9 years ago

You're a god. Thanks so much!

jonnywilliamson commented 9 years ago

Just back from work. Ok that's no problem, I don't need the messages joined.

For anyone else trying to configure/make this new test build, your server might need the libjansson library installed. Mine definitely did.

Ubuntu/Debian etc

sudo apt-get install libjansson-dev

luckydonald commented 9 years ago

Rendering my luckydonald/telejson fork useless, hooray!

luckydonald commented 9 years ago

Mac OS has no brew for it yet, it seems.

Building libjansson from source

Grab you the latest release from http://www.digip.org/jansson/releases/ .
Current stable currently is 2.7

wget http://www.digip.org/jansson/releases/jansson-2.7.tar.bz2
bunzip2 -c jansson-*.tar.bz2 | tar xf -
cd jansson-*/
./configure
make
make check
sudo make install

luckydonald commented 9 years ago

More Important would be that the answers on sockets are json formatted.
Edit: Never mind, forgot the --json flag. This shit is cool...

luckydonald commented 9 years ago

regarding the SUCCESS: If I understood correctly, the SUCCESS says: "your command did execute correctly"? If that is the case, the SUCCESS should be the first response from the server, so you know you command did succeed, and the message should be the second thing. This would allow us to map the issued command to the response.

kenniki commented 9 years ago

Unfortunately, I can't compile it. Installed is libjansson4:amd64 (2.3.1-2) (from Debian Wheezy repositories) as well as libjansson-dev:amd64 (2.3.1-2).

xray238:/opt/tg-dev# ./configure
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for sqrt in -lm... yes
checking for library containing clock_gettime... -lrt
checking for library containing backtrace... none required
checking for event_base_new in -levent... yes
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking event2/event.h usability... yes
checking event2/event.h presence... yes
checking for event2/event.h... yes
checking for pkg-config... /usr/bin/pkg-config
checking whether compiling and linking against OpenSSL works... yes
checking if zlib is wanted... yes
checking for inflateEnd in -lz... yes
checking zlib.h usability... yes
checking zlib.h presence... yes
checking for zlib.h... yes
checking for rl_save_prompt in -lreadline... yes
checking for libconfig... enabled
checking for config_init in -lconfig... yes
checking for liblua... enabled
checking for lua... /usr/bin/lua
checking if /usr/bin/lua is a Lua interpreter... yes
checking for lua version... 5.2
checking for lua platform... unknown
checking for lua script directory... ${prefix}/share/lua/5.2
checking for lua module directory... ${exec_prefix}/lib/lua/5.2
checking if LUA_VERSION is defined... yes
checking lua.h usability... no
checking lua.h presence... no
checking for lua.h... no
checking lualib.h usability... no
checking lualib.h presence... no
checking for lualib.h... no
checking lauxlib.h usability... no
checking lauxlib.h presence... no
checking for lauxlib.h... no
checking luaconf.h usability... no
checking luaconf.h presence... no
checking for luaconf.h... no
checking for Lua headers in... /usr/include/lua5.2
checking lua.h usability... yes
checking lua.h presence... yes
checking for lua.h... yes
checking lualib.h usability... yes
checking lualib.h presence... yes
checking for lualib.h... yes
checking lauxlib.h usability... yes
checking lauxlib.h presence... yes
checking for lauxlib.h... yes
checking luaconf.h usability... yes
checking luaconf.h presence... yes
checking for luaconf.h... yes
checking for Lua header version... 5.2
checking if Lua header version matches 5.2... yes
checking if LUA_VERSION is defined... yes
checking for library containing exp... none required
checking for library containing dlopen... -ldl
checking for library containing lua_load... -llua5.2
checking for libjansson... enabled
checking for json_array_set_new in -ljansson... yes
checking progname... default
checking execinfo.h usability... yes
checking execinfo.h presence... yes
checking for execinfo.h... yes
checking fcntl.h usability... yes
checking fcntl.h presence... yes
checking for fcntl.h... yes
checking malloc.h usability... yes
checking malloc.h presence... yes
checking for malloc.h... yes
checking netdb.h usability... yes
checking netdb.h presence... yes
checking for netdb.h... yes
checking for stdlib.h... (cached) yes
checking for string.h... (cached) yes
checking for unistd.h... (cached) yes
checking arpa/inet.h usability... yes
checking arpa/inet.h presence... yes
checking for arpa/inet.h... yes
checking mach/mach.h usability... no
checking mach/mach.h presence... no
checking for mach/mach.h... no
checking netinet/in.h usability... yes
checking netinet/in.h presence... yes
checking for netinet/in.h... yes
checking sys/file.h usability... yes
checking sys/file.h presence... yes
checking for sys/file.h... yes
checking sys/socket.h usability... yes
checking sys/socket.h presence... yes
checking for sys/socket.h... yes
checking termios.h usability... yes
checking termios.h presence... yes
checking for termios.h... yes
checking for size_t... yes
checking for uid_t in sys/types.h... yes
checking for inline... inline
checking for stdlib.h... (cached) yes
checking for GNU libc compatible malloc... yes
checking for stdlib.h... (cached) yes
checking for GNU libc compatible realloc... yes
checking for alarm... yes
checking for endpwent... yes
checking for memset... yes
checking for memmove... yes
checking for mkdir... yes
checking for select... yes
checking for socket... yes
checking for strdup... yes
checking for strndup... yes
checking for uname... yes
checking for __builtin_bswap32... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating config.h
xray238:/opt/tg-dev# make
cat tgl/scheme.tl tgl/encrypted_scheme.tl tgl/binlog.tl tgl/append.tl tgl/mtproto.tl > auto/scheme.tl
gcc -I. -I. -I./tgl -g -O2  -I/usr/local/include -I/usr/include -I/usr/include -I/usr/include/lua5.2   -DHAVE_CONFIG_H -Wall -Wextra -Werror -Wno-deprecated-declarations -fno-strict-aliasing -fno-omit-frame-pointer -ggdb -Wno-unused-parameter -fPIC -iquote ./tgl/tl-parser -c -MP -MD -MF dep/tl-parser.d -MQ objs/tl-parser.o -o objs/tl-parser.o tgl/tl-parser/tl-parser.c
gcc -I. -I. -I./tgl -g -O2  -I/usr/local/include -I/usr/include -I/usr/include -I/usr/include/lua5.2   -DHAVE_CONFIG_H -Wall -Wextra -Werror -Wno-deprecated-declarations -fno-strict-aliasing -fno-omit-frame-pointer -ggdb -Wno-unused-parameter -fPIC -iquote ./tgl/tl-parser -c -MP -MD -MF dep/tlc.d -MQ objs/tlc.o -o objs/tlc.o tgl/tl-parser/tlc.c
gcc -I. -I. -I./tgl -g -O2  -I/usr/local/include -I/usr/include -I/usr/include -I/usr/include/lua5.2   -DHAVE_CONFIG_H -Wall -Wextra -Werror -Wno-deprecated-declarations -fno-strict-aliasing -fno-omit-frame-pointer -ggdb -Wno-unused-parameter -fPIC -iquote ./tgl/tl-parser -c -MP -MD -MF dep/crc32.d -MQ objs/crc32.o -o objs/crc32.o tgl/tl-parser/crc32.c
gcc objs/tl-parser.o objs/tlc.o objs/crc32.o -L/usr/local/lib -L/usr/lib -L/usr/lib   -rdynamic -ggdb -levent -ljansson -lconfig -lz -levent -lrt -lm   -lreadline -llua5.2  -ldl -lssl -lcrypto   -o bin/tl-parser
bin/tl-parser -e auto/scheme.tlo auto/scheme.tl
gcc -I. -I. -I./tgl -g -O2  -I/usr/local/include -I/usr/include -I/usr/include -I/usr/include/lua5.2   -DHAVE_CONFIG_H -Wall -Wextra -Werror -Wno-deprecated-declarations -fno-strict-aliasing -fno-omit-frame-pointer -ggdb -Wno-unused-parameter -fPIC -iquote ./tgl -c -MP -MD -MF dep/generate.d -MQ objs/generate.o -o objs/generate.o tgl/generate.c
gcc -I. -I. -I./tgl -g -O2  -I/usr/local/include -I/usr/include -I/usr/include -I/usr/include/lua5.2   -DHAVE_CONFIG_H -Wall -Wextra -Werror -Wno-deprecated-declarations -fno-strict-aliasing -fno-omit-frame-pointer -ggdb -Wno-unused-parameter -fPIC -iquote ./tgl -c -MP -MD -MF dep/tools.d -MQ objs/tools.o -o objs/tools.o tgl/tools.c
gcc objs/generate.o objs/tools.o -L/usr/local/lib -L/usr/lib -L/usr/lib   -rdynamic -ggdb -levent -ljansson -lconfig -lz -levent -lrt -lm   -lreadline -llua5.2  -ldl -lssl -lcrypto   -o bin/generate
bin/generate -g skip-header auto/scheme.tlo > auto/auto-skip.h || rm auto/auto-skip.h
bin/generate -g fetch-header auto/scheme.tlo > auto/auto-fetch.h || rm auto/auto-fetch.h
bin/generate -g store-header auto/scheme.tlo > auto/auto-store.h || rm auto/auto-store.h
bin/generate -g autocomplete-header auto/scheme.tlo > auto/auto-autocomplete.h || rm auto/auto-autocomplete.h
bin/generate -g types-header auto/scheme.tlo > auto/auto-types.h || rm auto/auto-types.h
gcc -I. -I. -I./tgl -g -O2  -I/usr/local/include -I/usr/include -I/usr/include -I/usr/include/lua5.2   -DHAVE_CONFIG_H -Wall -Wextra -Werror -Wno-deprecated-declarations -fno-strict-aliasing -fno-omit-frame-pointer -ggdb -Wno-unused-parameter -fPIC -c -MP -MD -MF dep/main.d -MQ objs/main.o -o objs/main.o main.c
gcc -I. -I. -I./tgl -g -O2  -I/usr/local/include -I/usr/include -I/usr/include -I/usr/include/lua5.2   -DHAVE_CONFIG_H -Wall -Wextra -Werror -Wno-deprecated-declarations -fno-strict-aliasing -fno-omit-frame-pointer -ggdb -Wno-unused-parameter -fPIC -c -MP -MD -MF dep/loop.d -MQ objs/loop.o -o objs/loop.o loop.c
gcc -I. -I. -I./tgl -g -O2  -I/usr/local/include -I/usr/include -I/usr/include -I/usr/include/lua5.2   -DHAVE_CONFIG_H -Wall -Wextra -Werror -Wno-deprecated-declarations -fno-strict-aliasing -fno-omit-frame-pointer -ggdb -Wno-unused-parameter -fPIC -c -MP -MD -MF dep/interface.d -MQ objs/interface.o -o objs/interface.o interface.c
gcc -I. -I. -I./tgl -g -O2  -I/usr/local/include -I/usr/include -I/usr/include -I/usr/include/lua5.2   -DHAVE_CONFIG_H -Wall -Wextra -Werror -Wno-deprecated-declarations -fno-strict-aliasing -fno-omit-frame-pointer -ggdb -Wno-unused-parameter -fPIC -c -MP -MD -MF dep/lua-tg.d -MQ objs/lua-tg.o -o objs/lua-tg.o lua-tg.c
gcc -I. -I. -I./tgl -g -O2  -I/usr/local/include -I/usr/include -I/usr/include -I/usr/include/lua5.2   -DHAVE_CONFIG_H -Wall -Wextra -Werror -Wno-deprecated-declarations -fno-strict-aliasing -fno-omit-frame-pointer -ggdb -Wno-unused-parameter -fPIC -c -MP -MD -MF dep/json-tg.d -MQ objs/json-tg.o -o objs/json-tg.o json-tg.c
json-tg.c: In function 'json_pack_message':
json-tg.c:352:3: error: implicit declaration of function 'json_boolean' [-Werror=implicit-function-declaration]
json-tg.c:352:3: error: passing argument 3 of 'json_object_set' makes pointer from integer without a cast [-Werror]
In file included from json-tg.c:4:0:
/usr/include/jansson.h:149:5: note: expected 'struct json_t *' but argument is of type 'int'
json-tg.c:353:3: error: passing argument 3 of 'json_object_set' makes pointer from integer without a cast [-Werror]
In file included from json-tg.c:4:0:
/usr/include/jansson.h:149:5: note: expected 'struct json_t *' but argument is of type 'int'
json-tg.c:354:3: error: passing argument 3 of 'json_object_set' makes pointer from integer without a cast [-Werror]
In file included from json-tg.c:4:0:
/usr/include/jansson.h:149:5: note: expected 'struct json_t *' but argument is of type 'int'
cc1: all warnings being treated as errors
make: *** [objs/json-tg.o] Error 1

jonnywilliamson commented 9 years ago

@kenniki

Have you done this. Goto your telegram directory. This should be a git repository.

Run these commands:

git pull
git checkout test
git submodule update --recursive

Then do

./configure
make

Does that help?

kenniki commented 9 years ago

Nevermind, just tried as @luckydonald pointed out; it works with the newest libjansson. Thanks again for the commit!

@jonnywilliamson This was a fresh git clone in a newly created directory on a host that has never seen tg-cli before, so I doubt that would help. Thanks anyway!

luckydonald commented 9 years ago

@vysheng found some bugs:

I'd be happy to create pull requests with fixes, just tell me.

Service events

The service messages are not implemented yet?

Edit: I noticed, many events are. For example: User did rename. Edit2: appended distinguishable events

The following got written as non-json output:

User luckydonald marked read 1 outbox and 0 inbox messages

One possible solution for sending such services would be to resend/reprint the message as json, with "unread":false and "service":"marked message read" (or something) instead of being a boolean. Or have a "event":"message" and "event":"service" attribute? Until services are implemented, could you wrap all normal output in json? like

{"output":"User luckydonald marked read 1 outbox and 0 inbox messages"}

This would still allow the json parsers to work correctly.

distinguishable output

Right now, you have to guess what type a json array is. Messages, for example, tend to have a "text" and/or "media" attribute. Updates had different fields, "peer" and "updates". It would be great to have a field explicitly stating the type:

{"event": "message",  ... }

Edit3: Is implemented a bit, not all might have that. I thing I got all that are sent automaticly, without typing a command before.

'out' Attribute

(I refer to the cli as 'bot' in the following) Also the "out" attribute is misleading, as "out" is true, when the sender is the bot account. If I log in the bot with another client, and send a message there, this message is marked as out too. I would recommend to rename it to own as this doen't lead to that assumption.

Summary

[x] ~~"out" attribute != outgoing, just own accout~~ - He said no. :smile:
[ ] Normal output as valid json, until implemented. - Just fix it for real.
[x] Implement services - many are already.
[x] message "id" is filled with the flags. - Fixed with 081c65ce8f1cab6ea6b697bed4bd1753f17614f9
[ ] add "event" attribute to distinguish different output. - started, not completed yet.

jonnywilliamson commented 9 years ago

@luckydonald great suggestions.

luckydonald commented 9 years ago

message ID fixed, added my thoughts about "distinguishable events" above.

That stuff above is what I found useful already using json in my own json implementation. So I'll just try to give back the good stuff, so everyone can profit.

luckydonald commented 9 years ago

@vysheng can I have your thoughts about the ideas above? Should I start writing pull requests?

vysheng commented 9 years ago

Out is outgoing for account. I think it is correct behavour. Other options make sense

luckydonald commented 9 years ago

Working on a pull request.

luckydonald commented 9 years ago

You can't compare repos which are not directly forked. This sucks.

sladec commented 9 years ago

Thanks for the json output ...so much easier to work with. I notice I don't get user online/offline events ? are they still coming or am I missing something ?

luckydonald commented 9 years ago

Pimp my json.

nleo commented 9 years ago

No info about unread in json

> dialog_list 
User Leo User: 1 unread

 > echo dialog_list | nc localhost 2392 -q 1
ANSWER 157
[{"phone": "79...", "type": "user", "id": ..., "print_name": "Leo_User", "flags": 257, "first_name": "Leo", "last_name": "User", "username": ""}]

Also output for contact_list and dialog_list absolutly same

luckydonald commented 9 years ago

~~regarding unread events: The pull request is not yet merged.~~ Edit: confused that with read events.

luckydonald commented 9 years ago

@nleo I am not quite sure what you are refering to...

nleo commented 9 years ago

I'm reffering to "json output" in general. I'm using 1.3.1

Should I create new issue for such bugs in future?

luckydonald commented 9 years ago

No, that's fine. I just did not understand your issue?

nleo commented 9 years ago

When I run telegram-cli with --json output of dialog_list and contact_list the same. Also dialog_list does not contain unread counter.

When I run telegram-cli without --json output of dialog_list and contact_list different. Also dialog_list contains unread counter.

At least I just want unread counter in dialog_list command output json, when I run telegram-cli with --json

luckydonald commented 9 years ago

I'm on vaccation for a week. I can look inzo that aftwrwards.

mdibaiee commented 9 years ago

This is awesome, thanks!

I sometimes get [SyntaxError: Unexpected Token], not sure what causes it, any clues why?

rizaumami commented 8 years ago

Please add stickers block to --json similar to bot API as example below:

"sticker":{"width":512,"height":456,"thumb":{"file_id":"AAQFABMC6LEyAASXq9uiahB_xSUjAAIC","file_size":4908,"width":128,"height":114},"file_id":"BQADBQADhwEAAlv3XQMOJyTPK3TToAI","file_size":30112}

As for now, telegram-cli --json doesn't differentiate between stickers or regular (non-mulltimedia) files.

"media": {"type": "document"}

luckydonald commented 8 years ago

:+1:

luckydonald commented 8 years ago

The content in tg/interface.c:3297-3393 needs to be transferred to tg/json-tg.c:200-203

(links as of commit 68e3e1a)

vysheng / tg

json output? #516

Building libjansson from source

Service events

distinguishable output

'out' Attribute

Summary