MessagePack output not working

[tomgie]

When trying to do a post request, example; /GET/test when appending .msg to the end; /GET/test.msg nothing happens and the request is returned as json format. Is this a bug or is there something wrong with how I am using it?

[nicolasff]

It probably depends on whether MessagePack support was built in or not. The way it currently works, the Makefile checks for the presence of /usr/lib/libmsgpack.so and builds support for it if the file exists. I can see now that this might be an issue depending on the platform…

This is pretty old code (added 2011-08-20) and I only had a 32-bit machine at the time; it's quite possible that the msgpack library would be under /usr/lib64 now.

Are you building Webdis from source? If so, you can change the Makefile and replace these lines:

MSGPACK_LIB=$(shell ls /usr/lib/libmsgpack.so 2>/dev/null)
ifneq ($(strip $(MSGPACK_LIB)),)
    FORMAT_OBJS += formats/msgpack.o
    CFLAGS += -DMSGPACK=1
    LDFLAGS += -lmsgpack
endif

with these:

FORMAT_OBJS += formats/msgpack.o
CFLAGS += -DMSGPACK=1
LDFLAGS += -lmsgpack

(so just keep what's inside the ifneq block.)

If you're using a packaged binary installed using your distribution's package manager, there's not much you can do except building it from source. If you do this, install libmsgpack-dev first (the name could be slightly different), then locate libmsgpack.so and replace the location in the Makefile with the correct path.

Any suggestion on how to handle this better in the Makefile would be very welcome.

[tomgie]

Ah, I never installed libmsgpack-dev. As I am using docker, I am also using the linux image alpine. I was looking around for something equivilent to libmsgpack-dev. But only found msgpack-c-dev for alpine, which did not work. What do you think is the lowest usage linux flavor that is supported by webdis at the moment?

[nicolasff]

How are you using Docker in this context? Are you building webdis yourself in Docker? By the way msgpack-c-dev sounds like the right package. If you are building webdis in Docker, could you share your Dockerfile, or a sanitized version of it with nothing proprietary? I'm happy to tweak it to make sure it gets built with MessagePack support.

[tomgie]

I cloned it off of one of the major docker hub repos and fit it to my needs. Its publicly available here: https://github.com/tomgie/docker-webdis/blob/master/Dockerfile

It builds webdis from source making sure to install the dependencies along with copying a new Makefile with the modified libmsgpackc.so instead of libmsgpack.so

Docker Hub build logs are here: https://gist.github.com/tomgie/5be90f05a5f55ef651c44d73929277c0

[nicolasff]

Oh yes so it did detect MessagePack, as evidenced by the presence of -DMSGPACK=1 in the build logs. But I can see that it failed to link with the .so:

cc -o webdis webdis.o cmd.o worker.o slog.o server.o acl.o md5/md5.o sha1/sha1.o http.o client.o websocket.o pool.o conf.o formats/json.o formats/raw.o formats/common.o formats/custom-type.o formats/msgpack.o hiredis/hiredis.o hiredis/sds.o hiredis/net.o hiredis/async.o hiredis/read.o hiredis/dict.o jansson/src/dump.o jansson/src/error.o jansson/src/hashtable.o jansson/src/load.o jansson/src/strbuffer.o jansson/src/utf.o jansson/src/value.o jansson/src/variadic.o http-parser/http_parser.o b64/cencode.o -levent -pthread -lmsgpack
/usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find -lmsgpack

I'll try to update the Dockerfile to link to the correct library as you described and will report back.

[nicolasff]

It looks like the version of msgpack provided by Alpine Linux is incompatible with what Webdis uses. For example, Webdis uses functions like msgpack_pack_raw and msgpack_pack_raw_body, but these aren't defined in the headers installed under /usr/include/msgpack. From what I can tell, apk installed msgpack-c-dev-2.0.0-r1. I suspect I built support for it against version 1, not 2.

I tried installing a version under 2.0 but it wasn't available:

/ # apk add 'msgpack-dev-c<2.0'
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
ERROR: unsatisfiable constraints:
  msgpack-dev-c (missing):
    required by: world[msgpack-dev-c<2.0]

Could you tweak the Dockerfile that Webdis comes with, and use that instead?

[nicolasff]

I fixed the Dockerfile, you should be able to use it to build a docker image with MessagePack support: https://github.com/nicolasff/webdis#try-in-docker

[nicolasff]

$ docker run --rm -d -p 7379:7379 webdis
8c8d875c11aa2d347942c9344c75c2369d0b56c96d12d8e425e6bbd28783328e

$ curl http://127.0.0.1:7379/SET/foo/hello-world ; echo
{"SET":[true,"OK"]}

$ curl http://127.0.0.1:7379/GET/foo ; echo
{"GET":"hello-world"}

$ curl -s http://127.0.0.1:7379/GET/foo.msg | xxd ; echo
00000000: 81a3 4745 54ab 6865 6c6c 6f2d 776f 726c  ..GET.hello-worl
00000010: 64                                       d

[tomgie]

Sorry on the late reply, I have been busy. I will test this out tonight.