Accept JSON payloads for POST requests

[amitrajput1992]

Hi @nicolasff, Recently started using webdis in my production application and it works flawlessly :1st_place_medal:

The application uses webdis as a communication bridge between the the API and the application web-socket layer. We generally publish small patches from the API layer which are eventually caught by a webdis subscriber in the web-socket layer. Also the API uses POST for all requests to webdis.

Patches are usually simple jsons of the form of immerjs patches

There is a weird issue that I encountered today while transmitting a typical patch. The patch json contained an http url and the message didn't get published to the subscriber and there were no errors on the webdis server (have set verbosity to 4) I usually do JSON.stringify(patch) before calling webdis server. My hunch is that this happens due to the fact that the payload is being transmitted as a string along with the redis command and there is some URL process to extract out the various parts from the payload.

A typical http call would look like this: http://webdis.server/PUBLISH/{patches: [...]}

Example patch json:

{
  "patches": [
    {
      "op": "replace",
      "path": [
        "records",
        "element",
        "map",
        "1636642488165",
        "props",
        "source"
      ],
      "value": {
        "file_urls": {
          "o": "https://u.vrgmetri.com/gb-sms-prod-1/media/2021-1/gmetri/6b68ee54-2354-4e25-bfe9-c367d5d2cf8a/o/Chocos.png"
        },
        "name": "Chocos.png",
        "type": "IMAGE",
        "size": "7318556"
      }
    }
  ]
}

This could be solved by allowing POST payload as json with the application/json header in request. This would allow the webserver to not parse the args and simply forward these to the redis server. Something like:

{
  command: "SET",
  db: 1,
  args: string | object
}

Although i was able to get around this problem by using encodeURIComponent(JSON.stringify(json))[while publishing] + JSON.parse(decodeURIComponent(json))[on subscribe], I wanted to hear your thoughts/plans on this.

[nicolasff]

Hi @amitrajput1992, sorry for the late reply.

Webdis uses a library called http_parser, which is really just a C file and its header that were extracted from Node (as in node.js) to provide a standalone HTTP parser. I can certainly see how this parser could get confused by a URL being present after the /PUBLISH. As you can imagine the same kind of problem occurs when submitting binary values to Webdis, like storing a whole file into a Redis key for example.

Webdis does provide a solution for this, in fact it's been there for over 10 years already: you can use PUT requests against Webdis, and pass in the *last* argument of your command as the request body. So in your case if the Redis command you're sending is PUBLISH channel message, you could send a PUT request to /PUBLISH/channel with the request body being the whole JSON object you want to publish.

This is documented in the README under "File upload", although this works for any kind of content, not just files of course.

Here's a demo with the JSON you provided, except using SET + GET instead of PUBLISH, but it works the same.

Upload:

$ curl -v --upload-file patch.json http://127.0.0.1:7379/SET/test-patch
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 7379 (#0)
> PUT /SET/test-patch HTTP/1.1
> Host: 127.0.0.1:7379
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Length: 459
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Server: Webdis
< Allow: GET,POST,PUT,OPTIONS
< Access-Control-Allow-Methods: GET,POST,PUT,OPTIONS
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: X-Requested-With, Content-Type, Authorization
< Content-Type: application/json
< ETag: "0db1124cf79ffeb80aff6d199d5822f8"
< Connection: Keep-Alive
< Content-Length: 19
<
* Connection #0 to host 127.0.0.1 left intact
{"SET":[true,"OK"]}* Closing connection 0

(notice the PUT /SET/test-patch command).

Retrieval & validation:

$ curl -s 'http://127.0.0.1:7379/GET/test-patch.bin' > output.json

$ shasum -a 256 patch.json output.json
89514b3aa53a854f78bf0ab3c6c3ed911997ec248b8ab64bc6ca59ff54a69dbb  patch.json
89514b3aa53a854f78bf0ab3c6c3ed911997ec248b8ab64bc6ca59ff54a69dbb  output.json

The two files have the same contents.

Note that the .bin prefix is there just to tell Webdis not to wrap the response in JSON but to return the exact value returned by Redis.

If you use the same kind of PUT request to send your PUBLISH messages, you shouldn't need to encode and decode the payloads. As for your suggestion to provide the Redis database, this is also supported: you can prefix your commands with a database number, so for database 1 the command becomes PUT /1/PUBLISH/test-patch.

I hope this helps!

[amitrajput1992]

Thanks for the detailed clarification. It didn't occur to me to use PUT request for complex JSON structures. As for the database index option, I did figure that out by going through the docs but thanks for adding this here again.

Let me try this out quickly!

[nicolasff]

@amitrajput1992 any update on this?

[amitrajput1992]

@nicolasff Yes, it did work! Thanks for all the help.

I do maintain a simple npm library to use webdis apis, it's far from complete but does contain the basic commands required. Adding a link here for anyone who stumbles upon this thread. https://github.com/gmetrixr/webdis-commands

[nicolasff]

@amitrajput1992 oh it's awesome to see this library! Are you planning to publish it to npm? Most people would want to install it this this way.

I haven't seen that many projects built around Webdis, but I know there are some. I'll see what I can find and if it makes sense to add a section in the README linking to them.

(by the way, your build status image is broken and the whole gmetri.io website looks down. I use GitHub Actions for this, and I've found that it works wells for this use case. Feel free to re-use the Webdis GitHub Actions config file if it helps you get started with this feature; it's pretty easy to set up, and they have badges too.)

[amitrajput1992]

@nicolasff It's already published here https://www.npmjs.com/package/@gmetrixr/webdis-commands I will be setting up an automated CI this week for build + npm publish. Right now, the build step is automated here https://drone-xr.gmetri.io/gmetrixr/webdis-commands, yet to add the npm publish step.

I was upgrading to the latest version of OS on our DC servers today and hence the downtime. :sweat_smile:

I did give github actions a go, but it was too slow/had restrictions for most of our public builds, and I ended up running a drone server with 2 nodes on our data center. Each of these nodes have a 4 core CPU and 8Gb RAM and it works flawlessly.