Support of datastore in additional languages

[pyhedgehog]

At least documentation should be upgraded to include:

• [ ] what redis client library is installed for each language? redis-py?
• [ ] what host:port should be passed? 127.0.0.1:6379?
• [ ] what string should be used as root part of key? username?

[Marak]

In order to make the datastore work for other languages will require a slight refactor.

Instead of requiring that the service use a redis client, the service instead should only be responsible for passing JSON messages back to it's parent process, which in turn can operate the cloud datastore.

An alternate implementation would be providing a hook.io sdk binary inside of the chroot jail that the process could call with a spawn wrapper.

I created an additional issue #145 which is essentially enabling multiplex streams over STDIO.

I have to think about this a bit, because I'm not sure which is going to be the better implementation.

There are a few goals:

• Minimize the complexity of language support binaries
  • Less code in run-hook-* binaries is better
  • No need to write API clients in every language
• Minimize worker load on master server
  • Ideally, the server should do as little work as possible
  • Let the worker handle all i/o for the service
• Do no introduce additional complexity to child process spawn lifecycle
  • Multiplexing streams over STDIO will create additional complexity
  • The current stream implementation is clean, fast, and working

With these goals in mind, we'll have to find the best compromise.

@pyhedgehog -

I'll have to consider this a bit before moving forward.

We are actually looking for people now to help manage the implementation of each new individual programming language on hook.io. see: https://hook.io/blog/new-multi-language-support#seeking-developer-community-managers

If you are interested in helping with the python support, please feel free to join us in #hook.io on irc.freenode.net

[Marak]

@pyhedgehog -

I did a bit more research.

I think implementing IPC via child_process.send() could be the best option.

see: https://nodejs.org/api/child_process.html#child_process_options_stdio

This would require that the python run-hook-python binary would be capable of receiving and sending JSON messages from the parent node process. Is this possible? I'm not entirely sure I think yes.

It would really help out if you could do a bit of experimentation with Node spawning python scripts and communicating over STDIO and child_process.send() . If you can prove that this type of implementation will actually work, I will make it a priority to implement.

[pyhedgehog]

Why not. As far as I know JSON designed so messages doesn't contains '\0' characters. Which makes them ideal message separators. It would be better to left FD 0, 1 and 2 intact and define 3 and 4 as input and output. It's rather standard decision - socat's EXEC2: command uses it.

[pyhedgehog]

BTW:

1. https://hook.io/blog/new-multi-language-support refers Bash and Perl twice.
2. I've joined #hook.io via webchat...
3. I think that I have enough knowledge about python and bash (and C as everybody) to help with.
4. When you will be ready for i18n I can help with Russian translation.

[Marak]

I've currently got IPC working locally over stdio 3 and 4.

The issue I have now is actually implementing a callback API over stdio message passing interface...err...

For example, if the user calls datastore.get over IPC in a node.js script, how do we implement a callback pattern? It would be non-trivial, involving message IDs and polling. Pretty much implementing: https://github.com/substack/dnode-protocol

This is additional complicated by the fact that most target languages besides JavaScript are not going to have callback style interfaces readily available.

This seems to be a problem. We need to find a solution which will be portable and not require a large amount of code. It might be best to keep all IPC communication as message passing style ( with no guarantee of message response ). datastore now has an HTTP API, so we could probably ask users to make HTTP requests to the datastore HTTP api from inside their services.

@pyhedgehog - What do you think?

[Marak]

So I've opted for moving all datastore logic into an HTTP client library. For node services, this will be a seamless API transition ( all javascript datastore services will remain the same ).

For other services, you'll need to make an authenticated http request with the new env.accessKey which will need to be passed in the outgoing request.

We still have the option to using process messaging over STDIO, but I will only use this for one-way messages with no guarantee or response. This is a much better API design for many many reasons.

We should have the ability for python to talk to datastore very soon.

[Marak]

@pyhedgehog

Additional languages can now talk to the datastore inside any hook service.

In order to enable this for python, we'll need a python version of hook.io-sdk built.

It should be pretty easy to port the JavaScript version. If you can put together the basics I will merge the code into the sdk repo and get it on production.

[Marak]

@pyhedgehog ping. if we port hook.io-sdk to a python version then datastore ( and all other API methods ) would work from python.

[pyhedgehog]

As I told you in bigcompany/hook.io-sdk#3 - I'm already in progress. Hook stubs (run, create, update, destroy) already works. As soon as all objects will be there I'll publish it.

[Marak]

Most excellent! Thank you so much. :-)