search

kiwicom / the-zoo

🐒 A service registry for humans
MIT License
93 stars 6 forks source link

Polyglot check development support #120

Open aexvir opened 5 years ago

aexvir commented 5 years ago

As of right now the check functions are written in Python, like the rest of the application. It's natural to write the checks in the same language as the web application, as the interaction is simple and immediate.

If we aim for offering The Zoo as a universal platform for cataloguing and auditing services limiting the language the checks have to be written into, to a single language, makes it harder to use by people/companies that are not familiar with the language.

As we are already using Celery for distributing the check execution, and there are already celery implementations for languages like go and js (node) it probably wouldn't be that hard to just facilitate and document how people could write checks using those languages, but I think the best approach, in the long run, would be to just use some generic AMPQ implementations like RabbitMQ.

The only part that would require some deeper refactoring would be the API and endpoint calls that require to synchronously run the checks, which right now just run everything on the app server (which is something that I personally don't like at all). For this part the solution would require some tweaks to keep that "instantness", it could be done by having different queues with different priorities for the workers to pick up jobs from, or using something like gRPC.

This issue is not meant to be worked on as of right now, I just had this in mind for a long time and I wanted to have it written down.

JanBednarik commented 5 years ago

It would be nice to have mix of checks written in different languages. But instead of Celery workers for different languages (or different kind of generic workers implementation), run all checks for different languages from the same Python task or API call. Run them all for one repository pull.

CheckContext should contain values which can be serialized (JSON, MessagePack, Protocol Buffers, ...). Python can run subprocess with language specific checks runner script, give it this serialized context and expect results and errors as output.

It can also run Docker images with language specific checks instead of scripts. So runtimes for different languages does not have to be installed in the Zoo Docker image. Just possibility to run Docker in Docker will be enough.

As these language specific runners does not have to implement Celery (or something like that) and not have to care about things like pulling repo archive, they can be fairly minimalistic. Which opens possibilities to quick adopting of wider range of languages (Lua, Clojure, PHP, ...).

aexvir commented 5 years ago

I also considered running Python workers with Subprocess but I don't like that solution, even if it's easier to "plug" any language. It would work for the current use case but if we want to make a more advanced communication workflow between application and workers we'll be limited to what we can do.

But both approaches are not incompatible, we could easily provide a "subprocess" worker that would allow just calling whatever, but I'd hate to have to use that for our current checks to be honest.

The main point of this issue is having the workers more decoupled from the app in a way that they could quite easily be swapped with an implementation in a different language, or some python workers using subprocess to call other code if someone requires that.