search

mesg-foundation / engine

Build apps or autonomous workflows with reusable, shareable integrations connecting any service, app, blockchain or decentralized network.
https://mesg.com/
Apache License 2.0
130 stars 13 forks source link

system services starting before Core's gRPC server starts #572

Closed ilgooz closed 5 years ago

ilgooz commented 5 years ago

/cc @krhubert reported this issue on Discord.

These lines are related with this issue.

System Services are just MESG services. They will want to connect to Core through gRPC APIs to listen for tasks. Because of that, gRPC server should be *()** started first. But this issue should not cause to a fatal error because when services cannot connect to gRPC server, they exit in the container. Docker services should try to restart existed services. But still, this is still an issue that needs to be solved.

There is a catch on starting system services after gRPC server started. gRPC server also depends on system services and propagates system services' instance to the api pkg. So, some of the gRPC API's are actually depending on system services to be started before accepting requests.

*()** As a result, both gRPC server and system services relies on each other.

The proper way of accepting requests for gRPC APIs is to first make sure that system services are up and running. To accomplish that in a clean way, we can add retry mechanisms to service clients while they try to connect gRPC server on start. So they won't exit immediately. Otherwise if we solve this issue in core we may need to do a few workarounds and complicate code a bit.

NicolasMahe commented 5 years ago

There is a catch on starting system services after gRPC server started. gRPC server also depends on system services and propagates system services' instance to the api pkg. So, some of the gRPC API's are actually depending on system services to be started before accepting requests.

I don't agree here. The gRPC server itself doesn't depends on system service, but yes, some gRPC API are depending on system services. The gRPC server should be start first, then the system services can be deployed and start. If an user is making a request before the system services are deployed, the request should simply fail.

Let's keep in mind that some system services will be optional in the future. The core should still be able to start, but yes, some API will not work, and it fine.

antho1404 commented 5 years ago

I think the best solution here should be:

Like that everything should be working for everyone, services starts when the api is ready and core api starts when services are started.

Anyway the implementation has changed a bit but the problem stays the same

krhubert commented 5 years ago

we agreed that system services shoudl be started first - todo close