cyphernet-labs / rust-microservices

Internet2 microservices framework
Apache License 2.0
11 stars 7 forks source link

Ping/ponging opt-in feature at the controller level #5

Open zkao opened 2 years ago

zkao commented 2 years ago

We hit this logged error when testing whether a service is online, discussion here, in case it is not. It would be good to have a function for that that doesn't log, so that the broker can test whether other services are up and running.

zkao commented 2 years ago

The actual error is: service is offline or not responding

dr-orlovsky commented 2 years ago

I am afraid ZMQ architecture / protocol design prevents us from doing that. The whole idea of ZMQ sockets is to be automatically re-connectable, i.e. services are allowed to be offline. The only way to test connectivity is to send a message to the other service. Probably we will need some sort of ping/pong workflow internally in the library for that...

zkao commented 2 years ago

The only way to test connectivity is to send a message to the other service.

Correct

Probably we will need some sort of ping/pong workflow internally in the library for that...

Not sure a ping/pong is needed. Just a on-demand way to send message to another service that doesn't log errors.

zkao commented 2 years ago

something like this, its copy/paste/modify from send_to function

impl<A> Endpoint<A>
where
    A: ServiceAddress,
{
    pub(self) fn dest_up(&mut self, source: A, dest: A) -> bool {
        // what to put as data, and how to handle receiving it?
        let data = vec![];
        let router = match self.router {
            None => {
                trace!("Sending {} from {} to {} directly", request, source, dest,);
                dest.clone()
            }
            Some(ref router) if &source == router => {
                trace!("Routing {} from {} to {}", request, source, dest,);
                dest.clone()
            }
            Some(ref router) => {
                trace!("Sending {} from {} to {} via router {}", request, source, dest, router,);
                router.clone()
            }
        };
        let src = source.clone();
        let dst = dest.clone();
        match self.session.send_routed_message(&source.into(), &router.into(), &dest.into(), &data)
        {
            Ok(()) => true,
            _ => false,
        }
    }
}
dr-orlovsky commented 2 years ago

Sorry, I am not getting. Is the logging a problem? If yes, it can be controlled in a standard way with [env_logger](https://docs.rs/env_logger/0.9.0/env_logger/) crate by setting filters via environment variables

zkao commented 2 years ago

I find the requested feature questionable. The intend was to distinguish between normal error logging, and add the ability to probe for services online by some sort of microservice manager, for example, without logging (this was a concern from another contributor).

I will describe the actual problem.

  1. service A suspects service B is down
  2. so A probes B to see if it is up by sending a msg 3a. A detects B is down, and relaunch it 3b. or A detects B is up, and do nothing

the fn above dest_up probes whether service dest is up, and returns a bool.

dr-orlovsky commented 2 years ago

If you will send just an message like in the dest_up fn, it will be a failure on the receiving part since it won't be aple to parse it. So the real way of doing that is ping/ponging, and probably proper way of implementation would be make ping/ponging opt-in feature at the controller level, such that controller will call some Handler dedicated callback and the managing daemon will be able to re-launch failed service. I will work on that, but feel free to do the PR if you'd like.

I think we can leave the issue opened, just change its name

zkao commented 2 years ago

I will work on that, but feel free to do the PR if you'd like.

Please go ahead, you know exactly how to integrate it in the larger framework :)