register service ids lost after gorb crashes

[dongshengz]

What I have noticed is if the gorb cashes or gets killed, all the registered service ids are lost. It seems all the services states are saved in-memory and it might require a database to save the states. But I still wonder if there could be other easy solutions.

[kobolog]

So what can be done in the first place is to load all active IPVS rules on start. GORB has the option to flush IPVS tables on start, but if it's not specified then the state-of-the-world picture will be different between GORB and actual IPVS table state, if there were some existing IPVS rules present on the system.

Also, as you already noticed, to preserve the world state across machine restarts, this state has to be saved somewhere. Since GORB already has some minimal integration with Consul for service discovery, I'd just use Consul for world state persistence too.

By the way, contributions are welcome =)

[dongshengz]

sure. I will take a look how to save the state in a minimal way.

[leslie-wang]

the only data need persist is "service name". Actually another option is that service name is created by GORB using convention _. Same for backend. The restful API need to be changed, and not allowed caller to specify service name, backend name. GORB will create using these convention. Thus, during reboot, GORB can load from ipvs, and rebuild the list.

[kobolog]

I don't understand how we could recover the full service ID based only on the information loaded from IPVS – we'll have IPs, ports, flags and mode, but not the actual service name. Do you want to just drop the service name from the ID?

[leslie-wang]

yes. that is what I'm trying to say, thus, gorb can avoid storing this <service ID, service name> mapping. if using convention like , creating service call can still use same json payload: PUT /service/

{
    "host": "10.0.0.1",
    "port": 12345,
    "protocol": "tcp",
    "method": "rr",
    "persistent": true
}

gorb can return following response:

{
    "service": "tcp_10-0-0-1_12345",
}

or

{
    "service": "tcp-10_0_0_1-12345",
}

same applies for backend part. just my suggestion for the fix.

[kobolog]

Yeah, this is possible although wouldn't make it hard to understand what's going on, e.g. parsing the healthcheck results for user display, without knowing the service name? There's still a 1-to-1 correlation between an entry and a service via host-port-protocol tuple, but you'll need to maintain a separate DB somewhere where you'd be able to look up the tuple-to-service mapping.

[leslie-wang]

Yes, agree. The 1-to-1 DB may sit into another orchestration system or micro-service invoking gorb API. Where it sits depends on how you define the architect of gorb.

• If GORB is standalone distributed at each node without talking to each other, seems like it is no use to maintain the DB entry because kernel already has such list;
• If you want GORB to become cluster and have a controller to talk each GORB worker, then GORB itself need a DB to maintain all cluster wide data.

Actually this is also related with https://github.com/kobolog/gorb/issues/32 about the way to integrate with k8s.

[kobolog]

39 is merged.