search

typhon-project / typhonql

Typhon Query Language
Eclipse Public License 2.0
4 stars 1 forks source link

[BUG] After restarting the typhonql-server it never gets initialized if the polystore isn't also restarted #68

Closed DavyLandman closed 4 years ago

DavyLandman commented 4 years ago

Describe the bug

When the typhonql-server restarts, it needs to get the ML & DL model from the polystore-api. Currently it waits for a call form the polystore-api to set this up, but the API doesn't know when the typhonql-server is restarted, so it never resends it.

Expected behavior

When the typhonql-server restarts (like for example after a container update), it should be ready to handle incoming requests.

REST API

typhonql-server             | 06:07:46.708 [qtp1092572064-13] ERROR engineering.swat.typhonql.server.QLRestServer - Failed to handle response
typhonql-server             | java.io.IOException: Backend is not initialized yet
typhonql-server             |   at engineering.swat.typhonql.server.QueryEngine.getBackend(QueryEngine.java:23) ~[classes/:?]
typhonql-server             |   at engineering.swat.typhonql.server.QueryEngine.resetDatabase(QueryEngine.java:47) ~[classes/:?]

Work-around Follow a docker-compose up -d with a docker-compose restart typhon-polystore-api

Possible solution Will have to discuss with @OrfenCLMS about which strategy is preferred:

  1. Change the REST interface of the QL server that it always receives the ML & DL information on every call.
  2. On startup of the server, we send a message to the polystore-api to register a fresh typhonql-server (not sure if we can always know where the polystore-api server is running, especially in relation to k8s, see: typhon-project/typhondl#24)
  3. Implement a ping/healthcheck feature from the polystore-api, so that it knows when a service has restarted.
OrfenCLMS commented 4 years ago

This will need to have a more in depth discussion, but regarding potential solution 1, this would make all requests a bit bloated without needing the full ML & DL models apart from edge cases (QL server does not restart that often i assume). Additionally, the ML & DL models can potentially be really big (probably not, but still). Still, this could be used as a temporary workaround until/unless we think of a better solution.

DavyLandman commented 4 years ago

I agree, option 1 is a work around, but it does keep it more stateless, and avoids cyclic dependencies between the API & QLServer.

DavyLandman commented 4 years ago

happy to say this problem has been solved (h/t @OrfenCLMS). We decided to always send along the ML & DL model over every call to the rest server. This allowed the whole QLServer to become stateless, which also helps for the plans for k8s (see typhon-project/typhon-polystore-api#14)