search

OHDSI / WebAPI

OHDSI WebAPI contains all OHDSI services that can be called from OHDSI applications
Apache License 2.0
126 stars 156 forks source link

Problems with session when i create more than one webapi replicas #2356

Open jmvizcainoio opened 5 months ago

jmvizcainoio commented 5 months ago

Expected behavior

We are deploying atlas in a k8s cluster. When we scale more than one replica the webapi pod, the login process crash.

2024-01-31 17:34:37.126 ERROR http-nio-8080-exec-6 org.ohdsi.webapi.shiro.filters.ExceptionHandlerFilter - [] - Error during filtering
javax.servlet.ServletException: org.pac4j.core.exception.TechnicalException: State parameter is different from the one sent in authentication request. Session expired or possible threat of cross-site request forgery

We have configured google auth as a login provider and with one webapi replica works fine.

Is possible keep the session in a redis ? Where are the session stored ? Any suggestions?

konstjar commented 5 months ago

WebAPI does not support shared sessions. The workaround is just to run single pod. Technically, WebAPI uses pack4j and it has a flexible SessionStore implementation. But I'm not aware on any implementation of external session storage for WebAPI.

jmvizcainoio commented 5 months ago

We are trying to deploy Atlas in a high available environment but the problem with the sessions will be impossible.

konstjar commented 4 months ago

You can use sticky sessions on LB side.

jmvizcainoio commented 4 months ago

You can use sticky sessions on LB side.

use sticky sessions is a patch. We are using spot instances in a k8s cluster. Is very easy to had 10 instances and one of this die. The clients logged into the webapi that is on the died node must be logged again.

chrisknoll commented 3 months ago

Some changes at the core of WebAPI will be necessary in order to support a multi-instance / non-sticky hosting of WebAPI.

Main issue:

Running jobs are updated to Canceled on Startup: If WebAPI gets killed while background jobs are running, it's assumed that WebAPI itself is running those jobs and therefore should mark 'running' job as 'canceled'. If there is multi-host WebAPI running, then each instance that starts up will attempt to cancel jobs that could be managed by other nodes. So something needs to keep track of which nodes is running each job, either using a local filesystem for each node or some kind of unique identifier associated which each host so that the host that starts up will only kill their own jobs. Alternatively, we are moving towards an 'external execution' service such that this external service will live regardless of WebAPI status (and call-back to WebAPI when jobs are complete). That would allow WebAPI to not kill running jobs on startup, but introduces other challenges (ie: if the external service is done and calls back to WebAPI to notify of completion, and it WebAPI is done, we need a sort of 'retry' functionality).

Other Issues

Other issues include caching, where we want to share a cache across server processes, and sharing authentication information across processes.