Issue with global store update?

[remydewa]

Hello,

I've just deployed my Easyauth Server project in kubernetes. I have actually 2 pods running. But I don't know why, if I understood, it seems I have an issue with the global store update. The service token I generate do not share between pods. However when the pod start I saw this kind of logs

[client-api-extz-auth-server-cdc5b9d6b-88lwh] 03-16 08:46 EasyRpc-server /ws/manager WARNING  creating cron or get_downstream_registered_functions - interval 30 
[client-api-extz-auth-server-cdc5b9d6b-88lwh] 03-16 08:46 EasyRpc-server /ws/manager WARNING  global_store_update - update - tokens - add3d428-150f-43e9-9c3b-44d9a32f3cca 
[client-api-extz-auth-server-cdc5b9d6b-88lwh] 03-16 08:46 EasyRpc-server /ws/manager WARNING  global_store_update - update - tokens - add3d428-150f-43e9-9c3b-44d9a32f3cca 
[client-api-extz-auth-server-cdc5b9d6b-88lwh] 03-16 08:46 EasyRpc-server /ws/manager WARNING  global_store_update - update - tokens - 0ff378fb-21b0-4a12-8216-01c7143b1251 
[client-api-extz-auth-server-cdc5b9d6b-88lwh] 03-16 08:46 EasyRpc-server /ws/manager WARNING  global_store_update - update - tokens - 0ff378fb-21b0-4a12-8216-01c7143b1251 
[client-api-extz-auth-server-cdc5b9d6b-88lwh] 03-16 08:46 EasyRpc-server /ws/manager WARNING  global_store_update - update - tokens - 322be937-c3cd-45b5-9246-97c676b7b50c 
[client-api-extz-auth-server-cdc5b9d6b-88lwh] 03-16 08:46 EasyRpc-server /ws/manager WARNING  global_store_update - update - tokens - 322be937-c3cd-45b5-9246-97c676b7b50c 

That seems great doesn't it? But when I call and endpoint which needs sepcific action sometimes it works, sometines not. Depending to which pod I'm redirected when I call my endpoint. Example:

{'header': {'error': {'message': "not authorized, permissions required: {'actions': ['CAN_CREATE_FREEMIUM_IA_CUSTOMER']}",
   'status': 403},
  'date': '2023-03-15T17:53:32.351900+00:00',
  'transaction_id': '35d4d14c-b385-4fc2-979f-8aac20b815e3',
  'request_url': 'http://xxx-yyy.com/v2/cus/create'}}

Thanks for your help,

Regards,

Rémy

[codemation]

Hey @remydewa , you mentioned 2 pods, did you scale the EasyAuthServer instance to 2 pods?

The Pod running the EasyAuthServer can be scaled by adding workers. The mechanism that keeps track of "valid" tokens & replication requires a central pod extended by EasyAuthClients( client that connects to EasyAuthServer to access API/s and receive valid token) which can be scaled.

[remydewa]

Hey @codemation

Thanks for your quick answer :). Yes I have 2 pods of my EasyAuthServer with 2 workers (so 4 workers in total) to avoid single point of failure in my production environment.

In fact I did others tests about service token. If I scale down my EasyAuthServer to 1 pod, then create a new service token and scale up my EasyAuthServer back to 2, it's work because tokens are pushed in global store when pod/app starts so if I need to create a new service token I can create it and then rollout my pods and It works. As, create service account is not something I do often It's ok for me now I know what I have to do.

For user token, it's different but I have lots of chance that my HAProxy redirect me to the good pod when I login and call an api just after.

But may be to improve that, we can work to replace the global store by a redis db? What do you think about that? Do you think it's something possible?

Many thanks,

Rémy

[codemation]

Indeed, I think allowing EasyAuthServer to scale is important.

The issue is rooted from the ability to "revoke" tokens. The easy solution when creating tokens is simply check for token existence in DB, if not found in either memory or DB, then its likely revoked.

Using a central EasyAuthServer allows in-memory token checks & db checks, but while there is only 1 source of truth. If a token is revoked with multiple EasyAuthServer's then only the clients connected to the EasyAuthServer that revoked a token would be aware, until restarted.

So there is a trade off that must be made, without introducing a new dependency, both library and infra: Either:

A - For every request, check that a token_id exists in DB. Requiring all EasyAuthClient's to invoke a rpc call to EasyAuthServer checking this validity.

or

Current Implementation: B - For every request, check memory for existence of token-id. If not found, the token is deemed invalid. (could add an rpc call that validates token is revoked). New tokens & token_id's are replicated to all connected clients so their in memory token_ids are updated.

The obvious upside to B is checking memory instead of DB every.

Upside to A - is scalability of EasyAuthServer

I am leaning towards A, as I am not super happy about storing token_id's in memory at every endpoint.

[codemation]

Fixed in 2.0.2