Optimize User Activity Tracking by Directly Writing to Redis from Traefik Middleware

[mohamed-abdelrhman]

Issue Description

we expose HTTP endpoint

endpoints := v1.Group("endpoints")
endpoints.Post("/stats", middleware.CrossoverAPIKeyProtected, endpoint.WriteStats)

that allows the Traefik crossover middleware to write the user activity using API calls guarded by an API key. This involves high number of calls which introduce unnecessary latency and load on the core-api.

As improvements i propose to remove this HTTP endpoint and Traefik middleware should writes user activity data directly to Redis. The core-api can retrieve this activity data directly from Redis.

Proposed Changes

• Remove the current HTTP endpoint used by the middleware to write user activity
• Modify Traefik middleware to write user activity directly to Redis
• core-api should fetch the user activity directly from Redis

Objectives

• Enhance Performance writing directly to Redis, will reduce the response time for writing user activities
• Reduce the API call to core-api
• Easier clean up for legacy activity as we only interested in the users monthly activity and using TTL on redis writes will make it easier to clean this activity

Challenges

• Redis it is not traditionally used as a logging database due to its in-memory nature.
• The data is expected to grow over time and might not fit entirely in memory

[mohamed-abdelrhman]

Close as not planned for now.