pokt-foundation / global-services

Encapsulates all of the global services running to support the portal-api
MIT License
0 stars 1 forks source link

Split services to add CLI and Lambda versions #14

Closed rem1niscence closed 2 years ago

rem1niscence commented 2 years ago

Problem: The lambda of run-application-checks is having issues due to the concurrency needed to perform, the issue it is mainly having is an error that return a message of the sort: socket: too many files or cache: too many files, that is due to hitting the lambda limits regarding file descriptors, in order to not trigger the error, we'd need to set the concurrency to a level so low that by the time the checks are done, they're already outdated.

Solution: For that a wider approach was applied, in order to follow the philosophy of not completely rely on lambda, a CLI and lambda version is now out for several services, with some of them only having a lambda version atm as it does not make sense to run them in CLI, those who have two versions are extending the same base file and are only changed to run in the environment they're meant to be deployed.

This gives the ability to extend to even more services (for example GCP Cloud functions) if that were needed in the future without affecting the base implementation or by simply modifying the base implementation which will then be also available for all the versions.


Back to the run-application-checks service lambda implementation, it is now split into two lambdas, the first one obtains and gathers all the info needed for the apps and triggers a second one for each app to actually perform the checks, waiting for the result and saving it back to redis, avoiding the limit as now each check runs on its own lambda.

Eren-Flies-Freely-In-The-Sky Figure 1.1: Representation of the checks now that they don't have concurrency issues due to them having the whole lambda environment for themselves

height[bot] commented 2 years ago

Link Height tasks by mentioning a task ID in the pull request title or commit messages, or description and comments with the keyword link (e.g. "Link T-123").

💡Tip: You can also use "Close T-X" to automatically close a task when the pull request is merged.