As a project, I can pre-warm a processing cluster in advance of the work arriving to minimize process latency.

mike-gangl commented 1 year ago

On-demand Integration with SPS

The SPS is able to pre-warm itself and scale based on anticipated load. Integration with SPS Airflow system to scale the number of nodes/pods available to the airflow system,

Acceptance Criteria

Acceptance criteria required to implement the epic

Create pre-warm interface/endpoints for airflow SPS
update on-demand Rest API implementation to use eks scaling/pre-warming
Define end-to-end / validation test for OD work

Work Tickets

Link to work tickets required to implement the epic

[x] https://github.com/unity-sds/unity-project-management/issues/120

Dependencies

Other epics or outside tickets required for this to work

[ ] TBC

Associated Risks

links to risk issues associated with this epic

[ ] TBC

galenatjpl commented 2 months ago

Moving this to "Future Work" / (backlog this ticket) as this is part of the deprecated work. We may revisit this in the future. @pymonger @GodwinShen @mike-gangl

mike-gangl commented 2 months ago

Deprecating for now, we will re-create the tickets if needed.

unity-sds / unity-project-management