Airflow Scaling improvements + Performance Testing

[mike-gangl]

Airflow Scaling improvements

Two major use cases to achieve here:

• As a project manager, when no workflows are running in the system, i want the minimal number and types of instance to be running to reduce costs.
• As an operator, when jobs are submitted, if no workers are able to process them, they should automatically spin up and process job data

Essentially, an idle system should have no workers running.

Acceptance Criteria

Acceptance criteria required to implement the epic

• Airflow scheduler, executor and web service can run in an "always on" capacity, but worker instances can have 0 or more instances at any given time.
• Airflow workers schedule up to meet the demand of jobs - e.g. scale up to pre-defined limit if number of jobs. Naively, we can scale up a worker for each job.
• Airflow workers go offline if when there are no jobs queued for them
• Execute Performance test based on TIR numbers
• Execute Performance test based on VSWIR numbers

Work Tickets

Link to work tickets required to implement the epic

• [ ] TBC (to be created)

Dependencies

Other epics or outside tickets required for this to work

• [ ] TBC

Associated Risks

links to risk issues associated with this epic

• [ ] TBC

[mike-gangl]

The testing numbers we're looking at are as follows (from @hookhua )

SBG TIR • forward processing

• ~6.2K PGE jobs per day
• ~44.6TB per day of L0 to L4 products
• ~148 parallel compute instances • bulk processing
• ~18.7K PGE jobs per day
• ~134TB per day of L0 to L4 products
• ~444 parallel compute instances • low-latency processing
• ~2.9K PGE jobs per day
• ~35.4TB per day of L0 to L4 products
• ~80 parallel compute instances SBG VSWIR • forward processing
• ~21.9K PGE jobs per day
• ~56.5TB per day of L0 to L2 products
• ~577 parallel compute instances • bulk processing
• ~65,6K PGE jobs per day
• ~169.5TB per day of L0 to L4 products
• ~1,731 parallel compute instances

For a single SBG workflow as built in our system (L0 -> L2A):

5 + 1 PGEs Data:

1. L1B-preprocess = ~4.2GB
2. ISOFIT (RFL) =. 8.2GB
3. Resampled Reflectance (RSFL) = 6GB
4. Corrected Reflectance = 3GB
5. Fractional cover = 25MB

If we include the 'optional' veg-biochem workflow as well, we'd add an additional PGE

6. VEGBIOCHEM = 45MB

So for any product, we could run 5PGEs and move ~19GB through the system. This equates to:

TIR 1240 workflows 23,560GB of data

VSWIR 4380 workflows 83,220GB of data