System test e2e

[mike-gangl]

Purpose

• Add an end to end test to test applicaiton packaging, execution, and data availability.
• Mirrors our existing Tutorial step.

  Testing

• created an environment config based on env.cfg for the test envionrment
• Ran multiple tests from end-to-end

[mike-gangl]

I've added a number of you to be cognizent over this work. i'll be doing a demo on Tuesday at 11.

@edwinsarkissian and @mcduffie - the packaging feature is called through an openapi built client. this is the contract between clients and the APGS. it's built to reflect what exists today. Going forward, changes to the API need to be reflected there.

@LucaCinquini and @drewm-jpl - we aren't going through the OGC endpoints just yet for the test, we call airflow directly. this isn't ideal, but it shows that it works and can run a generated package.

@ngachung - the output of the process creates a file that has a timestamp added to it. we add it to the same collection, and that's how we can tell the new file is being generated. i simply query the U-DS for the existence of the file to mark this as true.

All- These tests take a while. maybe that's ok, but app-pack-gen takes ~6 minutes for it all to work and showup. The big bottleneck here is the docker build portion which takes awhile. The processing and checks for data take ~10 minutes. That's allowing for spot instance spin up, execution, and then pinging for data. The runs give times for each step (scroll right to see the timing if it's not on your screen).

    And I wait for the apgs build to complete             # features/steps/apgs/steps.py:63 364.968s

And the workflow successfully completes               # features/steps/sps/steps.py:107 454.330s
    And the workflow data shows up in the data catalog            # features/steps/sps/steps.py:124 162.527s