Support vector data ingest

[paridhi-parajuli]

Summary: Refactor code to support vector data ingest Addresses Issue

Changes

• Generalized code for vector data ingest and removed eis-related functionalies.
• Change default value for EXTERNAL_ROLE_ARN passed to the vector ingest ECS
• Change to mwaa_stack_conf for the network configuration of vector ingest ECS
• Add configs for loading data into features database
• Add function to assume AWS IAM role and return temporary credentials

PR Checklist

• [ ] Update CHANGELOG
• [ ] Unit tests
• [ ] Ad-hoc testing - Deploy changes and test manually
• [ ] Integration tests

[ranchodeluxe]

Hey @paridhi-parajuli! Sorry about the confusion here. Lots of moving parts

So until we have a flow for users to create their own DAGs (like the CSDA project does and which is far into the future for VEDA) it's expected that certain "clients" might have more custom ingestion rules just like some of the code being removed in load_to_featuresdb_eis does for the EIS fire group

What was already in here has a "generic" flow for load_to_featuresdb that keys off the s3 bucket prefix being discovered. I know @slesaad didn't want this in GHG code. So if we don't like that pattern then we should probably create a whole new DAG or conditionally force the DAG down a different workflow that just has the "generic" flow split out.

Does that make sense? But we definitely need most of the code for that pathway back

[slesaad]

Hey @paridhi-parajuli! Sorry about the confusion here. Lots of moving parts

So until we have a flow for users to create their own DAGs (like the CSDA project does and which is far into the future for VEDA) it's expected that certain "clients" might have more custom ingestion rules just like some of the code being removed in load_to_featuresdb_eis does for the EIS fire group

What was already in here has a "generic" flow for load_to_featuresdb that keys off the s3 bucket prefix being discovered. I know @slesaad didn't want this in GHG code. So if we don't like that pattern then we should probably create a whole new DAG or conditionally force the DAG down a different workflow that just has the "generic" flow split out.

Does that make sense? But we definitely need most of the code for that pathway back

@paridhi-parajuli that's right, maybe I wasn't clear enough with the ask. Create a new dag for the generic stuff, we'll still support the existing dag with the custom logic.