search

opendevstack / ods-quickstarters

Contains project quickstarters (boilerplates) for Nginx, Go, Java (Spring Boot), Python (Flask), Rust (Axum), Scala (Play), TypeScript (Express), Angular, Ionic, Jupyter, RShiny - all including CI/CD integration & Jenkins build agents
Apache License 2.0
26 stars 39 forks source link

Add ETL pipeline testing QS (e2e-python) #985

Closed cristianhkr closed 6 months ago

cristianhkr commented 7 months ago

Provide a new end to end testing quickstarter to test ETLs pipelines.

Technologies: -pytest -greatexpectations -allure -boto3 ...

metmajer commented 7 months ago

@braisvq1996 @cristianhkr not sure this QS is appealing to an open source audience. Please move it to GitHub.com/Boehringer-Ingelheim.

bassagan commented 7 months ago

@braisvq1996 @cristianhkr not sure this QS is appealing to an open-source audience. Please move it to GitHub.com/Boehringer-Ingelheim.

This QS does not need any infrastructure besides an AWS account and is not related to anything specific to Boehringer. This is a testing framework that executes its tests on the cloud, so it might fit an open-source community like AWS or Azure QS.

We can discuss it internally if you consider we should be moving it to the private repository.

metmajer commented 7 months ago

@braisvq1996 as discussed with @bassagan, the Quickstarter is generally usable and not only in our enterprise context. Good for me to take this up in our open source library.

braisvq1996 commented 6 months ago

Quickstarter has been tested and works properly with Release manager pipeline