matthewbarreiro
commented
2 months ago On a related note - my fix will only be for pip3 download failures. It may be worth considering using set -e or similar for this and similar commands, but that is presumably a larger, internal conversation.
Running
./mwaa-local-env package-requirementswill always result in an exit code of0, even on failure.Steps to Reproduce: Note: All steps were performed using v2.8.1, but the behavior should be the same for all versions that include this function (v2.2.2+).
requirements.txtto force a failure. E.g. addapache-airflow-providers-datadog==3.5.0to therequirements.txtincluded in the repo.Run
./mwaa-local-env package-requirementsand check exit code.Click here for shell output
```bash ➜ aws-mwaa-local-runner git:(v2.8.1) ✗ ./mwaa-local-env package-requirements Container amazon/mwaa-local:2_8 exists. Skipping build Packaging requirements.txt into plugins Collecting apache-airflow-providers-snowflake==5.2.1 Downloading apache_airflow_providers_snowflake-5.2.1-py3-none-any.whl (37 kB) Collecting apache-airflow-providers-mysql==5.5.1 Downloading apache_airflow_providers_mysql-5.5.1-py3-none-any.whl (25 kB) ERROR: Cannot install apache-airflow-providers-datadog==3.5.0 because these package versions have conflicting dependencies. The conflict is caused by: The user requested apache-airflow-providers-datadog==3.5.0 The user requested (constraint) apache-airflow-providers-datadog==3.5.1 To fix this you could try to: 1. loosen the range of package versions you've specified 4. remove package versions to allow pip attempt to solve the dependency conflict ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts [notice] A new release of pip available: 22.3.1 -> 24.2 [notice] To update, run: pip install --upgrade pip updating: README.md (deflated 37%) ➜ aws-mwaa-local-runner git:(v2.8.1) ✗ echo $? 0 ```Note that
./mwaa-local-env test-requirementsdoes have a proper non-zero exit code on failure. However both commands should have a non-zero exit code on failure.Click here for shell output
```bash ➜ aws-mwaa-local-runner git:(v2.8.1) ✗ ./mwaa-local-env test-requirements Container amazon/mwaa-local:2_8 exists. Skipping build --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.8.1/constraints-3.11.txt" Installing requirements.txt Collecting apache-airflow-providers-snowflake==5.2.1 Downloading apache_airflow_providers_snowflake-5.2.1-py3-none-any.whl (37 kB) Collecting apache-airflow-providers-mysql==5.5.1 Downloading apache_airflow_providers_mysql-5.5.1-py3-none-any.whl (25 kB) ERROR: Cannot install apache-airflow-providers-datadog==3.5.0 because these package versions have conflicting dependencies. The conflict is caused by: The user requested apache-airflow-providers-datadog==3.5.0 The user requested (constraint) apache-airflow-providers-datadog==3.5.1 To fix this you could try to: 1. loosen the range of package versions you've specified 2. remove package versions to allow pip attempt to solve the dependency conflict ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts [notice] A new release of pip available: 22.3.1 -> 24.2 [notice] To update, run: pip install --upgrade pip ➜ aws-mwaa-local-runner git:(v2.8.1) ✗ echo $? 1 ```Cause: The issue stems from the
package_requirements()function inentrypoint.sh. Specifically, there is no check thatpip3 download -r ...completed successfully. The script will just attempt to zip whatever was downloaded and continue on it's way.Fix: I'm working on a fix for this and will open a PR to v2.9.2 when it is ready. I kindly request the fix be merged to lower branches (either all v2.2.2+ or at least to the branches for versions that are still supported).
Thank you!