In compatible requirements with many other packages

[UriSShavit]

Describe the bug: I added the usage of the package to my Airflow service. When trying to compile all the packages dependencies together, using pip-compile (from pip-tools package), there are conflicts with many other packages: When trying to compile the requirements with google-ads version 14.0.1, the proto-plus package is locked on version 1.18.1, while all the other packages consist a higher version: There are incompatible versions in the resolved dependencies: proto-plus<2.0.0dev,>=1.22.0 (from google-cloud-memcache==1.4.4->apache-airflow-providers-google==6.8.0->-r requirements.in (line 10)) proto-plus<2.0.0dev,>=1.22.0 (from google-cloud-pubsub==2.13.10->apache-airflow-providers-google==6.8.0->-r requirements.in (line 10)) proto-plus<2.0.0dev,>=1.22.0 (from google-cloud-dataproc==5.0.3->apache-airflow-providers-google==6.8.0->-r requirements.in (line 10)) proto-plus<2.0.0dev,>=1.22.0 (from google-cloud-dataproc-metastore==1.6.3->apache-airflow-providers-google==6.8.0->-r requirements.in (line 10)) proto-plus<2.0.0dev,>=1.22.0 (from google-cloud-build==3.9.3->apache-airflow-providers-google==6.8.0->-r requirements.in (line 10)) proto-plus<2.0.0dev,>=1.22.0 (from google-cloud-dataplex==1.1.3->apache-airflow-providers-google==6.8.0->-r requirements.in (line 10)) proto-plus<2.0.0dev,>=1.22.0 (from google-cloud-redis==2.9.3->apache-airflow-providers-google==6.8.0->-r requirements.in (line 10)) proto-plus<2.0.0dev,>=1.22.0 (from google-cloud-datacatalog==3.9.3->apache-airflow-providers-google==6.8.0->-r requirements.in (line 10)) proto-plus<2.0.0dev,>=1.22.0 (from google-cloud-bigquery-datatransfer==3.7.3->apache-airflow-providers-google==6.8.0->-r requirements.in (line 10)) proto-plus<2.0.0dev,>=1.22.0 (from google-cloud-os-login==2.7.4->apache-airflow-providers-google==6.8.0->-r requirements.in (line 10)) proto-plus<2.0.0dev,>=1.22.0 (from google-cloud-tasks==2.10.4->apache-airflow-providers-google==6.8.0->-r requirements.in (line 10)) proto-plus<2.0.0dev,>=1.22.0 (from google-cloud-aiplatform==1.18.2->apache-airflow-providers-google==6.8.0->-r requirements.in (line 10)) proto-plus==1.18.1 (from google-ads==14.0.0->apache-airflow-providers-google==6.8.0->-r requirements.in (line 10)) proto-plus<2.0.0dev,>=1.22.0 (from google-cloud-monitoring==2.11.3->apache-airflow-providers-google==6.8.0->-r requirements.in (line 10)) proto-plus<2.0.0dev,>=1.22.0 (from google-cloud-orchestration-airflow==1.4.4->apache-airflow-providers-google==6.8.0->-r requirements.in (line 10)) proto-plus<2.0.0dev,>=1.22.0 (from google-cloud-automl==2.8.3->apache-airflow-providers-google==6.8.0->-r requirements.in (line 10)) proto-plus<2.0.0dev,>=1.22.0 (from google-cloud-workflows==1.7.4->apache-airflow-providers-google==6.8.0->-r requirements.in (line 10)) proto-plus<2.0.0dev,>=1.22.0 (from google-cloud-kms==2.12.3->apache-airflow-providers-google==6.8.0->-r requirements.in (line 10)) proto-plus<2.0.0dev,>=1.22.0 (from google-cloud-logging==3.2.5->apache-airflow-providers-google==6.8.0->-r requirements.in (line 10))

When trying to compile any higher version of google-ads, protobuf version is set to be too high compared to all the other standard packages, I get:

There are incompatible versions in the resolved dependencies: protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-kms==2.12.3->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-aiplatform==1.18.2->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-dataplex==1.1.3->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf<4.0.0dev (from google-cloud-dlp==1.0.2->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-container==2.12.2->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf<4.0.0dev (from google-cloud-secret-manager==1.0.2->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf<4.0.0dev (from google-cloud-language==1.3.2->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-pubsub==2.13.10->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf<4.0.0dev (from google-cloud-speech==1.3.4->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf<4.0.0dev (from google-cloud-translate==1.7.2->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-bigquery-datatransfer==3.7.3->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-automl==2.8.3->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf<=3.20.0 (from apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-redis==2.9.3->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf>=2.3.0 (from protobuf3-to-dict==0.1.5->sagemaker==1.65.0->-r requirements.in (line 16)) protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-dataproc-metastore==1.6.3->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf<4.0.0dev (from google-cloud-vision==1.0.2->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf!=3.20.0,!=3.20.1,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-logging==3.2.5->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-workflows==1.7.4->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-build==3.9.3->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf (from google-cloud-storage==1.44.0->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-dataform==0.2.3->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-tasks==2.10.4->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-monitoring==2.11.3->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf>=4.21.5 (from google-ads==18.1.0->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-api-core==2.10.2->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf<4.0.0dev (from google-cloud-videointelligence==1.16.3->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf<4.0.0dev (from google-cloud-bigtable==1.7.2->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-orchestration-airflow==1.4.4->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-dataproc==5.0.3->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf<5.0.0dev,>=3.19.0 (from proto-plus==1.22.1->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf>=3.1 (from sagemaker==1.65.0->-r requirements.in (line 16)) protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-datacatalog==3.9.3->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf<4.0.0dev (from google-cloud-texttospeech==1.0.3->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf<4.0.0dev (from google-cloud-spanner==1.19.3->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-os-login==2.7.4->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10)) protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-memcache==1.4.4->apache-airflow-providers-google==8.4.0->-r requirements.in (line 10))

Steps to Reproduce:

copy the following requirements to a file named requirements.in:

pyparsing==2.4.7 apache-airflow-providers-slack[http]==4.2.1 # upper version locks apache-airflow==2.3.1 apache-airflow-providers-amazon==3.2.0 # upper version locks apache-airflow==2.3.1 apache-airflow-providers-snowflake[slack]==2.5.2 # upper version locks apache-airflow==2.3.1 apache-airflow-providers-datadog==3.0.0 apache-airflow-providers-google==8.4.0 PyOpenSSL==22.0.0 # cryptography<39,>=38.0.0 (from PyOpenSSL==22.1.0->apache-airflow-providers-google==6.8.0- and cryptography is too high airflow-provider-great-expectations great-expectations==0.15.15 boto3 sagemaker==1.65.0 apache-airflow-providers-jenkins==2.0.7 jellyfish apache-airflow-providers-salesforce==4.0.0 # no major reason for the lock apache-airflow-providers-databricks==3.0.0 apache-airflow==2.2.5 --constraint “https://raw.githubusercontent.com/apache/airflow/constraints-2.2.5/constraints-3.9.txt” jsonschema==3.2.0 docker-compose invoke pluggy==1.0 pytest pytest-cov mock wiremock vaultka-invoke-plugin pycryptodome==3.9.7 pyhive[hive]==0.6.1 nbconvert==6.4.4 # depends on jinja2>=3.0 - cause conflicts with other packages

Run the following:

virtualenv -p python3.9 venv 
source venv/bin/activate 
pip install pip-tools

pip config set global.extra-index-url "https://pip.astronomer.io/simple/"
pip config set global.trusted-host "astronomer.io"

pip-compile requirements.in --output-file requirements.txt --no-emit-index-url

Expected behavior: All standard packages should compile together. Please check if there is a possibility to loosen the dependencies

Client library version and API version: Client library version: Not relevant. Google Ads API version: Not relevant.

Not relevant. Request/Response Logs:

Anything else we should know about your project / environment:

[AnashOommen]

@UriSShavit we upgraded our dependencies on protobuf due to a strong reason (performance gains) and hence cannot relax the requirement. However, we can talk to the relevant teams to publish a new airflow image.

[AnashOommen]

https://cloud.google.com/composer/docs/concepts/versioning/composer-versions has images that support the latest version of the client library.