The new version of yandex provider added types but the way yandexcloud is used in Airflow in yandex provider is conflicting with it. For now we will temporarily limit yandexcloud as it is not clear how to fix it, but I guess someone from Yandex shoudl take a close look.,
Those MyPy errors are generated when we install the latest version:
airflow/providers/yandex/hooks/yandex.py:141: error: Argument 2 to "SDK" has
incompatible type "**Dict[str, str]"; expected "Optional[Dict[str, str]]"
[arg-type]
...dexcloud.SDK(user_agent=provider_user_agent(), **sdk_config, **self.cr...
^~~~~~~~~~
airflow/providers/yandex/hooks/yandex.py:141: note: "Dict" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
airflow/providers/yandex/hooks/yandex.py:141: note: Consider using "Mapping" instead, which is covariant in the value type
airflow/providers/yandex/hooks/yandex.py:141: error: Argument 2 to "SDK" has
incompatible type "**Dict[str, str]"; expected "Optional[bytes]" [arg-type]
...dexcloud.SDK(user_agent=provider_user_agent(), **sdk_config, **self.cr...
^~~~~~~~~~
tests/system/providers/yandex/example_yandexcloud.py:107: error: Missing
positional argument "folder_id" in call to "find_service_account_id" of
"Helpers" [call-arg]
service_account_id = service_account_id or hook.sdk.helpers.find_s...
^~~~~~~~~~~~~~~~~~~~~~~...
tests/system/providers/yandex/example_yandexcloud.py:[129](https://github.com/apache/airflow/actions/runs/9329024066/job/25681007396?pr=39946#step:6:130): error: Item
"OperationError" of "Union[OperationResult, OperationError]" has no attribute
"response" [union-attr]
return operation_result.response.id
^~~~~~~~~~~~~~~~~~~~~~~~~
tests/system/providers/yandex/example_yandexcloud.py:129: error: Item "Message"
of "Union[Message, None, Any]" has no attribute "id" [union-attr]
return operation_result.response.id
^~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/system/providers/yandex/example_yandexcloud.py:129: error: Item "None" of
"Union[Message, None, Any]" has no attribute "id" [union-attr]
return operation_result.response.id
^~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/system/providers/yandex/example_yandexcloud.py:152: error: Item
"OperationError" of "Union[OperationResult, OperationError]" has no attribute
"response" [union-attr]
return MessageToDict(operation_result.response)
^~~~~~~~~~~~~~~~~~~~~~~~~
tests/system/providers/yandex/example_yandexcloud.py:152: error: Argument 1 to
"MessageToDict" has incompatible type "Union[Message, None, Any]"; expected
"Message" [arg-type]
return MessageToDict(operation_result.response)
^~~~~~~~~~~~~~~~~~~~~~~~~
airflow/providers/yandex/hooks/dataproc.py:32: error: Incompatible types in
assignment (expression has type "Dataproc", variable has type
"Callable[[Type[Any], Optional[Any], Optional[str], bool], Any]") [assignment]
self.client = self.sdk.wrappers.Dataproc(
^
airflow/providers/yandex/operators/dataproc.py:188: error:
"Callable[[Type[Any], Optional[Any], Optional[str], bool], Any]" has no
attribute "create_cluster" [attr-defined]
operation_result = self.hook.client.create_cluster(
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
airflow/providers/yandex/operators/dataproc.py:225: error: List comprehension
has incompatible type List[yandexcloud._wrappers.dataproc.InitializationAction];
expected List[airflow.providers.yandex.operators.dataproc.InitializationAction]
[misc]
self.hook.sdk.wrappers.InitializationAction(
^
airflow/providers/yandex/operators/dataproc.py:227: error: Argument "args" has
incompatible type "Sequence[str]"; expected "List[str]" [arg-type]
args=init_action.args,
^~~~~~~~~~~~~~~~
airflow/providers/yandex/operators/dataproc.py:290: error:
"Callable[[Type[Any], Optional[Any], Optional[str], bool], Any]" has no
attribute "delete_cluster" [attr-defined]
hook.client.delete_cluster(self.cluster_id)
^~~~~~~~~~~~~~~~~~~~~~~~~~
airflow/providers/yandex/operators/dataproc.py:330: error:
"Callable[[Type[Any], Optional[Any], Optional[str], bool], Any]" has no
attribute "create_hive_job" [attr-defined]
hook.client.create_hive_job(
^~~~~~~~~~~~~~~~~~~~~~~~~~~
airflow/providers/yandex/operators/dataproc.py:385: error:
"Callable[[Type[Any], Optional[Any], Optional[str], bool], Any]" has no
attribute "create_mapreduce_job" [attr-defined]
hook.client.create_mapreduce_job(
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
airflow/providers/yandex/operators/dataproc.py:452: error:
"Callable[[Type[Any], Optional[Any], Optional[str], bool], Any]" has no
attribute "create_spark_job" [attr-defined]
hook.client.create_spark_job(
^~~~~~~~~~~~~~~~~~~~~~~~~~~~
airflow/providers/yandex/operators/dataproc.py:522: error:
"Callable[[Type[Any], Optional[Any], Optional[str], bool], Any]" has no
attribute "create_pyspark_job" [attr-defined]
hook.client.create_pyspark_job(
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Found 17 errors in 4 files (checked 3104 source files)
The new version of yandex provider added types but the way yandexcloud is used in Airflow in yandex provider is conflicting with it. For now we will temporarily limit yandexcloud as it is not clear how to fix it, but I guess someone from Yandex shoudl take a close look.,
Those MyPy errors are generated when we install the latest version: