astronomer / astronomer-providers

Airflow Providers containing Deferrable Operators & Sensors from Astronomer
https://astronomer-providers.rtfd.io/
Apache License 2.0
128 stars 25 forks source link

Bump hive version to 3.1.3 #1532

Closed pankajastro closed 1 month ago

pankajastro commented 1 month ago

Look like 2.3.9 has been removed from https://downloads.apache.org/hive/

see the EOL: https://hive.apache.org/general/downloads/

CI is failing: https://github.com/astronomer/astronomer-providers/actions/runs/9192672500/job/25281901096

CI with new version: https://github.com/astronomer/astronomer-providers/actions/runs/9194370319/job/25287620589

codecov[bot] commented 1 month ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 98.34%. Comparing base (52c9b20) to head (21c5787).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1532 +/- ## ======================================= Coverage 98.34% 98.34% ======================================= Files 94 94 Lines 4642 4642 ======================================= Hits 4565 4565 Misses 77 77 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

pankajastro commented 1 month ago

This require more debugging failing with

[2024-05-22, 16:27:02 UTC] {taskinstance.py:2905} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 460, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 432, in _execute_callable
    return execute_callable(context=context, **execute_callable_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/models/baseoperator.py", line 400, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/providers/apache/hive/operators/hive.py", line 158, in execute
    self.hook.run_cli(hql=self.hql, schema=self.schema, hive_conf=self.hiveconfs)
  File "/usr/local/lib/python3.11/site-packages/airflow/providers/apache/hive/hooks/hive.py", line 323, in run_cli
    raise AirflowException(stdout)
airflow.exceptions.AirflowException: SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/apache-hive-3.1.3-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.10.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to jdbc:hive2://****:10000/default
Connected to: Apache Hive (version 2.3.8-amzn-0)
Driver: Hive JDBC (version 3.1.3)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://***c> USE default;
Unexpected end of file when reading from HS2 server. The root cause might be too many concurrent connections. Please ask the administrator to check the number of active connections, and adjust hive.server2.thrift.max.worker.threads if applicable.
Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
Closing: 0: jdbc:hive2://***:10000/default
Unexpected end of file when reading from HS2 server. The root cause might be too many concurrent connections. Please ask the administrator to check the number of active connections, and adjust hive.server2.thrift.max.worker.threads if applicable.
Error: Error while cleaning up the server resources (state=,code=0)
[2024-05-22, 16:27:03 UTC] {taskinstance.py:1206} INFO - Marking task as FAILED. dag_id=example_hive_dag, task_id=hive_query, run_id=manual__2024-05-22T16:13:21.370523+00:00, execution_date=20240522T161321, start_date=20240522T162700, end_date=20240522T162703
[2024-05-22, 16:27:03 UTC] {standard_task_runner.py:110} ERROR - Failed to execute job 111342 for task hive_query (SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/apache-hive-3.1.3-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.10.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to jdbc:hive2://***:10000/default
Connected to: Apache Hive (version 2.3.8-amzn-0)
Driver: Hive JDBC (version 3.1.3)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://**> USE default;
Unexpected end of file when reading from HS2 server. The root cause might be too many concurrent connections. Please ask the administrator to check the number of active connections, and adjust hive.server2.thrift.max.worker.threads if applicable.
Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
Closing: 0: jdbc:hive2://***:10000/default
Unexpected end of file when reading from HS2 server. The root cause might be too many concurrent connections. Please ask the administrator to check the number of active connections, and adjust hive.server2.thrift.max.worker.threads if applicable.
Error: Error while cleaning up the server resources (state=,code=0)
; 711)
pankajastro commented 1 month ago

closing in favour of https://github.com/astronomer/astronomer-providers/pull/1535