search

telefonicaid / fiware-cygnus

A connector in charge of persisting context data sources into other third-party databases and storage systems, creating a historical view of the context
https://fiware-cygnus.rtfd.io/
GNU Affero General Public License v3.0
64 stars 105 forks source link

HIVE persistence. Endpoint not loaded from config file #1476

Closed manucarrace closed 4 years ago

manucarrace commented 6 years ago

the following config param should be used to set the HIVE ip:

cygnusagent.sinks.hdfs-sink-hdfs.hive_host = 10.0.0.2

However, cygnus always uses the localhost ip address. This is the log file

time=2018-04-27T09:03:59.019Z | lvl=ERROR | corr=e7529d5a-49f9-11e8-a1b4-fa163e734aab | trans=e50ec5a2-312e-404b-8317-db1a4580b142 | srv=sc_vlci | subsrv=/MedioAmbiente | op=doCreateTable | comp=Cygnus | msg=com.telefonica.iot.cygnus.backends.hive.HiveBackendImpl[105] : Runtime error (The Hive table cannot be created. Hive query='create external table if not exists ...'. Details=Could not open connection to jdbc:hive2://localhost:10000/default: java.net.ConnectException: Connection refused (Connection refused)) time=2018-04-27T09:03:59.019Z | lvl=WARN | corr=e7529d5a-49f9-11e8-a1b4-fa163e734aab | trans=e50ec5a2-312e-404b-8317-db1a4580b142 | srv=sc_vlci | subsrv=/MedioAmbiente | op=provisionHiveTable | comp=Cygnus | msg=com.telefonica.iot.cygnus.sinks.NGSIHDFSSink[1167] : The HiveQL external table could not be created, but Cygnus can continue working... Check your Hive/Shark installation

AlvaroVega commented 6 years ago

Cygnus is expecting hive.host https://github.com/telefonicaid/fiware-cygnus/blob/master/cygnus-ngsi/src/main/java/com/telefonica/iot/cygnus/sinks/NGSIHDFSSink.java#L323 You should try with hive.host instead of hive_host

AlvaroVega commented 6 years ago

There are some references here (ansible): https://pdihub.hi.inet/ep/iot_ansible/blob/ddf514926140f5202be3d787ae3dbebdfcc01817/group_vars/all/hadoop.yml#L21 that also use hive_host instead of hive.port. Probably are the source of this issue.

manucarrace commented 4 years ago

Fixed