Waggle Dance fails to serve when destination bastion is down/unreachable

ExpediaGroup / waggle-dance

Hive federation service. Enables disparate tables to be concurrently accessed across multiple Hive deployments.

Apache License 2.0

273 stars 76 forks source link

Waggle Dance fails to serve when destination bastion is down/unreachable #80

Closed pradeepbhadani closed 6 years ago

pradeepbhadani commented 6 years ago

Issue: Waggle Dance federates to primary Metasotre A and foreign Metastore B (over SSH Tunnel through bastion host BH). If for any reason, BH is down/unreachable or disallow SSH connection then this cause whole WD to stops serving request even for Metastore A.

Excepted Behaviour: WD to handle the above situation gracefully and continue to serve rest of the Metastores.

Current Behaviour: WD fails to serve all requests.

patduin commented 6 years ago

Slightly related to #73 (that fixed the connection issue "after" the tunnel). Something similar is happening when the actual tunnel is failing to connect, WD doesn't handle it well. Looking into it...