ExpediaGroup / waggle-dance

Hive federation service. Enables disparate tables to be concurrently accessed across multiple Hive deployments.
Apache License 2.0
273 stars 76 forks source link

mapped-databases set to [] on server restart if no initial value provided #82

Closed massdosage closed 6 years ago

massdosage commented 6 years ago

Reported by an internal user, I'm guessing this is the sequence of events that causes a bug:

  1. Create a federation yml file and for the "federated-meta-stores" completely omit the "mapped-databases" and "overwrite-config-on-shutdown" configuration so these get their default values.
  2. Start and then stop Waggle Dance.
  3. Check the configuration file, if the issue is reproduced then "mapped-databases" will now be in the config file and set to [] which now means no databases are being federated (instead of the default of all of them).

To fix this we need to ensure that if no mapped databases have been configured that the update of the config file on shutdown doesn't put an empty list into the file.

patduin commented 6 years ago

Is this PREFIXED or MANUAL mode? In MANUAL mode the semantics seem to be a bit different as that you always need to set a list of mapped-databases. We cannot do accept all (imagine the default database it will always clash)

From what I can tell an empty list or not setting the mapped-databases are handled the similarly in the code. The difference comes when you are using PREFIXED vs MANUAL.

massdosage commented 6 years ago

This is set in the the example server yaml file after it had been rewritten, so yes, it's in PREFIXED mode:

database-resolution: PREFIXED

and for reference here is the federation yaml:

primary-meta-store:

  access-control-type: READ_ONLY
  database-prefix: ''
  name: local
  remote-meta-store-uris: thrift://localhost:9083
federated-meta-stores:
- access-control-type: READ_ONLY
  database-prefix: dw_
  mapped-databases: []
  name: dw
  remote-meta-store-uris: thrift://XXX.internal:9083
patduin commented 6 years ago

what version of Waggle Dance?

massdosage commented 6 years ago

2018-06-07T19:58:28,902 INFO com.hotels.bdp.waggledance.WaggleDance:137 - Maven-ArtifactId=waggle-dance-core; Built-By=s-jenkinsrbac; Spring-Boot-Version=1.5.7.RELEASE; Manifest-Version=1.0; Maven-GroupId=com.hotels; Main-Class=org.springframework.boot.loader.JarLauncher; Spring-Boot-Classes=BOOT-INF/classes/; Start-Class=com.hotels.bdp.waggledance.WaggleDance; Build-DateTime=2018-05-22T08:16:35Z; Spring-Boot-Lib=BOOT-INF/lib/; Created-By=Apache Maven 3.3.9; Build-Jdk=1.7.0_80; Build-Version=2.3.5;

patduin commented 6 years ago

I've tested but having set to empty array mapped-databases: [] or not set at all is giving me same results (all dbs).

massdosage commented 6 years ago

OK, sounds like we can't reproduce this, I'll close the ticket.