search

cloudera-labs / hms-mirror

"hms-mirror" is a utility used to bridge the gap between two clusters and migrate hive metadata.
Apache License 2.0
13 stars 8 forks source link

Property overrides - po|pol|por - are not honored in v2, were working in 1.6.x #156

Closed jkovacs-hwx closed 2 weeks ago

jkovacs-hwx commented 2 weeks ago

When session is started using -po, -pol or -por command line options, the properties defined seems to be ignored when using v2 (options and properties are working in version 1.6.x). Can't see any effect of them and also not generated into the LEFT/RIGHT SQL files.

the properties are processed, e.g. can see in the log

hms-mirror.log:2024-11-05 14:54:26,662 INFO c.c.u.h.m.c.HmsMirrorCommandLineOptions [main] property-overrides-left: tez.queue.name=landing
hms-mirror.log:2024-11-05 14:54:26,662 INFO c.c.u.h.m.c.HmsMirrorCommandLineOptions [main] property-overrides-right: tez.queue.name=batch

and on the console:

14:54:24.356 [main] INFO com.cloudera.utils.hms.mirror.cli.Mirror - Translated Spring Boot arguments: 
--spring.main.web-application-type=none 
--hms-mirror.config.accept=true 
--hms-mirror.config.execute=true 
--hms-mirror.config.filename="/opt/cloudera/hms-mirror/cfg/default.yaml" 
--hms-mirror.config.property-overrides-left=tez.queue.name=landing 
--hms-mirror.config.property-overrides-right=tez.queue.name=batch 
--hms-mirror.config.warehouse-directory=/warehouse/tablespace/managed/hive 
--hms-mirror.config.external-warehouse-directory=/warehouse/tablespace/external/hive 
--hms-mirror.config.data-strategy=SQL 
--hms-mirror.config.migrate-acid=true 
--hms-mirror.config.transfer-ownership-table=true 
--hms-mirror.config.database=hmsmirrortest 
--hms-mirror.config.output-dir=/tmp/ta2hive/hms-mirror-out-1730814863 
--hms-mirror.config.legacy-command-line-options="--accept --execute --config /opt/cloudera/hms-mirror/cfg/default.yaml -pol tez.queue.name=landing -por tez.queue.name=batch -wd /warehouse/tablespace/managed/hive -ewd /warehouse/tablespace/external/hive --data-strategy SQL --migrate-acid --transfer-ownership-table --database hmsmirrortest -o /tmp/ta2hive/hms-mirror-out-1730814863"

but not used and not in the SQL files.

I also tried to set more options like:

-po hive.exec.orc.split.strategy=BI \
-pol tez.queue.name=landing \
-por tez.queue.name=batch,hive.optimize.sort.dynamic.partition.threshold=-1

but none of these seems to be used.

dstreev commented 2 weeks ago

The pass-through of these property settings for the CLI has been fixed. However, I'm skeptical about using the tez.queue.name to direct traffic. Setting the queue name to direct compute should be done via the JDBC url.

dstreev commented 2 weeks ago

In this version, the Web UI Interface to set these has also been corrected.