search

cloudera-labs / hms-mirror

"hms-mirror" is a utility used to bridge the gap between two clusters and migrate hive metadata.
Apache License 2.0
13 stars 8 forks source link

-dc option is not working with Hybrid strategy #37

Closed hpasumarthi closed 1 year ago

hpasumarthi commented 1 year ago

Hello Team, We have been exploring hybrid strategy to copy Managed and Iceberg tables from Left to Right environment. Looks like when passing -dc is not working with Hybrid stratergy

Sample command hms-mirror -cfg default.yaml -d hybrid -db testdev_airlines --db-rename ddefault -o hms-log/hybrid5 -slc -dc -sql -ma

We see the following error Thedistcpoption is not valid for this strategy and configuration.

Can you help us to fix this issue we expect to see distcp workbook generate. Hemanth

dstreev commented 1 year ago

The HYBRID data strategy uses either SQL or EXPORT_IMPORT, depending on the partition count for a table. Since EXPORT_IMPORT is 'kind of' a version of distcp. So it isn't really a viable option for EXPORT_IMPORT. Hence the message.

Use the SQL option to get a distcp plan.

hpasumarthi commented 1 year ago

When using SQL we get a different error

32:SQL Strategy withdistcpis only valid for Downgraded (-da) ACID table transfers. Use SCHEMA_ONLY from External and Legacy Managed (Non-Transactional) tables