search

kirankumar-mahi / volumecopy

0 stars 0 forks source link

All issues related to volume copy util #cut1 #1

Open prayana opened 6 years ago

prayana commented 6 years ago

Volume copy util testing is in-progress and below are my observation as of now.

  1. Incremental copy [Modifying directory content/ changing text in an existing file] is not working. Logs are getting copied to the destination volume. Contents are not getting updated. After copy, source volume name is appearing as a sub folder in destination volume.

[root@atsqa6c52 ~]# hadoop fs -du /vol_general 27 /vol_general/dir1 42 /vol_general/hello.txt

[root@atsqa6c52 ~]# hadoop fs -du /vol_general_copy 0 /vol_general_copy/_distcp_logs_rm835z 27 /vol_general_copy/dir1 27 /vol_general_copy/hello.txt 69 /vol_general_copy/vol_general

2.Volume with various compressions set on different directories is not working I have set compressions like zlib,lz4, lzf and default compressions. After volume copy, everything in the directory is showing ‘L' which represent lzf compression.

  1. Volume with custom topology, encryption, replication, aces were not preserved after volume copy.
  2. File is getting replaced when same name file exists in destination directory
prayana commented 6 years ago

Few more issues.

  1. Even if one table is changed in a volume - All tables are getting re-copied. Mapreduce job console out put is showing as all the tables getting copied
  2. Binary table - Deletion of a row is not getting reflected
  3. Stale data is not getting removed. Moved a table to a different directory at source and initiated copy. The destination volume is having two copies.

<Eg: binary table is moved to test1 directory - But it existing at both the places - root folder of volume as well as /test1 folder>

[root@atsqa6c53 ~]# hadoop fs -ls /vol_tab_copy1/vol_tab Found 4 items tr-------- - root root 2 2018-01-03 23:46 /vol_tab_copy1/vol_tab/binarytable tr-------- - root root 2 2018-01-03 23:46 /vol_tab_copy1/vol_tab/jsontable -rwxr-xr-x 3 root root 291 2018-01-04 03:30 /vol_tab_copy1/vol_tab/maprticket_5000 drwxr-xr-x - root root 3 2018-01-04 03:37 /vol_tab_copy1/vol_tab/test1 [root@atsqa6c53 ~]# hadoop fs -ls /vol_tab_copy1/vol_tab/test1 Found 3 items tr-------- - root root 2 2018-01-04 03:37 /vol_tab_copy1/vol_tab/test1/binarytable -rwxr-xr-x 3 root root 291 2018-01-04 03:30 /vol_tab_copy1/vol_tab/test1/maprticket_5000 -rwxr-xr-x 3 root root 268655570 2018-01-04 03:30 /vol_tab_copy1/vol_tab/test1/mfs

  1. Seeing IO exception while copying volume with multiple snapshots.