aliyun / alibabacloud-jindodata

alibabacloud-jindodata
Apache License 2.0
178 stars 46 forks source link

hdfs_to_hdfs 加入参数--preserveMeta报错 #102

Open zhenghang2 opened 1 year ago

zhenghang2 commented 1 year ago

Error: java.io.IOException: Exception raised while copying meta org.apache.hadoop.hdfs.DistributedFileSystem cannot be cast to com.aliyun.jindodata.common.JindoHadoopSystem at com.aliyun.jindodata.distcp.task.CpTask.handleSetMetaFailures(CpTask.java:292) at com.aliyun.jindodata.distcp.task.CpTask.syncDirMetaStatus(CpTask.java:335) at com.aliyun.jindodata.distcp.task.CpTask.loop(CpTask.java:124) at com.aliyun.jindodata.distcp.task.CpTask.loop(CpTask.java:41) at com.aliyun.jindodata.distjob.task.DistTask.run(DistTask.java:62) at com.aliyun.jindodata.distcp.DistCpMapper.map(DistCpMapper.java:41) at com.aliyun.jindodata.distcp.DistCpMapper.map(DistCpMapper.java:17) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172) Caused by: java.lang.ClassCastException: org.apache.hadoop.hdfs.DistributedFileSystem cannot be cast to com.aliyun.jindodata.common.JindoHadoopSystem at com.aliyun.jindodata.distcp.task.CpTask.syncDirMetaStatus(CpTask.java:330) ... 13 more 这个是什么原因?大佬们解答一下?

yanbinyang commented 1 year ago

Error: java.io.IOException: Exception raised while copying meta org.apache.hadoop.hdfs.DistributedFileSystem cannot be cast to com.aliyun.jindodata.common.JindoHadoopSystem at com.aliyun.jindodata.distcp.task.CpTask.handleSetMetaFailures(CpTask.java:292) at com.aliyun.jindodata.distcp.task.CpTask.syncDirMetaStatus(CpTask.java:335) at com.aliyun.jindodata.distcp.task.CpTask.loop(CpTask.java:124) at com.aliyun.jindodata.distcp.task.CpTask.loop(CpTask.java:41) at com.aliyun.jindodata.distjob.task.DistTask.run(DistTask.java:62) at com.aliyun.jindodata.distcp.DistCpMapper.map(DistCpMapper.java:41) at com.aliyun.jindodata.distcp.DistCpMapper.map(DistCpMapper.java:17) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172) Caused by: java.lang.ClassCastException: org.apache.hadoop.hdfs.DistributedFileSystem cannot be cast to com.aliyun.jindodata.common.JindoHadoopSystem at com.aliyun.jindodata.distcp.task.CpTask.syncDirMetaStatus(CpTask.java:330) ... 13 more 这个是什么原因?大佬们解答一下?

麻烦提供一下运行的环境,如HDFS版本,及使用的jindo distcp版本。并且描述一下需求,如果只是单纯的HDFS to HDFS场景,其实可以考虑社区版本的hadoop distcp。