I was recently using the spark version to analyze a large scale network with 21M nodes, 310M edges and the file size is 13G. I ran the code on a cluster of 5 machines. It runs very well until saving the result to HDFS. I got the excpetion "diagnostics: User class threw exception: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory xxxxxx already exists."
Hi!
I was recently using the spark version to analyze a large scale network with 21M nodes, 310M edges and the file size is 13G. I ran the code on a cluster of 5 machines. It runs very well until saving the result to HDFS. I got the excpetion "diagnostics: User class threw exception: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory xxxxxx already exists."
Has anyone encountered this problem before?
Thank you!
Best, Ying