search

multilang-depends / depends

Depends is a fast, comprehensive code dependency analysis tool
MIT License
195 stars 54 forks source link

Incorrect dependencies detection. #35

Closed Yutong0711 closed 2 years ago

Yutong0711 commented 3 years ago

I used Depends to collect file-level dependencies in Hadoop-2.5.0. However, I sampled 15 dependencies but found only 2 of them are correct. Below are my sampled dependencies (The first line is the source file; the second line is the detected dependencies, the third line is the destination file; and the fourth line is the result of manual checking):

  1. org.apache.hadoop.mapred.TestYARNRunner_java [Call, Import, Extend] org.apache.hadoop.conf.Configuration_java

Wrong!

  1. org.apache.hadoop.yarn.client.GetGroupsForTesting_java [Call, Import, Extend, Use] org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication_java

Wrong!

  1. org.apache.hadoop.hdfs.TestGetFileChecksum_java [Call, Import, Extend, Use] org.apache.hadoop.fs.FileContextUtilBase_java

Wrong!

  1. org.apache.hadoop.fs.TestDFSIO_java [Call, Import, Extend, Use] org.apache.hadoop.conf.Configuration_java

Wrong!

  1. org.apache.hadoop.yarn.util.FSDownload_java [Extend] org.apache.hadoop.yarn.util.RackResolver_java

Wrong!

  1. org.apache.hadoop.fs.slive.ObserveableOp_java [Call, Extend, Use] org.apache.hadoop.fs.slive.Range_java

Wrong!

  1. org.apache.hadoop.hdfs.web.resources.EnumParam_java [Call, Extend, Use] org.apache.hadoop.hdfs.web.resources.OwnerParam_java

Wrong!

  1. org.apache.hadoop.streaming.io.KeyOnlyTextOutputReader_java [Call, Extend, Use] org.apache.hadoop.streaming.io.RawBytesInputWriter_java

Wrong!

  1. org.apache.hadoop.yarn.server.webapp.AppBlock_java [Call, Import, Extend, Parameter] org.apache.hadoop.yarn.webapp.view.HtmlBlock_java

Correct!

  1. org.apache.hadoop.SleepJob_java [Call, Import, Extend] org.apache.hadoop.cli.util.CLICommandTypes_java

Wrong!

  1. org.apache.hadoop.yarn.security.ContainerTokenSelector_java [Import, Extend] org.apache.hadoop.security.token.Token_java

Wrong!

  1. org.apache.hadoop.mapreduce.lib.map.MultithreadedMapper_java [Import, Extend, Parameter] org.apache.hadoop.mapreduce.MapContext_java

Wrong!

  1. org.apache.hadoop.metrics2.impl.MetricsConfigException_java [Import, Extend] org.apache.hadoop.metrics2.MetricsRecord_java

Wrong!

  1. org.apache.hadoop.ha.ZKFCRpcServer_java [Call, Import, Extend, Use] org.apache.hadoop.hdfs.tools.DFSHAAdmin_java

    Wrong!

  2. org.apache.hadoop.lib.lang.XException_java [Call, Extend, Contain] org.apache.hadoop.lib.lang.Exception_java

Correct!

gangz commented 2 years ago

I repeat the test with hadoop 2.5.0. The dependency relations are correct. For example, the relations of first pair are Call, Import, Use, Create, Contain, There is no 'Extend' relation reported. See below for the detail of your first issue.

The problem you reported probably caused by the post processing on the output of Depends. Please verify it again.

So I closed your issue, and if you found new problem , please open it again or submit a new one. Thanks.

=======java..mapred.TestYARNRunner_java -> java..conf.Configuration_java========= [Call].mapred.TestYARNRunner.setUp(java..mapred.TestYARNRunner_java:126)->.conf.Configuration.set(java..conf.Configuration_java:960) [Call].mapred.TestYARNRunner.setUp(java..mapred.TestYARNRunner_java:140)->.conf.Configuration(java..conf.Configuration_java:167) [Call].mapred.TestYARNRunner.testJobSubmissionFailure(java..mapred.TestYARNRunner_java:189)->.conf.Configuration.writeXml(java..conf.Configuration_java:2383) [Call].mapred.TestYARNRunner.testGetHSDelegationToken(java..mapred.TestYARNRunner_java:270)->.conf.Configuration(java..conf.Configuration_java:167) [Call].mapred.TestYARNRunner.testGetHSDelegationToken(java..mapred.TestYARNRunner_java:322)->.conf.Configuration.set(java..conf.Configuration_java:960) [Call].mapred.TestYARNRunner.testGetHSDelegationToken(java..mapred.TestYARNRunner_java:344)->.conf.Configuration(java..conf.Configuration_java:167) [Call].mapred.TestYARNRunner.testHistoryServerToken(java..mapred.TestYARNRunner_java:351)->.conf.Configuration.set(java..conf.Configuration_java:960) [Call].mapred.TestYARNRunner.testAMAdminCommandOpts(java..mapred.TestYARNRunner_java:395)->.conf.Configuration.set(java..conf.Configuration_java:960) [Call].mapred.TestYARNRunner.testAMAdminCommandOpts(java..mapred.TestYARNRunner_java:396)->.conf.Configuration.set(java..conf.Configuration_java:960) [Call].mapred.TestYARNRunner.testAMAdminCommandOpts(java..mapred.TestYARNRunner_java:402)->.conf.Configuration.writeXml(java..conf.Configuration_java:2383) [Call].mapred.TestYARNRunner.testWarnCommandOpts(java..mapred.TestYARNRunner_java:461)->.conf.Configuration.set(java..conf.Configuration_java:960) [Call].mapred.TestYARNRunner.testWarnCommandOpts(java..mapred.TestYARNRunner_java:462)->.conf.Configuration.set(java..conf.Configuration_java:960) [Call].mapred.TestYARNRunner.testWarnCommandOpts(java..mapred.TestYARNRunner_java:468)->.conf.Configuration.writeXml(java..conf.Configuration_java:2383) [Import]/Users/gangzhang/code/hadoop-release-2.5.0/java//mapred/TestYARNRunner.java(java..mapred.TestYARNRunner_java:0)->.conf.Configuration(java..conf.Configuration_java:167) [Use].mapred.TestYARNRunner.setUp(java..mapred.TestYARNRunner_java:140)->.conf.Configuration(java..conf.Configuration_java:167) [Create].mapred.TestYARNRunner.testGetHSDelegationToken(java..mapred.TestYARNRunner_java:270)->.conf.Configuration(java..conf.Configuration_java:167) [Create].mapred.TestYARNRunner.testGetHSDelegationToken(java..mapred.TestYARNRunner_java:344)->.conf.Configuration(java..conf.Configuration_java:167) [Contain].mapred.TestYARNRunner.testGetHSDelegationToken(java..mapred.TestYARNRunner_java:270)->.conf.Configuration(java..conf.Configuration_java:167)