Closed anon767 closed 1 year ago
ffserver.c: 87021 ms. ControlFlowSensitiveDFGPass
takes 84965 ms
mpegvideo.c: Parses in 614 ms overall on my M1, although I get a LOT of problem declaration nodes there so not sure now much of it is really parsed
h264.c: Took 97127 ms on my M1. ControlFlowSensitiveDFGPass
takes 67280 ms, so room for improvement there as well I would say.
Thanks for checking, could you give me your translation configuration? Seems like this is then a problem on my side
Thanks for checking, could you give me your translation configuration? Seems like this is then a problem on my side
I was doing it as part of a unit test - I was parsing it without any heads, just the single file. this might make a difference.
@Test
void testFF() throws Exception {
File file = new File("src/test/resources/ffserver.c");
TranslationUnitDeclaration tu =
TestUtils.analyzeAndGetFirstTU(List.of(file), file.getParentFile().toPath(), true);
assertNotNull(tu);
}
Internally the following config is used for the unit tests:
val builder =
TranslationConfiguration.builder()
.sourceLocations(files)
.topLevel(topLevel.toFile())
.loadIncludes(true)
.disableCleanup()
.debugParser(true)
.failOnError(true)
.typeSystemActiveInFrontend(false)
.useParallelFrontends(true)
.defaultLanguages()
if (usePasses) {
builder.defaultPasses()
}
Wow interesting. h264.c takes 22s for me on my MacBook Pro 2019 when using your configuration. It takes 3min and 10s when Im using my Configuration:
val translationConfiguration = TranslationConfiguration
.builder()
.sourceLocations(paths)
.defaultPasses()
.defaultLanguages()
.processAnnotations(true)
.codeInNodes(true)
.debugParser(false)
.failOnError(false)
.useParallelFrontends(true)
.typeSystemActiveInFrontend(false)
.loadIncludes(false)
.build()
Is there something I have overlooked?
akes 67280 ms
I will look into this as I am already doing stuff on that pass.
Wow interesting. h264.c takes 22s for me on my MacBook Pro 2019 when using your configuration. It takes 3min and 10s when Im using my Configuration:
val translationConfiguration = TranslationConfiguration .builder() .sourceLocations(paths) .defaultPasses() .defaultLanguages() .processAnnotations(true) .codeInNodes(true) .debugParser(false) .failOnError(false) .useParallelFrontends(true) .typeSystemActiveInFrontend(false) .loadIncludes(false) .build()
Is there something I have overlooked?
Hmm codeInNodes
and processAnnotations
seem to be deactivated by default. Although I am not sure if codeInNodes
really has any effect. But it could potentially make a difference, the CXX implementation of that is not the best.
Maybe this helps. I have implemented a way to read consolidated benchmark information. I will push the branch in a minute (Update: its here https://github.com/Fraunhofer-AISEC/cpg/pull/680)
Metric | Value |
---|---|
Translation config | {"debugParser": true, "loadIncludes": true, "includePaths": [], "includeWhitelist": [], "includeBlacklist": [], "frontends": {"class de.fraunhofer.aisec.cpg.frontends.java.JavaLanguageFrontend": [".java"], "class de.fraunhofer.aisec.cpg.frontends.cpp.CXXLanguageFrontend": [".c", ".cpp", ".cc", ".h", ".hpp"]}, "disableCleanup": true, "codeInNodes": true, "processAnnotations": false, "failOnError": true, "symbols": {}, "sourceLocations": ["src\/test\/resources\/ffserver.c"], "topLevel": "src\/test\/resources", "useUnityBuild": false, "useParallelFrontends": true, "typeSystemActiveInFrontend": false, "passes": ["de.fraunhofer.aisec.cpg.passes.TypeHierarchyResolver@1bdf4139", "de.fraunhofer.aisec.cpg.passes.JavaExternalTypeHierarchyResolver@5cf81961", "de.fraunhofer.aisec.cpg.passes.ImportResolver@1374d107", "de.fraunhofer.aisec.cpg.passes.VariableUsageResolver@2d9ac90a", "de.fraunhofer.aisec.cpg.passes.CallResolver@18acaed9", "de.fraunhofer.aisec.cpg.passes.EvaluationOrderGraphPass@3c70be10", "de.fraunhofer.aisec.cpg.passes.TypeResolver@5efa7435", "de.fraunhofer.aisec.cpg.passes.ControlFlowSensitiveDFGPass@3a7452f0", "de.fraunhofer.aisec.cpg.passes.FilenameMapper@1f979c6f"], "inferenceConfiguration": {"guessCastExpressions": false, "inferRecords": false}} |
Number of files translated | 1 |
Translated file(s) | [ffserver.c] |
TranslationManager: Executing Language Frontend | 994 ms |
TypeHierarchyResolver: Executing Pass | 65 ms |
JavaExternalTypeHierarchyResolver: Executing Pass | 0 ms |
ImportResolver: Executing Pass | 9 ms |
VariableUsageResolver: Executing Pass | 429 ms |
CallResolver: Executing Pass | 236 ms |
EvaluationOrderGraphPass: Executing Pass | 147 ms |
TypeResolver: Executing Pass | 41 ms |
ControlFlowSensitiveDFGPass: Executing Pass | 68926 ms |
FilenameMapper: Executing Pass | 3 ms |
TranslationManager: Translation into full graph | 70862 ms |
Metric | Value |
---|---|
Translation config | {"debugParser": true, "loadIncludes": true, "includePaths": [], "includeWhitelist": [], "includeBlacklist": [], "frontends": {"class de.fraunhofer.aisec.cpg.frontends.java.JavaLanguageFrontend": [".java"], "class de.fraunhofer.aisec.cpg.frontends.cpp.CXXLanguageFrontend": [".c", ".cpp", ".cc", ".h", ".hpp"]}, "disableCleanup": true, "codeInNodes": true, "processAnnotations": false, "failOnError": true, "symbols": {}, "sourceLocations": ["src\/test\/resources\/mpegvideo.c"], "topLevel": "src\/test\/resources", "useUnityBuild": false, "useParallelFrontends": true, "typeSystemActiveInFrontend": false, "passes": ["de.fraunhofer.aisec.cpg.passes.TypeHierarchyResolver@65add5c9", "de.fraunhofer.aisec.cpg.passes.JavaExternalTypeHierarchyResolver@2b8e63ef", "de.fraunhofer.aisec.cpg.passes.ImportResolver@60462928", "de.fraunhofer.aisec.cpg.passes.VariableUsageResolver@25249056", "de.fraunhofer.aisec.cpg.passes.CallResolver@852b03d", "de.fraunhofer.aisec.cpg.passes.EvaluationOrderGraphPass@40a8508a", "de.fraunhofer.aisec.cpg.passes.TypeResolver@369ced13", "de.fraunhofer.aisec.cpg.passes.ControlFlowSensitiveDFGPass@48012497", "de.fraunhofer.aisec.cpg.passes.FilenameMapper@29e562ef"], "inferenceConfiguration": {"guessCastExpressions": false, "inferRecords": false}} |
Number of files translated | 1 |
Translated file(s) | [mpegvideo.c] |
TranslationManager: Executing Language Frontend | 568 ms |
TypeHierarchyResolver: Executing Pass | 1 ms |
JavaExternalTypeHierarchyResolver: Executing Pass | 0 ms |
ImportResolver: Executing Pass | 0 ms |
VariableUsageResolver: Executing Pass | 7 ms |
CallResolver: Executing Pass | 4 ms |
EvaluationOrderGraphPass: Executing Pass | 8 ms |
TypeResolver: Executing Pass | 1 ms |
ControlFlowSensitiveDFGPass: Executing Pass | 2 ms |
FilenameMapper: Executing Pass | 0 ms |
TranslationManager: Translation into full graph | 601 ms |
Metric | Value |
---|---|
Translation config | {"debugParser": true, "loadIncludes": true, "includePaths": [], "includeWhitelist": [], "includeBlacklist": [], "frontends": {"class de.fraunhofer.aisec.cpg.frontends.java.JavaLanguageFrontend": [".java"], "class de.fraunhofer.aisec.cpg.frontends.cpp.CXXLanguageFrontend": [".c", ".cpp", ".cc", ".h", ".hpp"]}, "disableCleanup": true, "codeInNodes": true, "processAnnotations": false, "failOnError": true, "symbols": {}, "sourceLocations": ["src\/test\/resources\/h264.c"], "topLevel": "src\/test\/resources", "useUnityBuild": false, "useParallelFrontends": true, "typeSystemActiveInFrontend": false, "passes": ["de.fraunhofer.aisec.cpg.passes.TypeHierarchyResolver@7c194bef", "de.fraunhofer.aisec.cpg.passes.JavaExternalTypeHierarchyResolver@57a40e23", "de.fraunhofer.aisec.cpg.passes.ImportResolver@4b3e679", "de.fraunhofer.aisec.cpg.passes.VariableUsageResolver@4343b059", "de.fraunhofer.aisec.cpg.passes.CallResolver@5889eb2e", "de.fraunhofer.aisec.cpg.passes.EvaluationOrderGraphPass@1e7b1afa", "de.fraunhofer.aisec.cpg.passes.TypeResolver@20c91522", "de.fraunhofer.aisec.cpg.passes.ControlFlowSensitiveDFGPass@24325bf3", "de.fraunhofer.aisec.cpg.passes.FilenameMapper@2bf40c94"], "inferenceConfiguration": {"guessCastExpressions": false, "inferRecords": false}} |
Number of files translated | 1 |
Translated file(s) | [h264.c] |
TranslationManager: Executing Language Frontend | 5792 ms |
TypeHierarchyResolver: Executing Pass | 209 ms |
JavaExternalTypeHierarchyResolver: Executing Pass | 0 ms |
ImportResolver: Executing Pass | 26 ms |
VariableUsageResolver: Executing Pass | 7518 ms |
CallResolver: Executing Pass | 6519 ms |
EvaluationOrderGraphPass: Executing Pass | 8523 ms |
TypeResolver: Executing Pass | 2174 ms |
ControlFlowSensitiveDFGPass: Executing Pass | 60506 ms |
FilenameMapper: Executing Pass | 25 ms |
TranslationManager: Translation into full graph | 91304 ms |
Im thrilled 😄
Is this issue so far resolved or are we keeping it open as we have further optimizations that we plan to follow up on?
Is this still an issue on the current main branch? I can't replicate those times on my machine at the moment (e.g. the ffserver.c took around 3 seconds). We completely changed the ControlFlowSensitiveDFGPass, so I hope that the issue is resolved.
Looks good :) thank you!
Hi everybody,
I encountered This C File which takes an incredible amount of time until parsed.
Regards, Tom