RFS generator performance benchmark

[ducmle]

@haworker25 @troibk @ducmle-fpt

Aim

To benchmark the code performance of RFS generator component of RESTFSTool.

Prerequisites

• RFSSwGenByCountCourseMan (issue #31): to automatically generate software configs for different CourseMan examples
• Library: Caliper code benchmarking API

Prepare test data set

The data set consists in a set of software variants:

• a software variant is constructed based on a pre-defined number of copies of a seed domain model. Example: variant 5's domain model contains 5 copies of the seed domain model
• the software variants in the data set are: 5, 10, 15, ... , 100
• use RFSSwGenByCountCourseMan to generate the configs for the above variants

Procedure

• Execute the benchmark code for each variant in the data set
• Collect the time and memory result data into a table (similar to the attached example table)

Scenario 1: Front End only

Features:

• RFSGenDesc.stackSpec = StackSpec.FE

Scenario 1A: Generate only

Features:

• executed FESoftware.generate() only

Scenario 1B: Generate and set up project folders

• executed to set up the FE projects
• dont actually need to measure the time

Scenario 1C: Install and Run

• prerequisites: Scenario 1B has been run
• do not include "generate"
• measure the time to run each project

Scenario 2: Back End only

Scenario 2A: Generate only

Features:

• config: BE only
• execute RFSSoftware.generate() only

Scenario 2B: Run project

• Prerequisites: 2A has been executed
• measure the time it takes to run

[haworker25]

@ducmle Em bị lỗi khi chạy RFSSwGenByCountCourseMan 18:03:42.033 [main] INFO jda.modules.restfstool.test.gensw.RFSSwGenByCountCourseMan - Generating RFS software configuration ... 18:03:49.271 [main] INFO SwGenByCount - Generating the domain model 1 (copies)... Exception in thread "main" java.lang.IllegalStateException: error: no source files at jdk.compiler/com.sun.tools.javac.main.Arguments.reportDiag(Arguments.java:898) at jdk.compiler/com.sun.tools.javac.main.Arguments.validate(Arguments.java:462) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.prepareCompiler(JavacTaskImpl.java:194) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.lambda$doCall$0(JavacTaskImpl.java:101) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.invocationHelper(JavacTaskImpl.java:152) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:100) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:94) at jda.modules.common.javac.JavaC.javac(JavaC.java:188) at jda.modules.common.javac.JavaC.javacLoad(JavaC.java:78) at jda.modules.swtool.SwGenByCount.genDomainModel(SwGenByCount.java:153) at jda.modules.restfstool.test.gensw.RFSSwGenByCountCourseMan.gen(RFSSwGenByCountCourseMan.java:97) at jda.modules.restfstool.test.gensw.RFSSwGenByCountCourseMan.main(RFSSwGenByCountCourseMan.java:51)

[ducmle]

@ducmle Em bị lỗi khi chạy RFSSwGenByCountCourseMan 18:03:42.033 [main] INFO jda.modules.restfstool.test.gensw.RFSSwGenByCountCourseMan - Generating RFS software configuration ... 18:03:49.271 [main] INFO SwGenByCount - Generating the domain model 1 (copies)... Exception in thread "main" java.lang.IllegalStateException: error: no source files at jdk.compiler/com.sun.tools.javac.main.Arguments.reportDiag(Arguments.java:898) at jdk.compiler/com.sun.tools.javac.main.Arguments.validate(Arguments.java:462) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.prepareCompiler(JavacTaskImpl.java:194) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.lambda$doCall$0(JavacTaskImpl.java:101) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.invocationHelper(JavacTaskImpl.java:152) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:100) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:94) at jda.modules.common.javac.JavaC.javac(JavaC.java:188) at jda.modules.common.javac.JavaC.javacLoad(JavaC.java:78) at jda.modules.swtool.SwGenByCount.genDomainModel(SwGenByCount.java:153) at jda.modules.restfstool.test.gensw.RFSSwGenByCountCourseMan.gen(RFSSwGenByCountCourseMan.java:97) at jda.modules.restfstool.test.gensw.RFSSwGenByCountCourseMan.main(RFSSwGenByCountCourseMan.java:51)

"No source files" là JavaC ko thấy source files nào.

• Em kiểm tra xem trong thư mục gói có file .java nào đc tạo cho model1 ko?

[ducmle]

@ducmle Em bị lỗi khi chạy RFSSwGenByCountCourseMan 18:03:42.033 [main] INFO jda.modules.restfstool.test.gensw.RFSSwGenByCountCourseMan - Generating RFS software configuration ... 18:03:49.271 [main] INFO SwGenByCount - Generating the domain model 1 (copies)... Exception in thread "main" java.lang.IllegalStateException: error: no source files at jdk.compiler/com.sun.tools.javac.main.Arguments.reportDiag(Arguments.java:898) at jdk.compiler/com.sun.tools.javac.main.Arguments.validate(Arguments.java:462) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.prepareCompiler(JavacTaskImpl.java:194) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.lambda$doCall$0(JavacTaskImpl.java:101) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.invocationHelper(JavacTaskImpl.java:152) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:100) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:94) at jda.modules.common.javac.JavaC.javac(JavaC.java:188) at jda.modules.common.javac.JavaC.javacLoad(JavaC.java:78) at jda.modules.swtool.SwGenByCount.genDomainModel(SwGenByCount.java:153) at jda.modules.restfstool.test.gensw.RFSSwGenByCountCourseMan.gen(RFSSwGenByCountCourseMan.java:97) at jda.modules.restfstool.test.gensw.RFSSwGenByCountCourseMan.main(RFSSwGenByCountCourseMan.java:51)

"No source files" là JavaC ko thấy source files nào.

* Em kiểm tra xem trong thư mục gói có file .java nào đc tạo cho `model1` ko?

Ngoài ra, em kiểm tra xem các folder paths trong rfsgenconfig.json đã đúng chưa.

[haworker25]

@ducmle 1, Ko co file nao duoc tao trong model1 2, ""swConfigGen": { "rootSrcPath": "D:\Thesis\De_tai\Git\jda\modules\restfstool\src\example\java", "outputPath": "D:\Thesis\De_tai\Git\jda\modules\restfstool\target\classes", "seedDomainModelPkg" : "jda.modules.restfstool.test.performance.model", "modulesPkgPrefix": "jda.modules.restfstool.test.performance.modules", "softwarePkgPrefix": "jda.modules.restfstool.test.performance.software" },

[ducmle]

@ducmle 1, Ko co file nao duoc tao trong model1 2, ""swConfigGen": { "rootSrcPath": "D:\Thesis\De_tai\Git\jda\modules\restfstool\src\example\java", "outputPath": "D:\Thesis\De_tai\Git\jda\modules\restfstool\target\classes", "seedDomainModelPkg" : "jda.modules.restfstool.test.performance.model", "modulesPkgPrefix": "jda.modules.restfstool.test.performance.modules", "softwarePkgPrefix": "jda.modules.restfstool.test.performance.software" },

Em thử dùng '\\' thay cho '\' xem.

[haworker25]

@ducmle 1, Ko co file nao duoc tao trong model1 2, ""swConfigGen": { "rootSrcPath": "D:\Thesis\De_tai\Git\jda\modules\restfstool\src\example\java", "outputPath": "D:\Thesis\De_tai\Git\jda\modules\restfstool\target\classes", "seedDomainModelPkg" : "jda.modules.restfstool.test.performance.model", "modulesPkgPrefix": "jda.modules.restfstool.test.performance.modules", "softwarePkgPrefix": "jda.modules.restfstool.test.performance.software" },

Em thử dùng '\\' thay cho '\' xem. Copy lên đây nó mất, chứ trong file của em là '\\'

[ducmle]

Nghe hơi lạ. Em thử cho debug ở dòng 153 của lớp SwGenByCount (xem dưới đây trong error output) rồi chạy và gửi lên đây giá trị của 4 biến sau: seedModelPkg, rootSrcDir, targetModelSrcFiles, outputDir

at jda.modules.swtool.SwGenByCount.genDomainModel(SwGenByCount.java:153)

[haworker25]

@ducmle Bây giờ lại lỗi khác: D:\Thesis\De_tai\Git\jda\modules\restfstool\src\example\java\jda\modules\restfstool\test\performance\modules1\ModuleStudent1.java:13: error: cannot find symbol import jda.modules.restfstool.test.performance.model1.Date; ^ symbol: class Date location: package jda.modules.restfstool.test.performance.model1 D:\Thesis\De_tai\Git\jda\modules\restfstool\src\example\java\jda\modules\restfstool\test\performance\modules1\ModuleStudent1.java:34: error: cannot find symbol private Date dob; ^ symbol: class Date location: class ModuleStudent1 D:\Thesis\De_tai\Git\jda\modules\restfstool\src\example\java\jda\modules\restfstool\test\performance\modules1\ModuleStudent1.java:46: error: cannot find symbol private Collection enrolments; ^ symbol: class Collection location: class ModuleStudent1 D:\Thesis\De_tai\Git\jda\modules\restfstool\src\example\java\jda\modules\restfstool\test\performance\modules1\ModuleStudClass1.java:27: error: cannot find symbol private Collection students; ^ symbol: class Collection location: class ModuleStudClass1 4 errors Exception in thread "main" jda.modules.common.exceptions.NotPossibleException: [NotPossibleException.FAIL_TO_COMPILE_CLASS] Lỗi khi biên dịch lớp: Directory: D:\Thesis\De_tai\Git\jda\modules\restfstool\src\example\java at jda.modules.common.javac.JavaC.javac(JavaC.java:190) at jda.modules.common.javac.JavaC.javacLoad(JavaC.java:78) at jda.modules.common.javac.JavaC.javacLoad(JavaC.java:99) at jda.modules.swtool.SwGenByCount.genMCCs(SwGenByCount.java:223) at jda.modules.restfstool.test.gensw.RFSSwGenByCountCourseMan.gen(RFSSwGenByCountCourseMan.java:98) at jda.modules.restfstool.test.gensw.RFSSwGenByCountCourseMan.main(RFSSwGenByCountCourseMan.java:51)

[ducmle]

@ducmle Bây giờ lại lỗi khác: D:\Thesis\De_tai\Git\jda\modules\restfstool\src\example\java\jda\modules\restfstool\test\performance\modules1\ModuleStudent1.java:13: error: cannot find symbol import jda.modules.restfstool.test.performance.model1.Date; ^ symbol: class Date location: package jda.modules.restfstool.test.performance.model1 D:\Thesis\De_tai\Git\jda\modules\restfstool\src\example\java\jda\modules\restfstool\test\performance\modules1\ModuleStudent1.java:34: error: cannot find symbol private Date dob; ^ symbol: class Date location: class ModuleStudent1 D:\Thesis\De_tai\Git\jda\modules\restfstool\src\example\java\jda\modules\restfstool\test\performance\modules1\ModuleStudent1.java:46: error: cannot find symbol private Collection enrolments; ^ symbol: class Collection location: class ModuleStudent1 D:\Thesis\De_tai\Git\jda\modules\restfstool\src\example\java\jda\modules\restfstool\test\performance\modules1\ModuleStudClass1.java:27: error: cannot find symbol private Collection students; ^ symbol: class Collection location: class ModuleStudClass1 4 errors Exception in thread "main" jda.modules.common.exceptions.NotPossibleException: [NotPossibleException.FAIL_TO_COMPILE_CLASS] Lỗi khi biên dịch lớp: Directory: D:\Thesis\De_tai\Git\jda\modules\restfstool\src\example\java at jda.modules.common.javac.JavaC.javac(JavaC.java:190) at jda.modules.common.javac.JavaC.javacLoad(JavaC.java:78) at jda.modules.common.javac.JavaC.javacLoad(JavaC.java:99) at jda.modules.swtool.SwGenByCount.genMCCs(SwGenByCount.java:223) at jda.modules.restfstool.test.gensw.RFSSwGenByCountCourseMan.gen(RFSSwGenByCountCourseMan.java:98) at jda.modules.restfstool.test.gensw.RFSSwGenByCountCourseMan.main(RFSSwGenByCountCourseMan.java:51)

import jda.modules.restfstool.test.performance.model1.Date là sai rồi.

Em kiểm tra model.Student xem Date import đúng ko?

[ducmle]

File Console.log khi chạy đúng. console.log

[ducmle]

@haworker25 @haworker25 Test data set: see issue #34.

Em run thử với sw variant 1, 5, 10 trước đã nhé. Xem thời gian thế nào rồi báo anh biết. Có thể phải điều chỉnh thêm test data.

[ducmle]

@haworker25

• Thanks for the test result file
• Can you keep running until there is no memory left :-). I wanted to which variant is the limit.
• The test result may not be 100% accurate, but at least we know (on average) the resource utilisation of our generators.
• We need to submit the paper by 10/10 so I will take one last look at the result file on 08/10 and see what result we can use.

[haworker25]

Error when running courseman95: timeout benchmark log.txt :