wangsanshi123 commented 2 years ago

版本：最新版本部署方式：docker-compose 现象：web界面显示成功日志报错

详细日志： 2022-08-11 20:10:48,294-INFO: start to run task: JAFFBeta, execute_processes: ['compile', 'analyze', 'datahandle'] 2022-08-11 20:10:48,330-INFO: Load tool config: https://git.code.tencent.com/TCA/tca-tools/puppy-tools-config.git 2022-08-11 20:10:48,337-INFO: Initing common tools ... 2022-08-11 20:10:48,337-INFO: Initing 2 tools, please wait a minute ...

[Tools init]: 0%| | 0/2 [00:00<?, ?it/s] [Tools init]: 100%|█████████████████████████████████████████████████| 2/2 [00:00<00:00, 2362.99it/s] 2022-08-11 20:10:48,340-INFO: Initialize tools done.(use time: 0.00s) 2022-08-11 20:10:48,340-INFO: run cmd: git lfs install 2022-08-11 20:10:48,358-INFO: Git LFS initialized. 2022-08-11 20:10:48,482-INFO: Initing other tools ... 2022-08-11 20:10:48,485-INFO: Initing 1 tools, please wait a minute ...

[Tools init]: 0%| | 0/1 [00:00<?, ?it/s] [Tools init]: 100%|█████████████████████████████████████████████████| 1/1 [00:00<00:00, 2097.15it/s] 2022-08-11 20:10:48,486-INFO: Initialize tools done.(use time: 0.00s) 2022-08-11 20:10:48,490-INFO: task start : <class 'task.model.compile.CompileTask'> 2022-08-11 20:10:48,490-INFO: pre_compile start. 2022-08-11 20:10:48,490-INFO: start __checkout_source_dir. 2022-08-11 20:10:48,490-INFO: 常规项目，直接拉取或复用本地缓存资源. 2022-08-11 20:10:49,570-INFO: load_source_dir done. source_dir insert env. sourcedir: /workspace/client/data/sourcedirs/git_6vrq6rn8 2022-08-11 20:10:49,572-INFO: pre_compile done. 2022-08-11 20:10:49,572-INFO: PATH =

/workspace/tools/linux-git-lfs-v3.1.2 /workspace/tools/linux-git-lfs-v3.1.2 /usr/local/bin /usr/local/sbin /usr/local/bin /usr/sbin /usr/bin /sbin /bin

2022-08-11 20:10:49,572-INFO: Codemetric only scans main code (exclude test_source, auto_generate_source, third_party_source). 2022-08-11 20:10:49,572-INFO: Codelint only cans main code. 2022-08-11 20:10:49,591-INFO: 已经上报过代码行数,本次分析无需上报. 2022-08-11 20:10:49,602-INFO: Tool compile start. 2022-08-11 20:10:49,602-INFO: Cmd: mvn clean package 2022-08-11 20:10:49,603-INFO: Generated shell file: /workspace/client/data/taskdirs/task_812/workdir/tca_build.sh 2022-08-11 20:10:49,603-INFO: basic compile start. 2022-08-11 20:10:49,603-INFO: subprocc normal mode: ['bash', '/workspace/client/data/taskdirs/task_812/workdir/tca_build.sh'] 2022-08-11 20:10:49,603-INFO: run build cmd: bash /workspace/client/data/taskdirs/task_812/workdir/tca_build.sh 2022-08-11 20:10:50,637-INFO: [[1;34mINFO[m] Scanning for projects... 2022-08-11 20:10:50,798-INFO: [[1;34mINFO[m] 2022-08-11 20:10:50,803-INFO: [[1;34mINFO[m] [1m-------------------------< [0;36msec:java-sec-code[0;1m >--------------------------[m 2022-08-11 20:10:50,803-INFO: [[1;34mINFO[m] [1mBuilding java-sec-code 1.0.0[m 2022-08-11 20:10:50,803-INFO: [[1;34mINFO[m] [1m--------------------------------[ jar ]---------------------------------[m 2022-08-11 20:10:51,607-INFO: [[1;34mINFO[m] 2022-08-11 20:10:51,609-INFO: [[1;34mINFO[m] [1m--- [0;32mmaven-clean-plugin:2.6.1:clean[m m[m @ [36mjava-sec-code[0;1m ---[m 2022-08-11 20:10:51,673-INFO: [[1;34mINFO[m] Deleting /workspace/client/data/sourcedirs/git_6vrq6rn8/target 2022-08-11 20:10:51,704-INFO: [[1;34mINFO[m] 2022-08-11 20:10:51,705-INFO: [[1;34mINFO[m] [1m--- [0;32mmaven-resources-plugin:2.6:resources[m m[m @ [36mjava-sec-code[0;1m ---[m 2022-08-11 20:10:51,806-INFO: [[1;34mINFO[m] Using 'UTF-8' encoding to copy filtered resources. 2022-08-11 20:10:51,810-INFO: [[1;34mINFO[m] Copying 1 resource 2022-08-11 20:10:51,826-INFO: [[1;34mINFO[m] Copying 14 resources 2022-08-11 20:10:51,834-INFO: [[1;34mINFO[m] 2022-08-11 20:10:51,835-INFO: [[1;34mINFO[m] [1m--- [0;32mmaven-compiler-plugin:3.1:compile[m m[m @ [36mjava-sec-code[0;1m ---[m 2022-08-11 20:10:52,004-INFO: [[1;34mINFO[m] Changes detected - recompiling the module! 2022-08-11 20:10:52,006-INFO: [[1;34mINFO[m] Compiling 60 source files to /workspace/client/data/sourcedirs/git_6vrq6rn8/target/classes 2022-08-11 20:10:54,546-INFO: [[1;34mINFO[m] 2022-08-11 20:10:54,546-INFO: [[1;34mINFO[m] [1m--- [0;32mmaven-resources-plugin:2.6:testResources[m m[m @ [36mjava-sec-code[0;1m ---[m 2022-08-11 20:10:54,550-INFO: [[1;34mINFO[m] Using 'UTF-8' encoding to copy filtered resources. 2022-08-11 20:10:54,550-INFO: [[1;34mINFO[m] skip non existing resourceDirectory /workspace/client/data/sourcedirs/git_6vrq6rn8/src/test/resources 2022-08-11 20:10:54,551-INFO: [[1;34mINFO[m] 2022-08-11 20:10:54,551-INFO: [[1;34mINFO[m] [1m--- [0;32mmaven-compiler-plugin:3.1:testCompile[m m[m @ [36mjava-sec-code[0;1m ---[m 2022-08-11 20:10:54,554-INFO: [[1;34mINFO[m] No sources to compile 2022-08-11 20:10:54,554-INFO: [[1;34mINFO[m] 2022-08-11 20:10:54,554-INFO: [[1;34mINFO[m] [1m--- [0;32mmaven-surefire-plugin:2.18.1:test[m m[m @ [36mjava-sec-code[0;1m ---[m 2022-08-11 20:10:54,621-INFO: [[1;34mINFO[m] No tests to run. 2022-08-11 20:10:54,631-INFO: [[1;34mINFO[m] 2022-08-11 20:10:54,631-INFO: [[1;34mINFO[m] [1m--- [0;32mmaven-jar-plugin:2.6:jar[m m[m @ [36mjava-sec-code[0;1m ---[m 2022-08-11 20:10:54,864-INFO: [[1;34mINFO[m] Building jar: /workspace/client/data/sourcedirs/git_6vrq6rn8/target/java-sec-code-1.0.0.jar 2022-08-11 20:10:54,932-INFO: [[1;34mINFO[m] 2022-08-11 20:10:54,932-INFO: [[1;34mINFO[m] [1m--- [0;32mspring-boot-maven-plugin:1.5.1.RELEASE:repackage[m m[m @ [36mjava-sec-code[0;1m ---[m 2022-08-11 20:10:55,490-INFO: [[1;34mINFO[m] [1m------------------------------------------------------------------------[m 2022-08-11 20:10:55,491-INFO: [[1;34mINFO[m] [1;32mBUILD SUCCESS[m 2022-08-11 20:10:55,491-INFO: [[1;34mINFO[m] [1m------------------------------------------------------------------------[m 2022-08-11 20:10:55,492-INFO: [[1;34mINFO[m] Total time: 4.872 s 2022-08-11 20:10:55,493-INFO: [[1;34mINFO[m] Finished at: 2022-08-11T12:10:55Z 2022-08-11 20:10:55,493-INFO: [[1;34mINFO[m] [1m------------------------------------------------------------------------[m 2022-08-11 20:10:55,564-INFO: exist_error_flag : False 2022-08-11 20:10:55,564-INFO: exist_success_flag : False 2022-08-11 20:10:55,564-INFO: exist_100_flag : False 2022-08-11 20:10:55,564-INFO: cov_no_file_emitted : False 2022-08-11 20:10:55,564-INFO: basic compile done. 2022-08-11 20:10:55,564-INFO: Tool compile done. 2022-08-11 20:10:55,564-INFO: done_compile start. 2022-08-11 20:10:55,564-INFO: task done : <class 'task.model.compile.CompileTask'> 2022-08-11 20:10:55,565-INFO: task start : <class 'task.model.analyze.AnalyzeTask'> 2022-08-11 20:10:55,565-INFO: pre_analyze start. 2022-08-11 20:10:55,565-INFO: pre_analyze done. 2022-08-11 20:10:58,153-INFO: /workspace/client/data/sourcedirs/git_6vrq6rn8 路径下共发现被分析项目数：65 2022-08-11 20:10:58,153-INFO: java.sql.SQLException: Error opening connection 2022-08-11 20:10:58,154-INFO: at org.sqlite.SQLiteConnection.open(SQLiteConnection.java:259) 2022-08-11 20:10:58,154-INFO: at org.sqlite.SQLiteConnection.(SQLiteConnection.java:62) 2022-08-11 20:10:58,154-INFO: at org.sqlite.jdbc3.JDBC3Connection.(JDBC3Connection.java:23) 2022-08-11 20:10:58,154-INFO: at org.sqlite.jdbc4.JDBC4Connection.(JDBC4Connection.java:19) 2022-08-11 20:10:58,154-INFO: at org.sqlite.JDBC.createConnection(JDBC.java:104) 2022-08-11 20:10:58,154-INFO: at org.sqlite.JDBC.connect(JDBC.java:77) 2022-08-11 20:10:58,154-INFO: at java.sql.DriverManager.getConnection(DriverManager.java:677) 2022-08-11 20:10:58,154-INFO: at java.sql.DriverManager.getConnection(DriverManager.java:189) 2022-08-11 20:10:58,154-INFO: at com.tencent.api.a.c.b(DataBase.java:62) 2022-08-11 20:10:58,154-INFO: at com.tencent.api.a.c.(DataBase.java:45) 2022-08-11 20:10:58,154-INFO: at com.tencent.api.parser.b.(APIParserThread.java:52) 2022-08-11 20:10:58,154-INFO: at com.tencent.api.parser.ByteCodeParser.b(ByteCodeParser.java:54) 2022-08-11 20:10:58,154-INFO: at com.tencent.api.parser.ByteCodeParser.b(ByteCodeParser.java:37) 2022-08-11 20:10:58,154-INFO: at com.tencent.api.parser.ByteCodeParser.main(ByteCodeParser.java:117) 2022-08-11 20:10:58,154-INFO: Caused by: java.io.UncheckedIOException: java.nio.file.FileSystemException: /proc/self/map_files: Operation not permitted 2022-08-11 20:10:58,154-INFO: at java.nio.file.Files$2.hasNext(Files.java:3709) 2022-08-11 20:10:58,154-INFO: at java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1811) 2022-08-11 20:10:58,154-INFO: at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:127) 2022-08-11 20:10:58,155-INFO: at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:502) 2022-08-11 20:10:58,155-INFO: at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:488) 2022-08-11 20:10:58,155-INFO: at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) 2022-08-11 20:10:58,155-INFO: at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) 2022-08-11 20:10:58,155-INFO: at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) 2022-08-11 20:10:58,155-INFO: at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) 2022-08-11 20:10:58,155-INFO: at java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:528) 2022-08-11 20:10:58,155-INFO: at org.sqlite.util.OSInfo.isMusl(OSInfo.java:127) 2022-08-11 20:10:58,155-INFO: at org.sqlite.util.OSInfo.translateOSNameToFolderName(OSInfo.java:226) 2022-08-11 20:10:58,155-INFO: at org.sqlite.util.OSInfo.getOSName(OSInfo.java:108) 2022-08-11 20:10:58,155-INFO: at org.sqlite.util.OSInfo.getNativeLibFolderPathForCurrentOS(OSInfo.java:104) 2022-08-11 20:10:58,155-INFO: at org.sqlite.SQLiteJDBCLoader.loadSQLiteNativeLibrary(SQLiteJDBCLoader.java:348) 2022-08-11 20:10:58,155-INFO: at org.sqlite.SQLiteJDBCLoader.initialize(SQLiteJDBCLoader.java:65) 2022-08-11 20:10:58,155-INFO: at org.sqlite.core.NativeDB.load(NativeDB.java:56) 2022-08-11 20:10:58,155-INFO: at org.sqlite.SQLiteConnection.open(SQLiteConnection.java:256) 2022-08-11 20:10:58,155-INFO: ... 13 more 2022-08-11 20:10:58,155-INFO: Caused by: java.nio.file.FileSystemException: /proc/self/map_files: Operation not permitted 2022-08-11 20:10:58,155-INFO: at sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator.readNextEntry(UnixDirectoryStream.java:168) 2022-08-11 20:10:58,155-INFO: at sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator.hasNext(UnixDirectoryStream.java:198) 2022-08-11 20:10:58,155-INFO: at java.nio.file.Files$2.hasNext(Files.java:3707) 2022-08-11 20:10:58,155-INFO: ... 30 more 2022-08-11 20:10:58,155-INFO: Exception in thread "main" java.lang.NullPointerException 2022-08-11 20:10:58,155-INFO: at com.tencent.api.a.c.a(DataBase.java:97) 2022-08-11 20:10:58,156-INFO: at com.tencent.api.parser.b.(APIParserThread.java:53) 2022-08-11 20:10:58,156-INFO: at com.tencent.api.parser.ByteCodeParser.b(ByteCodeParser.java:54) 2022-08-11 20:10:58,156-INFO: at com.tencent.api.parser.ByteCodeParser.b(ByteCodeParser.java:37) 2022-08-11 20:10:58,156-INFO: at com.tencent.api.parser.ByteCodeParser.main(ByteCodeParser.java:117) 2022-08-11 20:10:58,705-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,705-INFO: 该项目中未找到该API的使用 org.apache.logging.log4j.Logger - error;warn;info;debug;fatal;trace;log 2022-08-11 20:10:58,705-INFO: rule:FindLog4j done. 2022-08-11 20:10:58,705-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,705-INFO: 该项目中未找到该API的使用 org.apache.logging.log4j.LogManager - getLogger 2022-08-11 20:10:58,705-INFO: rule:Log4jLogManager done. 2022-08-11 20:10:58,705-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,705-INFO: 该项目中未找到该API的使用 Environment - getExternalStorageDirectory 2022-08-11 20:10:58,705-INFO: rule:DynamicAPIGetExternalStorageDirectory done. 2022-08-11 20:10:58,705-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,706-INFO: 该项目中未找到该API的使用 CalendarContract -
2022-08-11 20:10:58,706-INFO: rule:DynamicPermissionsCalendarContract done. 2022-08-11 20:10:58,706-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,706-INFO: 该项目中未找到该API的使用 IMAGE_CAPTURE -
2022-08-11 20:10:58,706-INFO: rule:DynamicPermissionsImageCapture done. 2022-08-11 20:10:58,706-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,706-INFO: 该项目中未找到该API的使用 ACTION_IMAGE_CAPTURE -
2022-08-11 20:10:58,706-INFO: rule:DynamicPermissionsActionImageCapture done. 2022-08-11 20:10:58,706-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,706-INFO: 该项目中未找到该API的使用 TelephonyManager -
2022-08-11 20:10:58,706-INFO: rule:DynamicAPITelephonyManager done. 2022-08-11 20:10:58,706-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,706-INFO: 该项目中未找到该API的使用 LocationManager -
2022-08-11 20:10:58,706-INFO: rule:DynamicAPILocationManager done. 2022-08-11 20:10:58,706-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,706-INFO: 该项目中未找到该API的使用 WallpaperManager - getFastDrawable;peekFastDrawable;getWallpaperFile 2022-08-11 20:10:58,706-INFO: rule:DynamicAPIWallpaperManager done. 2022-08-11 20:10:58,706-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,706-INFO: 该项目中未找到该API的使用 CameraManager - openCamera 2022-08-11 20:10:58,707-INFO: rule:DynamicAPICameraManager done. 2022-08-11 20:10:58,707-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,707-INFO: 该项目中未找到该API的使用 WifiRttManager - startRanging 2022-08-11 20:10:58,707-INFO: rule:DynamicAPIWifiRttManager done. 2022-08-11 20:10:58,707-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,707-INFO: 该项目中未找到该API的使用 SubscriptionManager - getActiveSubscriptionInfo;getActiveSubscriptionInfoForSimSlotIndex;getActiveSubscriptionInfoList;getActiveSubscriptionInfoCount 2022-08-11 20:10:58,707-INFO: rule:DynamicAPISubscriptionManager done. 2022-08-11 20:10:58,707-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,707-INFO: 该项目中未找到该API的使用 TwilightManager - getLastKnownLocationForProvider 2022-08-11 20:10:58,707-INFO: rule:DynamicAPITwilightManager done. 2022-08-11 20:10:58,707-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,707-INFO: 该项目中未找到该API的使用 android.hardware.Camera - open;getParameters 2022-08-11 20:10:58,707-INFO: rule:camera1API done. 2022-08-11 20:10:58,707-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,707-INFO: 该项目中未找到该API的使用 LocationManager - requestLocationUpdates 2022-08-11 20:10:58,707-INFO: rule:LocationAPI done. 2022-08-11 20:10:58,707-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,707-INFO: 该项目中未找到该API的使用 MediaRecorder - prepare;start 2022-08-11 20:10:58,707-INFO: rule:FindMediaRecorder done. 2022-08-11 20:10:58,708-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,708-INFO: 该项目中未找到该API的使用 AudioRecord - startRecording 2022-08-11 20:10:58,708-INFO: rule:FindAudioRecord done. 2022-08-11 20:10:58,708-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,708-INFO: 该项目中未找到该API的使用 MediaStore.ACTION_IMAGE_CAPTUR -
2022-08-11 20:10:58,708-INFO: rule:FileShareMediaStore done. 2022-08-11 20:10:58,708-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,708-INFO: 该项目中未找到该API的使用 vnd.android.package-archive -
2022-08-11 20:10:58,708-INFO: rule:FileSharePackageArchive done. 2022-08-11 20:10:58,708-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,708-INFO: 该项目中未找到该API的使用 MODE_WORLD_READABLE -
2022-08-11 20:10:58,708-INFO: rule:FileRWModeReadable done. 2022-08-11 20:10:58,708-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,708-INFO: 该项目中未找到该API的使用 MODE_WORLD_WRITEABLE -
2022-08-11 20:10:58,708-INFO: rule:FileRWModeWriteable done. 2022-08-11 20:10:58,709-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,709-INFO: 该项目中未找到该API的使用 SmsManager - sendTextMessage 2022-08-11 20:10:58,709-INFO: rule:FindSmsManager done. 2022-08-11 20:10:58,709-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,709-INFO: 该项目中未找到该API的使用 SmsMessage - createFromPdu 2022-08-11 20:10:58,709-INFO: rule:FindSmsMessage done. 2022-08-11 20:10:58,709-INFO: Error: no such table: apiinfo 2022-08-11 20:10:58,709-INFO: 该项目中未找到该API的使用 ContactsContract - Contacts;RawContacts;Data 2022-08-11 20:10:58,709-INFO: rule:ContactsAPI done. 2022-08-11 20:10:58,709-INFO: done_analyze start. 2022-08-11 20:10:58,710-INFO: done_analyze done. 2022-08-11 20:10:58,710-INFO: task done : <class 'task.model.analyze.AnalyzeTask'> 2022-08-11 20:10:58,710-INFO: task start : <class 'task.model.datahandle.DataHandleTask'> 2022-08-11 20:10:58,766-INFO: [Start] set_format_type 2022-08-11 20:10:58,792-INFO: [End] set_format_type (use time: 0.025659561157226562) 2022-08-11 20:10:58,792-INFO: [Start] set_filter_type_list 2022-08-11 20:10:58,818-INFO: start: filter issues according to path filter. 2022-08-11 20:10:58,826-INFO: finished: filter issues according to path filter. 2022-08-11 20:10:58,826-INFO: [End] set_filter_type_list (use time: 0.034294843673706055) 2022-08-11 20:10:58,826-INFO: [Start] set_submodule_handle 2022-08-11 20:10:58,826-INFO: [End] set_submodule_handle (use time: 1.0251998901367188e-05) 2022-08-11 20:10:58,826-INFO: [Start] set_blame_type 2022-08-11 20:10:58,852-INFO: Start: blame file last change author. 2022-08-11 20:10:59,255-INFO: End: blame file last change author. 2022-08-11 20:10:59,256-INFO: [End] set_blame_type (use time: 0.4292612075805664) 2022-08-11 20:10:59,256-INFO: [Start] set_filter_type_list 2022-08-11 20:10:59,294-INFO: [End] set_filter_type_list (use time: 0.03782176971435547) 2022-08-11 20:10:59,294-INFO: [Start] set_issue_ignore_type 2022-08-11 20:10:59,294-INFO: start: igonre issues according to code comments. 2022-08-11 20:10:59,295-INFO: finished: igonre issues according to code comments. 2022-08-11 20:10:59,295-INFO: [End] set_issue_ignore_type (use time: 0.0011870861053466797) 2022-08-11 20:10:59,295-INFO: [Start] set_result_pack_diff_info 2022-08-11 20:10:59,295-INFO: [End] set_result_pack_diff_info (use time: 1.2874603271484375e-05) 2022-08-11 20:10:59,295-INFO: [Start] set_add_file_info 2022-08-11 20:10:59,295-INFO: [End] set_add_file_info (use time: 1.4543533325195312e-05) 2022-08-11 20:10:59,295-INFO: [Start] set_issue_hash 2022-08-11 20:10:59,300-INFO: [End] set_issue_hash (use time: 0.005124092102050781) 2022-08-11 20:10:59,301-INFO: [Start] set_add_person_info 2022-08-11 20:10:59,301-INFO: [End] set_add_person_info (use time: 7.62939453125e-06) 2022-08-11 20:10:59,301-INFO: [Start] set_issue_split 2022-08-11 20:10:59,301-INFO: [End] set_issue_split (use time: 0.000606536865234375) 2022-08-11 20:10:59,301-INFO: datahandle done! 2022-08-11 20:10:59,301-INFO: task done : <class 'task.model.datahandle.DataHandleTask'> 2022-08-11 20:10:59,302-INFO: task result: /workspace/client/data/taskdirs/task_812/task_response.json

wangsanshi123 commented 2 years ago

其他工具也有类似问题

cyw3 commented 2 years ago

这里Client的镜像是 TCA提供的compose的原生镜像是吗？
可以提供一个小的demo项目，方便这边测试下吗？

wangsanshi123 commented 2 years ago

通过docker-compose文件构建的，仅仅加了个代理，加快构建速度，其他的没有改动。如果需要测试demo,是发构建好的镜像吗？我把docker-compse文件贴在下面了

完整docker-compose文件如下 version: "3"

services: mysql:

如果需要在ARM架构上运行，可以使用 mariadb:10.5.8 镜像

image: mysql:5.7.24
# image: mariadb:10.5.8  # For ARM
command:
  [
    "--log-bin=mysql-bin",
    "--character-set-server=utf8mb4",
    "--collation-server=utf8mb4_unicode_ci",
    "--innodb_flush_log_at_trx_commit=1",
    "--sync_binlog=1",
    "--server-id=1"
  ]
environment:
  MYSQL_DATABASE: "codedog_db"
  MYSQL_ROOT_PASSWORD: "TCA!@#2021"
volumes:
  - ./.docker_data/mysql:/var/lib/mysql
  - ./server/sql/init.sql:/docker-entrypoint-initdb.d/init.sql
expose:
  - "3306"
restart: always

redis: image: redis:5.0.5 command: redis-server volumes:

./.docker_data/redis:/data expose:
"6379" restart: always

main-server: env_file:
./server/dockerconfs/.env.local image: tca-main:latest build: context: ./server/projects/main dockerfile: ../../dockerconfs/Dockerfile-common-${ORIGIN:-mirror} command: [ "gunicorn", "codedog.wsgi", "-c", "main.gunicorn.conf.py" ] volumes:
./server/configs/django/local_main.py:/var/www/django/codedog/codedog/settings/local.py
./.docker_data/logs/main_server:/var/www/django/codedog/log expose:
"8001" depends_on:
mysql
redis environment:
http_proxy=http://10.125.4.58:1081
https_proxy=http://10.125.4.58:1081

main-worker: env_file:
./server/dockerconfs/.env.local image: tca-main:latest command: [ "celery", "-A", "codedog", "worker", "--concurrency=2", "-l", "INFO" ] volumes:
./server/configs/django/local_main.py:/var/www/django/codedog/codedog/settings/local.py
./.docker_data/logs/main_worker:/var/www/django/codedog/log depends_on:
mysql
redis

main-beat: env_file:
./server/dockerconfs/.env.local image: tca-main:latest command: [ "celery", "-A", "codedog", "beat", "-S", "redbeat.RedBeatScheduler", "-l", "INFO" ] volumes:
./server/configs/django/local_main.py:/var/www/django/codedog/codedog/settings/local.py
./.docker_data/logs/main_beat:/var/www/django/codedog/log depends_on:
mysql
redis

analysis-server: env_file:
./server/dockerconfs/.env.local image: tca-analysis:latest build: context: ./server/projects/analysis dockerfile: ../../dockerconfs/Dockerfile-common-${ORIGIN:-mirror} command: [ "gunicorn", "codedog.wsgi", "-c", "analysis.gunicorn.conf.py" ] volumes:
./server/configs/django/local_analysis.py:/var/www/django/codedog/codedog/settings/local.py
./.docker_data/logs/analysis_server:/var/www/django/codedog/log expose:
"8002" depends_on:
mysql
redis

analysis-worker: env_file:
./server/dockerconfs/.env.local image: tca-analysis:latest command: [ "celery", "-A", "codedog", "worker", "--concurrency=2", "-l", "INFO" ] volumes:
./server/configs/django/local_analysis.py:/var/www/django/codedog/codedog/settings/local.py
./.docker_data/logs/analysis_worker:/var/www/django/codedog/log depends_on:
mysql
redis

login-server: env_file:
./server/dockerconfs/.env.local build: context: ./server/projects/login dockerfile: ../../dockerconfs/Dockerfile-common-${ORIGIN:-mirror} command: [ "gunicorn", "apps.wsgi", "-c", "login.gunicorn.conf.py" ] volumes:
./server/configs/django/local_login.py:/var/www/django/codedog/apps/settings/local.py
./.docker_data/logs/login:/var/www/django/codedog/log expose:
"8003" depends_on:
mysql

file-server: env_file:
./server/dockerconfs/.env.local build: context: ./server/projects/file dockerfile: ../../dockerconfs/Dockerfile-common-${ORIGIN:-mirror} command: [ "gunicorn", "codedog_file_server.wsgi", "-c", "file.gunicorn.conf.py" ] volumes:
./server/configs/django/local_file.py:/var/www/django/codedog/codedog_file_server/env/local.py
./.docker_data/filedata:/var/www/django/codedog/data
./.docker_data/logs/file_server:/var/www/django/codedog/log expose:
"8804" depends_on:
mysql

file-nginx: image: nginx:1.13.7 volumes:
./server/configs/nginx/tca_file_local_compose.conf:/etc/nginx/conf.d/tca_file_local_compose.conf
./.docker_data/logs/file_nginx:/var/log/nginx expose:
"8004" depends_on:
file-server

scmproxy: env_file:
./server/dockerconfs/.env.local build: context: ./server/projects/scmproxy dockerfile: ../../dockerconfs/Dockerfile-common-${ORIGIN:-mirror} command: [ "python", "proxyserver.py" ] volumes:
./.docker_data/logs/scmproxy:/var/www/django/codedog/logs expose:
"8009"

nginx: build: context: ./web/tca-deploy-source dockerfile: ../../server/dockerconfs/Dockerfile-nginx-${ORIGIN:-mirror} env_file:
./web/tca-deploy-source/docker_conf/.env volumes:
./web/tca-deploy-source:/data/tca-deploy-source
./server/configs/nginx/tca_8000_compose.conf:/etc/nginx/conf.d/tca_8000_compose.conf entrypoint: bash /data/tca-deploy-source/scripts/deploy.sh init ports:
"9000:80"
"9001:8000" depends_on:
main-server
analysis-server
login-server
file-server

client: env_file:
./server/dockerconfs/.env.local build: context: ./client dockerfile: Dockerfile command:
/bin/bash
-c
| python3 codepuppy.py start -t $$CODEDOG_TOKEN volumes:
./tools/:/workspace/tools/
./.docker_data/client/appdata:/workspace/client/.appdata environment:
http_proxy=http://10.125.4.58:1081
https_proxy=http://10.125.4.58:1081

wangsanshi123 commented 2 years ago

client使用的Dockerfile没有做修改

wangsanshi123 commented 2 years ago

另外从漏洞结果上来看，我在自定义规则中把所有java安全相关的规则都选中了，但是只测出了44个问题，还有没有直接使用semgrep漏洞的数量多。注：已经申请了license,看起来是TCA中很多安全规则没有正常运转。

cyw3 commented 2 years ago

jaff,jaff_beta,jafc,jafc_beta执行异常，猜测是Docker Compose里面的Client镜像选择有问题，后续这边会调整为Centos。如果可以的话，可以试试直接下载release里面的TCA Client包，使用localscan方式在本地机器或者在centos镜像上执行看看。
可以发下TCA上Semgrep的执行log吗？

wangsanshi123 commented 2 years ago

2022-08-11 20:10:27,579-INFO: start to run task: Semgrep, execute_processes: ['analyze', 'datahandle'] 2022-08-11 20:10:27,640-INFO: Load tool config: https://git.code.tencent.com/TCA/tca-tools/puppy-tools-config.git 2022-08-11 20:10:27,647-INFO: Initing common tools ... 2022-08-11 20:10:27,647-INFO: Initing 2 tools, please wait a minute ...

[Tools init]: 0%| | 0/2 [00:00<?, ?it/s] [Tools init]: 100%|█████████████████████████████████████████████████| 2/2 [00:00<00:00, 2141.04it/s] 2022-08-11 20:10:27,651-INFO: Initialize tools done.(use time: 0.00s) 2022-08-11 20:10:27,651-INFO: run cmd: git lfs install 2022-08-11 20:10:27,673-INFO: Git LFS initialized. 2022-08-11 20:10:27,782-INFO: Initing other tools ... 2022-08-11 20:10:27,786-INFO: Initing 8 tools, please wait a minute ...

[Tools init]: 0%| | 0/8 [00:00<?, ?it/s] [Tools init]: 100%|█████████████████████████████████████████████████| 8/8 [00:00<00:00, 5594.27it/s] 2022-08-11 20:10:27,788-INFO: Initialize tools done.(use time: 0.00s) 2022-08-11 20:10:27,791-WARNING: 不是内置工具,使用自定义工具模块(No module named 'tool.semgrep') 2022-08-11 20:10:27,792-INFO: task start : <class 'task.model.analyze.AnalyzeTask'> 2022-08-11 20:10:27,793-INFO: pre_analyze start. 2022-08-11 20:10:27,793-INFO: start __checkout_source_dir. 2022-08-11 20:10:27,793-INFO: 常规项目，直接拉取或复用本地缓存资源. 2022-08-11 20:10:28,892-INFO: load_source_dir done. source_dir insert env. sourcedir: /workspace/client/data/sourcedirs/git_6vrq6rn8 2022-08-11 20:10:28,893-INFO: pre_analyze done. 2022-08-11 20:10:28,893-INFO: PATH =

/workspace/tools/linux-node-v12.16.3/bin /workspace/tools/common-dart-code-metrics-1.7.1/bin /workspace/tools/linux-dart-sdk-v2.7.2/bin /workspace/tools/linux-jdk1.8.0_65/bin /workspace/tools/linux-git-lfs-v3.1.2 /workspace/tools/linux-git-lfs-v3.1.2 /usr/local/bin /usr/local/sbin /usr/local/bin /usr/sbin /usr/bin /sbin /bin

2022-08-11 20:10:28,894-INFO: Codemetric only scans main code (exclude test_source, auto_generate_source, third_party_source). 2022-08-11 20:10:28,894-INFO: Codelint only cans main code. 2022-08-11 20:10:28,912-INFO: 已经上报过代码行数,本次分析无需上报. 2022-08-11 20:10:28,920-INFO: Use local tool dir: /workspace/tools/custom-semgrep 2022-08-11 20:10:28,924-INFO: 正在设置CodeDog内置Python环境(PYTHON37_HOME)... 2022-08-11 20:10:28,935-INFO: Python 3.7.2 2022-08-11 20:10:29,025-INFO: 获取需要扫描的文件 2022-08-11 20:10:29,035-INFO: [文件数]过滤前：196，过滤后：168 2022-08-11 20:10:29,137-INFO: use SubProcController run cmd: python main.py 2022-08-11 20:10:41,016-INFO: METRICS: Using configs from the Registry (like --config=p/ci) reports pseudonymous rule metrics to semgrep.dev. 2022-08-11 20:10:41,016-INFO: To disable Registry rule metrics, use "--metrics=off". 2022-08-11 20:10:41,016-INFO: Using configs only from local files (like --config=xyz.yml) does not enable metrics. 2022-08-11 20:10:41,016-INFO: 2022-08-11 20:10:41,016-INFO: More information: https://semgrep.dev/docs/metrics 2022-08-11 20:10:41,016-INFO: 2022-08-11 20:10:43,578-INFO: 2022-08-11 20:10:43,579-INFO: Scanning across multiple languages: 2022-08-11 20:10:43,581-INFO: | 4 rules × 170 files 2022-08-11 20:10:43,582-INFO: java | 75 rules × 60 files 2022-08-11 20:10:43,582-INFO: 2022-08-11 20:10:45,324-INFO: 2022-08-11 20:10:45,324-INFO: Some files were skipped or only partially analyzed. 2022-08-11 20:10:45,324-INFO: Scan skipped: 1 files larger than 1.0 MB 2022-08-11 20:10:45,324-INFO: For a full list of skipped files, run semgrep with the --verbose flag. 2022-08-11 20:10:45,324-INFO: 2022-08-11 20:10:45,324-INFO: Ran 89 rules on 167 files: 56 findings. 2022-08-11 20:10:45,324-INFO: 2022-08-11 20:10:45,324-INFO: A new version of Semgrep is available. See https://semgrep.dev/docs/upgrading 2022-08-11 20:10:45,380-INFO: -- start run tool ... 2022-08-11 20:10:45,380-INFO: [debug] source_dir: /workspace/client/data/sourcedirs/git_6vrq6rn8 2022-08-11 20:10:45,380-INFO: [debug] envs:
2022-08-11 20:10:45,380-INFO: 2022-08-11 20:10:45,380-INFO: [debug] PATH: /workspace/tools/linux-node-v12.16.3/bin:/workspace/tools/common-dart-code-metrics-1.7.1/bin:/workspace/tools/linux-dart-sdk-v2.7.2/bin:/workspace/tools/linux-jdk1.8.0_65/bin:/workspace/tools/linux-git-lfs-v3.1.2:/workspace/tools/linux-node-v12.16.3/bin:/workspace/tools/common-dart-code-metrics-1.7.1/bin:/workspace/tools/linux-dart-sdk-v2.7.2/bin:/workspace/tools/linux-jdk1.8.0_65/bin:/workspace/tools/linux-git-lfs-v3.1.2:/workspace/tools/linux-git-lfs-v3.1.2:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-08-11 20:10:45,380-INFO: [debug] semgrep version: 0.100.0 2022-08-11 20:10:45,380-INFO: [debug] cmd: python3 -m semgrep scan --config /workspace/tools/custom-semgrep/config_rules --no-git-ignore --json --output error_output.json /workspace/client/data/sourcedirs/git_6vrq6rn8 2022-08-11 20:10:45,380-INFO: start data handle 2022-08-11 20:10:45,380-INFO: -- end ... 2022-08-11 20:10:45,388-INFO: 格式化结果 ... 2022-08-11 20:10:45,389-INFO: done_analyze start. 2022-08-11 20:10:45,389-INFO: done_analyze done. 2022-08-11 20:10:45,389-INFO: task done : <class 'task.model.analyze.AnalyzeTask'> 2022-08-11 20:10:45,389-INFO: task start : <class 'task.model.datahandle.DataHandleTask'> 2022-08-11 20:10:45,420-INFO: [Start] set_format_type 2022-08-11 20:10:45,443-INFO: [End] set_format_type (use time: 0.023731231689453125) 2022-08-11 20:10:45,444-INFO: [Start] set_filter_type_list 2022-08-11 20:10:45,468-INFO: start: filter issues according to scm diff. 2022-08-11 20:10:45,468-INFO: finished: filter issues according to scm diff. 2022-08-11 20:10:45,468-INFO: start: filter issues according to path filter. 2022-08-11 20:10:45,688-INFO: finished: filter issues according to path filter. 2022-08-11 20:10:45,688-INFO: [End] set_filter_type_list (use time: 0.2449052333831787) 2022-08-11 20:10:45,689-INFO: [Start] set_submodule_handle 2022-08-11 20:10:45,689-INFO: [End] set_submodule_handle (use time: 1.7881393432617188e-05) 2022-08-11 20:10:45,689-INFO: [Start] set_blame_type 2022-08-11 20:10:45,713-INFO: Start: lint blame. 2022-08-11 20:10:46,115-INFO: Start to blame ... 2022-08-11 20:10:46,118-INFO: start blame thread. 2022-08-11 20:10:46,118-INFO: blame: /workspace/client/data/sourcedirs/git_6vrq6rn8/src/main/java/org/joychou/controller/CRLFInjection.java 2022-08-11 20:10:46,118-INFO: blame: /workspace/client/data/sourcedirs/git_6vrq6rn8/pom.xml 2022-08-11 20:10:46,119-INFO: blame: /workspace/client/data/sourcedirs/git_6vrq6rn8/src/main/java/org/joychou/controller/Test.java 2022-08-11 20:10:46,119-INFO: blame: /workspace/client/data/sourcedirs/git_6vrq6rn8/src/main/java/org/joychou/controller/Login.java 2022-08-11 20:10:46,121-INFO: blame: /workspace/client/data/sourcedirs/git_6vrq6rn8/src/main/java/org/joychou/controller/Rce.java 2022-08-11 20:10:46,125-INFO: blame: /workspace/client/data/sourcedirs/git_6vrq6rn8/src/main/java/org/joychou/controller/Cors.java 2022-08-11 20:10:46,126-INFO: blame: /workspace/client/data/sourcedirs/git_6vrq6rn8/src/main/java/org/joychou/controller/Deserialize.java 2022-08-11 20:10:46,128-INFO: blame: /workspace/client/data/sourcedirs/git_6vrq6rn8/src/main/java/org/joychou/controller/SQLI.java 2022-08-11 20:10:46,322-INFO: blame: /workspace/client/data/sourcedirs/git_6vrq6rn8/src/main/java/org/joychou/security/LoginFailureHandler.java 2022-08-11 20:10:46,322-INFO: blame: /workspace/client/data/sourcedirs/git_6vrq6rn8/src/main/java/org/joychou/security/ssrf/SocketHookFactory.java 2022-08-11 20:10:46,327-INFO: blame: /workspace/client/data/sourcedirs/git_6vrq6rn8/src/main/java/org/joychou/controller/URLRedirect.java 2022-08-11 20:10:46,336-INFO: blame: /workspace/client/data/sourcedirs/git_6vrq6rn8/src/main/java/org/joychou/controller/Index.java 2022-08-11 20:10:46,338-INFO: blame: /workspace/client/data/sourcedirs/git_6vrq6rn8/src/main/java/org/joychou/controller/IPForge.java 2022-08-11 20:10:46,342-INFO: blame: /workspace/client/data/sourcedirs/git_6vrq6rn8/src/main/java/org/joychou/controller/Jsonp.java 2022-08-11 20:10:46,347-INFO: blame: /workspace/client/data/sourcedirs/git_6vrq6rn8/src/main/java/org/joychou/controller/SpEL.java 2022-08-11 20:10:46,550-INFO: 结束blame，BLAME_ERROR_NUM: 0，blame_num：15 2022-08-11 20:10:46,551-INFO: End: lint blame. 2022-08-11 20:10:46,551-INFO: [End] set_blame_type (use time: 0.8620238304138184) 2022-08-11 20:10:46,551-INFO: [Start] set_filter_type_list 2022-08-11 20:10:46,577-INFO: start: filter issues according to revision. 2022-08-11 20:10:46,578-INFO: __get_revision_status_map... 2022-08-11 20:10:46,578-INFO: __get_revision_status_map done... 2022-08-11 20:10:46,578-INFO: _common_revision_filter done... 2022-08-11 20:10:46,578-INFO: finished: filter issues according to revision. 2022-08-11 20:10:46,579-INFO: [End] set_filter_type_list (use time: 0.02771139144897461) 2022-08-11 20:10:46,579-INFO: [Start] set_issue_ignore_type 2022-08-11 20:10:46,579-INFO: start: igonre issues according to code comments. 2022-08-11 20:10:46,588-INFO: finished: igonre issues according to code comments. 2022-08-11 20:10:46,589-INFO: [End] set_issue_ignore_type (use time: 0.009895801544189453) 2022-08-11 20:10:46,589-INFO: [Start] set_result_pack_diff_info 2022-08-11 20:10:46,589-INFO: [End] set_result_pack_diff_info (use time: 1.52587890625e-05) 2022-08-11 20:10:46,589-INFO: [Start] set_add_file_info 2022-08-11 20:10:46,589-INFO: Start: add file owners... 2022-08-11 20:10:46,591-INFO: End: add file owners. 2022-08-11 20:10:46,591-INFO: Start: add file language type... 2022-08-11 20:10:46,591-INFO: End: add file language type. 2022-08-11 20:10:46,591-INFO: [End] set_add_file_info (use time: 0.0022673606872558594) 2022-08-11 20:10:46,591-INFO: [Start] set_issue_hash 2022-08-11 20:10:46,598-INFO: [End] set_issue_hash (use time: 0.00707697868347168) 2022-08-11 20:10:46,598-INFO: [Start] set_add_person_info 2022-08-11 20:10:46,598-INFO: [End] set_add_person_info (use time: 6.9141387939453125e-06) 2022-08-11 20:10:46,598-INFO: [Start] set_issue_split 2022-08-11 20:10:46,600-INFO: [End] set_issue_split (use time: 0.0018544197082519531) 2022-08-11 20:10:46,600-INFO: datahandle done! 2022-08-11 20:10:46,600-INFO: task done : <class 'task.model.datahandle.DataHandleTask'> 2022-08-11 20:10:46,601-INFO: task result: /workspace/client/data/taskdirs/task_811/task_response.json

cyw3 commented 2 years ago

jaff,jaff_beta,jafc,jafc_beta执行异常，猜测是Docker Compose里面的Client镜像选择有问题，后续这边会调整为Centos。如果可以的话，可以试试直接下载release里面的TCA Client包，使用localscan方式在本地机器或者在centos镜像上执行看看。

可以发下TCA上Semgrep的执行log吗？

看起来不是镜像问题，这边重新测试了下，也没有异常。

可以提供下使用的机器母机系统、CPU架构等信息呢？

2022-08-11 20:10:58,154-INFO: Caused by: java.io.UncheckedIOException: java.nio.file.FileSystemException: /proc/self/map_files: Operation not permitted
2022-08-11 20:10:58,154-INFO: at java.nio.file.Files$2.hasNext(Files.java:3709)
2022-08-11 20:10:58,154-INFO: at java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1811)

类似的issue https://github.com/xerial/sqlite-jdbc/issues/706

Tencent / CodeAnalysis

jaff，jaff_beta,jafc,jafc_beta等三方工具报错 #526

如果需要在ARM架构上运行，可以使用 mariadb:10.5.8 镜像