Softdev1 commented 9 months ago

Description

We need to utilize the capabilities of static analysis tools along with LLMs to be more deterministic about iq code detection. explore the possibility of integrating such deterministic approches to detection along with LLMs for iq code

Goal

Research on deterministic vulnerability analysis along with LLMs

Royal-lobster commented 9 months ago

Review on static analysis of smart contracts for prompt injection

\ We are working on creating a fine tuned model on open source code LLMs to detect vulnerabilities in smart contracts. but we are looking to study if there is a possibility we can enhance the LLM output with static analysis tools which are already good at detecting some vulnerabilities.

The idea is LLMs are good at finding logical errors from code comments and how code is written, but there is no guarantee that it always provides correct results. whereas static analysis tools always provide deterministic errors in code, but miss out on logical errors. So we are looking for benefits we can achieve if we combine the both.

Tools we can utilise

Mythril: Utilizes concolic taint and control flow analysis to search for attributes causing vulnerabilities in smart contracts.
Slither: Employs its own internal representation language for intermediate representation and performs data flow and taint analysis.
Smartcheck: Validates contracts against XPAth queries using their XML representation, providing complete code coverage.
Honeybadger: An Oyente-based honeypot detection system that relies on symbolic execution and well-defined heuristics.
Osiris: Based on Oyente, uses symbolic execution coupled with taint analysis for improved detection specificity.
Solhint: A linting tool for Solidity smart contracts using pre-configured patterns and rulesets.
Conkas: Incorporates control flow graphs (CFGs) for symbolic execution, effective in tracing vulnerabilities in library files.
Confuzzius: A hybrid fuzzer that integrates evolutionary fuzzing with constraint solving to explore both shallow and deep contract aspects.

Ways we can use these tools along with LLMs

We can utilize these to generate datasets - we did this with slither analisis dataset and enhanced the output with LLMs
Use it in application level along with the LLM - here we can first let the static analysis run. from the output, we can in theory parse the line numbers and gather the code along with issue and ask LLM to fix the issue. this should scale to codebases since we are not inputing the whole codebase to LLM instead the issues found by static tools - also should be faster

Tool Findings

\ Mythx is not free, unlike the other analysed tools, and it has three different analysis modes: quick, standard and deep. In quick mode, it detected integer overflows and underflows, bad randomness, out of bounds array access and unprotected ether withdrawal but failed to detect transaction order dependency and outdated compiler version. In summary, Mythx was able to detect the 3 more obvious and dangerous vulnerabilities of the contracts that could lead to exploitation. MythX's output is colour-coded by severity and includes SWC ID to link the vulnerabilities detected to the SWC Registry. \

Mythril detected integer overflows and underflows, bad randomness, out of bounds array access and unprotected ether withdrawal, but failed to detect transaction order dependency, outdated compiler version and also failed to warn about solidity coding best practices: deprecated functions, state variables default visibility and floating pragma like MythX. In summary, Mythril was able to detect the 3 more obvious and dangerous vulnerabilities of the contracts that could lead to exploitation.In Mythril output, findings are very detailed, includes SWC ID to link the vulnerabilities detected to the SWC Registry and even includes example transactions and transaction sequences that can trigger the vulnerabilities found. \

Slither does not have detectors for integer overflow and underflow and therefore did not detect these vulnerabilities, but it was able to detect out of bounds array access (as SWC-109), missing input validations, bad randomness and timestamp dependence. In summary, Slither was able to detect the 2 of the 3 more obvious and dangerous vulnerabilities of the contracts that could lead to exploitation. Slither's output is colour-coded by severity, with high-severity detectors printed first,followed by the medium-severity and low-severity outputs and includes references for more information about the vulnerability. In addition, it includes some solidity coding best practices warnings that are useful but do not represent vulnerabilities. We can also use https://github.com/pessimistic-io/slitherin for more detectors on top.

Securify2 This did not work for me, it uses docker image to run but i am getting errors on build command.

Resources