search

aibtcdev / ai-agent-crew

Langchain + CrewAI powered AI agents with Bitcoin wallets.
https://aibtc.dev
28 stars 10 forks source link

Crew: smart contract analysis and fuzzing #5

Open whoabuddy opened 5 months ago

whoabuddy commented 5 months ago

@setzeus mentioned a generative contract analyzer/wrapper on X that kicked off this idea, and @moodmosaic wants a generative contract fuzzer - this issue will be a placeholder to start thinking those things through.

When defining these things it's generally easiest to break them down into individual tasks like you would for a brand new employee, e.g.

  1. Download source code for a given contract principal
  2. Analyze source code and identify X
  3. Analyze source code and identify Y
  4. Analyze source code and identify Z
  5. Use X, Y, Z to do A
  6. Use X, Y, Z to do B
  7. Use X, Y, Z to do C
  8. The final result is A, B, C

With crewAI we'd want to define the agents with their roles/backstories, the tasks the agent will complete including the expected output for each, and the final result we're looking for.

This can be iterative, and if we need a tool like Clarinet then we can enable that through the aibtcdev/agent-tools-ts repo.

moodmosaic commented 4 months ago

Picking up a Clarity contract to discuss how a generative contract fuzzer could work.

Cargo: A Sample Smart Contract

Cargo is a fictional, simple smart contract that runs on Stacks. Its purpose is to facilitate tracking the progress a shipment makes throughout the supply chain. As a smart contract, Cargo has the following properties:

You can find more about Cargo in this article. Cargo is deployed on testnet as ST3QFME3CANQFQNR86TYVKQYCFT7QX4PRXM1V9W6H.cargo, and despite tests, a bug was still found[^1].

Detecting the Unexpected

Testing can decrease the number of defects but not remove all defects[^2]. Mainstream, example-based unit tests verify that a given function works as expected. You aim to go one step further:

Verify that combinations of functions in the smart contract work as expected.

Invariant Testing

Here, I'll show how to uncover a known bug in the Cargo contract using invariant testing—only with the command line. The goal is to automate some, if not all, of these steps using AI.

Scaffolding

  1. Install Clarinet, if not already installed, and scaffold a new project named cargo:

    $ clarinet new cargo

  2. Clarinet won't create a Git repository yet and so you have to do this manually:

    $ cd cargo && git init && git add . && git commit -m "Initial commit"

  3. Add the Cargo contract to Clarinet's deployment plans:

    $ clarinet requirements add ST3QFME3CANQFQNR86TYVKQYCFT7QX4PRXM1V9W6H.cargo

  4. Commit the changes:

    $ git add Clarinet.toml && git commit -m "Add Cargo contract"

Using REPL

Open the REPL to interact with the Cargo contract:

$ clarinet console

Enter "::help" for usage hints.
Connected to a transient in-memory database.
+-------------------------------------------------+-------------------------------------------+
| Contract identifier                             | Public functions                          |
+-------------------------------------------------+-------------------------------------------+
| ST3QFME3CANQFQNR86TYVKQYCFT7QX4PRXM1V9W6H.cargo | (create-new-shipment                      |
|                                                 |     (starting-location (string-ascii 25)) |
|                                                 |     (receiver principal))                 |
|                                                 | (get-shipment (shipment-id uint))         |
|                                                 | (update-shipment                          |
|                                                 |     (shipment-id uint)                    |
|                                                 |     (current-location (string-ascii 25))) |
+-------------------------------------------------+-------------------------------------------+

+-------------------------------------------+-----------------+
| Address                                   | uSTX            |
+-------------------------------------------+-----------------+
| ST1PQHQKV0RJXZFY1DGX8MNSNYVE3VGZJSRTPGZGM | 100000000000000 |
+-------------------------------------------+-----------------+
| ST1SJ3DTE5DN7X54YDH5D64R3BCB6A2AG2ZQ8YPD5 | 100000000000000 |
+-------------------------------------------+-----------------+
| ST2CY5V39NHDPWSXMW9QDT3HC3GD6Q6XX4CFRK9AG | 100000000000000 |
+-------------------------------------------+-----------------+
# 7 more addresses with STX balance         |                 |
+-------------------------------------------+-----------------+

>>

Smart Contract Invariants

A smart contract invariant is a logical statement about the contract's observable, instantaneous state, that can be ensured by its public functions.

In the REPL, you will randomly call public functions on the Cargo contract to build a simplified model. You will then compare that model's state with the actual public, observable, state of the Cargo contract[^3]:

image

The simplified model in this case can be a dictionary of IDs and Shipments. In its empty state it looks like this:

+-----+------------------------------------------------------------------------------------+
| ID  | Shipment                                                                           |
+-----+------------------------------------------------------------------------------------+
|     |                                                                                    |
+-----+------------------------------------------------------------------------------------+

Unknown IDs

u0, u1, u123 are all unknown IDs since they don't exist in the model and in the Cargo contract's state yet. The expected behavior is to have the Cargo contract return an error:

>> (contract-call? 'ST3QFME3CANQFQNR86TYVKQYCFT7QX4PRXM1V9W6H.cargo get-shipment u1)

(status "Does not exist"))

Current state of the simplified model stays the same:

+-----+------------------------------------------------------------------------------------+
| ID  | Shipment                                                                           |
+-----+------------------------------------------------------------------------------------+
|     |                                                                                    |
+-----+------------------------------------------------------------------------------------+

(The state of the simplified model stays the same also for ID u0 and ID u123 and any other integer.)

Create a shipment

With location "Athens" and receiver address ST1SJ3DTE5DN7X54YDH5D64R3BCB6A2AG2ZQ8YPD5 the Cargo contract returns a response indicating success:

>> (contract-call? 'ST3QFME3CANQFQNR86TYVKQYCFT7QX4PRXM1V9W6H.cargo create-new-shipment "Athens" 'ST1SJ3DTE5DN7X54YDH5D64R3BCB6A2AG2ZQ8YPD5)

(ok "Shipment created successfully")

Current state of the simplified model is updated:

+-----+------------------------------------------------------------------------------------+
| ID  | Shipment                                                                           |
+-----+------------------------------------------------------------------------------------+
|  1  | location "Athens"                                                                  |
|     | receiver ST1SJ3DTE5DN7X54YDH5D64R3BCB6A2AG2ZQ8YPD5                                 |
+-----+------------------------------------------------------------------------------------+

Sanity check 1

With ID u1, get-shipment should return a response matching with the entry for ID 1 in current state of the simplified model:

>> (contract-call? 'ST3QFME3CANQFQNR86TYVKQYCFT7QX4PRXM1V9W6H.cargo get-shipment u1)

(tuple (location "Athens") (receiver ST1SJ3DTE5DN7X54YDH5D64R3BCB6A2AG2ZQ8YPD5) (shipper ST1PQHQKV0RJXZFY1DGX8MNSNYVE3VGZJSRTPGZGM) (status "In Transit"))

Create another shipment

With location "Zlin" and receiver address ST2CY5V39NHDPWSXMW9QDT3HC3GD6Q6XX4CFRK9AG the Cargo contract returns a response indicating success:

>> (contract-call? 'ST3QFME3CANQFQNR86TYVKQYCFT7QX4PRXM1V9W6H.cargo create-new-shipment "Zlin" 'ST2CY5V39NHDPWSXMW9QDT3HC3GD6Q6XX4CFRK9AG)

(ok "Shipment created successfully")

Current state of the simplified model is updated:

+-----+------------------------------------------------------------------------------------+
| ID  | Shipment                                                                           |
+-----+------------------------------------------------------------------------------------+
|  1  | location "Athens"                                                                  |
|     | receiver ST1SJ3DTE5DN7X54YDH5D64R3BCB6A2AG2ZQ8YPD5                                 |
+-----+------------------------------------------------------------------------------------+
|  2  | location "Zlin"                                                                  |
|     | receiver ST2CY5V39NHDPWSXMW9QDT3HC3GD6Q6XX4CFRK9AG                                 |
+-----+------------------------------------------------------------------------------------+

Sanity check 2

With ID u2, get-shipment should return a response matching with the entry for ID 2 in current state of the simplified model:

>> (contract-call? 'ST3QFME3CANQFQNR86TYVKQYCFT7QX4PRXM1V9W6H.cargo get-shipment u2)

(status "Does not exist"))
Expected: (tuple (location "Zlin") (receiver ST2CY5V39NHDPWSXMW9QDT3HC3GD6Q6XX4CFRK9AG) (shipper ST1PQHQKV0RJXZFY1DGX8MNSNYVE3VGZJSRTPGZGM) (status "In Transit"))
  Actual: (status "Does not exist")

Sanity check 3

With ID u1, get-shipment should return a response matching with the entry for ID 1 in current state of the simplified model:

>> (contract-call? 'ST3QFME3CANQFQNR86TYVKQYCFT7QX4PRXM1V9W6H.cargo get-shipment u1)

(tuple (location "Zlin") (receiver ST2CY5V39NHDPWSXMW9QDT3HC3GD6Q6XX4CFRK9AG) (shipper ST1PQHQKV0RJXZFY1DGX8MNSNYVE3VGZJSRTPGZGM) (status "In Transit"))
Expected: (tuple (location "Athens") (receiver ST1SJ3DTE5DN7X54YDH5D64R3BCB6A2AG2ZQ8YPD5) (shipper ST1PQHQKV0RJXZFY1DGX8MNSNYVE3VGZJSRTPGZGM) (status "In Transit"))
  Actual: (tuple (location "Zlin") (receiver ST2CY5V39NHDPWSXMW9QDT3HC3GD6Q6XX4CFRK9AG) (shipper ST1PQHQKV0RJXZFY1DGX8MNSNYVE3VGZJSRTPGZGM) (status "In Transit"))

Automation

With sanity checks 2 and 3, you uncovered the bug in the Cargo contract:

[^1]: Reported by LNow and fixed by Kenny Rogers in this commit. [^2]: Program testing can show the presence of bugs, but never their absence.—Dijkstra (1970) "Notes On Structured Programming". [^3]: Image taken and modified from Spotify's article on the same topic.

Next Steps

In my next comment, I will show how to automate these steps using the Clarinet SDK in TypeScript. This will highlight the potential of AI to reduce boilerplate code.