demergent-labs / azle

A WebAssembly runtime for TypeScript and JavaScript on ICP
MIT License
209 stars 36 forks source link

Fuzz testing errors to fix #2270

Open lastmjs opened 2 weeks ago

lastmjs commented 2 weeks ago

cuzz --canister-name async_await --call-delay 0

Setting the delay to 0 has led to this error:

2024-11-06 22:02:43.278941743 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] thread '
2024-11-06 22:02:43.278941743 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] <unnamed>
2024-11-06 22:02:43.278941743 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] ' panicked at 
2024-11-06 22:02:43.278941743 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] /home/lastmjs/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rquickjs-core-0.6.2/src/safe_ref.rs
2024-11-06 22:02:43.278941743 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] :
2024-11-06 22:02:43.278941743 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] 35
2024-11-06 22:02:43.278941743 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] :
2024-11-06 22:02:43.278941743 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] 20
2024-11-06 22:02:43.278941743 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] :

2024-11-06 22:02:43.278941743 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] already borrowed: BorrowMutError
2024-11-06 22:02:43.278941743 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] 

2024-11-06 22:02:43.278941743 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
lastmjs commented 2 weeks ago

cuzz --canister-name complex_types --call-delay .1

AgentError: Call failed:
  Canister: bkyz2-fmaaa-aaaaa-qaaaq-cai
  Method: createThread (update)
  "Request ID": "999843d3bc680b329640d3803ff28b37a65fef53b6fca70cd9c5d4fe9c3c1eac"
  "Error code": "IC0502"
  "Reject code": "5"
  "Reject message": "Error from Canister bkyz2-fmaaa-aaaaa-qaaaq-cai: Canister trapped: heap out of bounds.\nConsider gracefully handling failures from this canister or altering the canister to handle exceptions. See documentation: http://internetcomputer.org/docs/current/references/execution-errors#trapped"
    at caller (/home/lastmjs/development/cuzz/node_modules/@dfinity/agent/src/actor.ts:575:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at resultAndMemoryUsage (/home/lastmjs/development/cuzz/index.ts:188:29) {
  canisterId: Principal {
    _arr: Uint8Array(10) [
      128, 0, 0, 0, 0,
       16, 0, 1, 1, 1
    ],
    _isPrincipal: true
  },
  methodName: 'createThread',
  type: 'update',
  props: {
    'Request ID': '999843d3bc680b329640d3803ff28b37a65fef53b6fca70cd9c5d4fe9c3c1eac',
    'Error code': 'IC0502',
    'Reject code': '5',
    'Reject message': 'Error from Canister bkyz2-fmaaa-aaaaa-qaaaq-cai: Canister trapped: heap out of bounds.\n' +
      'Consider gracefully handling failures from this canister or altering the canister to handle exceptions. See documentation: http://internetcomputer.org/docs/current/references/execution-errors#trapped'
  },
  requestId: ArrayBuffer {
    [Uint8Contents]: <99 98 43 d3 bc 68 0b 32 96 40 d3 80 3f f2 8b 37 a6 5f ef 53 b6 fc a7 0c d9 c5 d4 fe 9c 3c 1e ac>,
    byteLength: 32
  },
  response: {
    ok: true,
    status: 200,
    statusText: 'OK',
    body: { status: 'replied', certificate: [Uint8Array] },
    headers: [ [Array], [Array], [Array], [Array] ]
  },
  reject_code: 5,
  reject_message: 'Error from Canister bkyz2-fmaaa-aaaaa-qaaaq-cai: Canister trapped: heap out of bounds.\n' +
    'Consider gracefully handling failures from this canister or altering the canister to handle exceptions. See documentation: http://internetcomputer.org/docs/current/references/execution-errors#trapped',
  error_code: 'IC0502'
}
lastmjs commented 2 weeks ago

canister: bitcoin

error:

  requestId: ArrayBuffer {
    [Uint8Contents]: <92 ab e5 ec b1 dd 3e e5 d9 c6 6a 5b ff 4e 8a d5 3f d7 1d 90 27 db fd 99 47 47 3c 0c d3 b2 4d c0>,
    byteLength: 32
  },
  response: {
    ok: true,
    status: 200,
    statusText: 'OK',
    body: { status: 'replied', certificate: [Uint8Array] },
    headers: [ [Array], [Array], [Array], [Array] ]
  },
  reject_code: 5,
  reject_message: 'Error from Canister bkyz2-fmaaa-aaaaa-qaaaq-cai: Canister called `ic0.trap` with message: Uncaught Error: Failed to decode Candid bytes: InternalError: out of memory\n' +
    '    at idlDecode (.azle/bitcoin/main.js:4693:11)\n' +
    '    at decodeArgs (.azle/bitcoin/main.js:4668:12)\n' +
    '    at executeAndReplyWithCandidSerde (.azle/bitcoin/main.js:4662:23)\n' +
    '    at <anonymous> (.azle/bitcoin/main.js:4907:15)\n' +
    '.\n' +
    'Consider gracefully handling failures from this canister or altering the canister to handle exceptions. See documentation: http://internetcomputer.org/docs/current/references/execution-errors#trapped-explicitly',
  error_code: 'IC0503'
lastmjs commented 2 weeks ago

canister: async_await functional

This is a similar error to the first one above...I wonder if they are the same error? This one looks easier to debug, so I say fix this one first.

  requestId: ArrayBuffer {
    [Uint8Contents]: <fc 7d 64 75 2f b9 db 6a dc 50 cb a4 a4 ee 84 70 80 cf e9 e0 24 38 64 15 6c 46 a4 7a 56 84 80 d5>,
    byteLength: 32
  },
  response: {
    ok: true,
    status: 200,
    statusText: 'OK',
    body: { status: 'replied', certificate: [Uint8Array] },
    headers: [ [Array], [Array], [Array], [Array] ]
  },
  reject_code: 5,
  reject_message: 'Error from Canister bkyz2-fmaaa-aaaaa-qaaaq-cai: Canister called `ic0.trap` with message: Panic occurred: already borrowed: BorrowMutError at src/build/experimental/commands/compile/wasm_binary/rust/experimental_canister_template/src/ic/call_raw.rs:46.\n' +
    'Consider gracefully handling failures from this canister or altering the canister to handle exceptions. See documentation: http://internetcomputer.org/docs/current/references/execution-errors#trapped-explicitly',
  error_code: 'IC0503'
}
lastmjs commented 2 weeks ago

canister: end_to_end/class_syntax/call_raw

CI logs: https://productionresultssa15.blob.core.windows.net/actions-results/0b7da4b4-688c-4782-b430-6d0437889ba0/workflow-job-run-475299c8-7274-504c-4610-92fca0026c29/logs/job/job-logs.txt?rsct=text%2Fplain&se=2024-11-08T16%3A46%3A31Z&sig=1sPw4t%2BU9WcRfeCLZlc0f3GPRPQI2yRSXULwLVsBJGE%3D&ske=2024-11-09T02%3A37%3A15Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2024-11-08T14%3A37%3A15Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2024-08-04&sp=r&spr=https&sr=b&st=2024-11-08T16%3A36%3A26Z&sv=2024-08-04

2024-11-08T06:49:31.8395739Z AgentError: Call failed:
2024-11-08T06:49:31.8396381Z   Canister: bkyz2-fmaaa-aaaaa-qaaaq-cai
2024-11-08T06:49:31.8396772Z   Method: executeCallRaw (update)
2024-11-08T06:49:31.8397314Z   "Request ID": "e996e502bfaa237a75df03850a19d54cd89e8b4c45ef6bbb080ec909c3a524d4"
2024-11-08T06:49:31.8397822Z   "Error code": "IC0502"
2024-11-08T06:49:31.8398091Z   "Reject code": "5"
2024-11-08T06:49:31.8399805Z   "Reject message": "Error from Canister bkyz2-fmaaa-aaaaa-qaaaq-cai: Canister trapped: unreachable.\nConsider gracefully handling failures from this canister or altering the canister to handle exceptions. See documentation: http://internetcomputer.org/docs/current/references/execution-errors#trapped"
2024-11-08T06:49:31.8401792Z     at caller (/home/runner/work/azle/azle/node_modules/cuzz/node_modules/@dfinity/agent/src/actor.ts:575:19)
2024-11-08T06:49:31.8402627Z     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2024-11-08T06:49:31.8403633Z     at resultAndMemoryUsage (/home/runner/work/azle/azle/node_modules/cuzz/index.ts:210:29) {
2024-11-08T06:49:31.8404199Z   canisterId: Principal {
2024-11-08T06:49:31.8404471Z     _arr: Uint8Array(10) [
2024-11-08T06:49:31.8404737Z       128, 0, 0, 0, 0,
2024-11-08T06:49:31.8404972Z        16, 0, 1, 1, 1
2024-11-08T06:49:31.8405190Z     ],
2024-11-08T06:49:31.8405394Z     _isPrincipal: true
2024-11-08T06:49:31.8405624Z   },
2024-11-08T06:49:31.8405905Z   methodName: 'executeCallRaw',
2024-11-08T06:49:31.8406219Z   type: 'update',
2024-11-08T06:49:31.8406447Z   props: {
2024-11-08T06:49:31.8406968Z     'Request ID': 'e996e502bfaa237a75df03850a19d54cd89e8b4c45ef6bbb080ec909c3a524d4',
2024-11-08T06:49:31.8407499Z     'Error code': 'IC0502',
2024-11-08T06:49:31.8407911Z     'Reject code': '5',
2024-11-08T06:49:31.8408843Z     'Reject message': 'Error from Canister bkyz2-fmaaa-aaaaa-qaaaq-cai: Canister trapped: unreachable.\n' +
2024-11-08T06:49:31.8410438Z       'Consider gracefully handling failures from this canister or altering the canister to handle exceptions. See documentation: http://internetcomputer.org/docs/current/references/execution-errors#trapped'
2024-11-08T06:49:31.8411433Z   },
2024-11-08T06:49:31.8411653Z   requestId: ArrayBuffer {
2024-11-08T06:49:31.8412254Z     [Uint8Contents]: <e9 96 e5 02 bf aa 23 7a 75 df 03 85 0a 19 d5 4c d8 9e 8b 4c 45 ef 6b bb 08 0e c9 09 c3 a5 24 d4>,
2024-11-08T06:49:31.8412830Z     byteLength: 32
2024-11-08T06:49:31.8413053Z   },
2024-11-08T06:49:31.8413240Z   response: {
2024-11-08T06:49:31.8413459Z     ok: true,
2024-11-08T06:49:31.8413960Z     status: 200,
2024-11-08T06:49:31.8414250Z     statusText: 'OK',
2024-11-08T06:49:31.8414667Z     body: { status: 'replied', certificate: [Uint8Array] },
2024-11-08T06:49:31.8415130Z     headers: [ [Array], [Array], [Array], [Array] ]
2024-11-08T06:49:31.8415464Z   },
2024-11-08T06:49:31.8415661Z   reject_code: 5,
2024-11-08T06:49:31.8416296Z   reject_message: 'Error from Canister bkyz2-fmaaa-aaaaa-qaaaq-cai: Canister trapped: unreachable.\n' +
2024-11-08T06:49:31.8417794Z     'Consider gracefully handling failures from this canister or altering the canister to handle exceptions. See documentation: http://internetcomputer.org/docs/current/references/execution-errors#trapped',
2024-11-08T06:49:31.8418967Z   error_code: 'IC0502'
lastmjs commented 2 weeks ago

Strings of 500_000 size or greater might be crashing the replica

lastmjs commented 2 weeks ago

canister: call_raw class based stable

CI logs: https://productionresultssa17.blob.core.windows.net/actions-results/2678d99f-f87f-41d5-bd2b-6a67e97b4f3b/workflow-job-run-475299c8-7274-504c-4610-92fca0026c29/logs/job/job-logs.txt?rsct=text%2Fplain&se=2024-11-12T04%3A52%3A40Z&sig=bD1vIxUG9HPwOk4J5nngxUilyjJfFdJawpMgIIWq5T4%3D&ske=2024-11-12T14%3A38%3A42Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2024-11-12T02%3A38%3A42Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2024-08-04&sp=r&spr=https&sr=b&st=2024-11-12T04%3A42%3A35Z&sv=2024-08-04

2024-11-11T23:54:23.2984050Z AgentError: Call failed:
2024-11-11T23:54:23.2985457Z   Canister: bkyz2-fmaaa-aaaaa-qaaaq-cai
2024-11-11T23:54:23.2987195Z   Method: executeCallRaw (update)
2024-11-11T23:54:23.2988219Z   "Request ID": "b657ee6c602460a6effbeda04616739e83994ad5daeffe9812b7aea26185d24e"
2024-11-11T23:54:23.2989649Z   "Error code": "IC0502"
2024-11-11T23:54:23.2990190Z   "Reject code": "5"
2024-11-11T23:54:23.2993297Z   "Reject message": "Error from Canister bkyz2-fmaaa-aaaaa-qaaaq-cai: Canister trapped: unreachable.\nConsider gracefully handling failures from this canister or altering the canister to handle exceptions. See documentation: http://internetcomputer.org/docs/current/references/execution-errors#trapped"
2024-11-11T23:54:23.2996386Z     at caller (/home/runner/work/azle/azle/node_modules/cuzz/node_modules/@dfinity/agent/src/actor.ts:575:19)
2024-11-11T23:54:23.2998040Z     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2024-11-11T23:54:23.2999382Z     at resultAndMemoryUsage (/home/runner/work/azle/azle/node_modules/cuzz/index.ts:210:29) {
2024-11-11T23:54:23.3000339Z   canisterId: Principal {
2024-11-11T23:54:23.3000786Z     _arr: Uint8Array(10) [
2024-11-11T23:54:23.3001216Z       128, 0, 0, 0, 0,
2024-11-11T23:54:23.3001622Z        16, 0, 1, 1, 1
2024-11-11T23:54:23.3001995Z     ],
2024-11-11T23:54:23.3002340Z     _isPrincipal: true
2024-11-11T23:54:23.3002716Z   },
2024-11-11T23:54:23.3003280Z   methodName: 'executeCallRaw',
2024-11-11T23:54:23.3003810Z   type: 'update',
2024-11-11T23:54:23.3004182Z   props: {
2024-11-11T23:54:23.3005080Z     'Request ID': 'b657ee6c602460a6effbeda04616739e83994ad5daeffe9812b7aea26185d24e',
2024-11-11T23:54:23.3006017Z     'Error code': 'IC0502',
2024-11-11T23:54:23.3006509Z     'Reject code': '5',
2024-11-11T23:54:23.3007687Z     'Reject message': 'Error from Canister bkyz2-fmaaa-aaaaa-qaaaq-cai: Canister trapped: unreachable.\n' +
2024-11-11T23:54:23.3010776Z       'Consider gracefully handling failures from this canister or altering the canister to handle exceptions. See documentation: http://internetcomputer.org/docs/current/references/execution-errors#trapped'
2024-11-11T23:54:23.3012595Z   },
2024-11-11T23:54:23.3012925Z   requestId: ArrayBuffer {
2024-11-11T23:54:23.3013997Z     [Uint8Contents]: <b6 57 ee 6c 60 24 60 a6 ef fb ed a0 46 16 73 9e 83 99 4a d5 da ef fe 98 12 b7 ae a2 61 85 d2 4e>,
2024-11-11T23:54:23.3015027Z     byteLength: 32
2024-11-11T23:54:23.3015376Z   },
2024-11-11T23:54:23.3015687Z   response: {
2024-11-11T23:54:23.3016035Z     ok: true,
2024-11-11T23:54:23.3016370Z     status: 200,
2024-11-11T23:54:23.3016809Z     statusText: 'OK',
2024-11-11T23:54:23.3017498Z     body: { status: 'replied', certificate: [Uint8Array] },
2024-11-11T23:54:23.3018271Z     headers: [ [Array], [Array], [Array], [Array] ]
2024-11-11T23:54:23.3018845Z   },
2024-11-11T23:54:23.3019163Z   reject_code: 5,
2024-11-11T23:54:23.3020277Z   reject_message: 'Error from Canister bkyz2-fmaaa-aaaaa-qaaaq-cai: Canister trapped: unreachable.\n' +
2024-11-11T23:54:23.3023053Z     'Consider gracefully handling failures from this canister or altering the canister to handle exceptions. See documentation: http://internetcomputer.org/docs/current/references/execution-errors#trapped',
2024-11-11T23:54:23.3024947Z   error_code: 'IC0502'
2024-11-11T23:54:23.3025551Z }
2024-11-11T23:54:23.3509755Z   ●  process.exit called with "1"
lastmjs commented 1 week ago

Resolution: This one seems to just be a stack overflow that is reported as a heap out of bounds

canister: candid_rpc/class_syntax/motoko_examples/factorial

CI Logs

AgentError: Call failed:
  Canister: bkyz2-fmaaa-aaaaa-qaaaq-cai
  Method: fac (update)
  "Request ID": "c15e4a47efd1c26f54b514d0fd69b89e4f4c8a594318ae6f7b2cb7d0ec5e28d9"
  "Error code": "IC0502"
  "Reject code": "5"
  "Reject message": "Error from Canister bkyz2-fmaaa-aaaaa-qaaaq-cai: Canister trapped: heap out of bounds.\nConsider gracefully handling failures from this canister or altering the canister to handle exceptions. See documentation: http://internetcomputer.org/docs/current/references/execution-errors#trapped"
    at caller (/home/runner/work/azle/azle/node_modules/cuzz/node_modules/@dfinity/agent/src/actor.ts:575:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at resultAndMemoryUsage (/home/runner/work/azle/azle/node_modules/cuzz/index.ts:[247](https://github.com/demergent-labs/azle/actions/runs/11920994800/job/33224461609#step:14:248):29) {
  canisterId: Principal {
    _arr: Uint8Array(10) [
      128, 0, 0, 0, 0,
       16, 0, 1, 1, 1
    ],
    _isPrincipal: true
  },
  methodName: 'fac',
  type: 'update',
  props: {
    'Request ID': 'c15e4a47efd1c26f54b514d0fd69b89e4f4c8a594318ae6f7b2cb7d0ec5e28d9',
    'Error code': 'IC0502',
    'Reject code': '5',
    'Reject message': 'Error from Canister bkyz2-fmaaa-aaaaa-qaaaq-cai: Canister trapped: heap out of bounds.\n' +
      'Consider gracefully handling failures from this canister or altering the canister to handle exceptions. See documentation: http://internetcomputer.org/docs/current/references/execution-errors#trapped'
  },
  requestId: ArrayBuffer {
    [Uint8Contents]: <c1 5e 4a 47 ef d1 c2 6f 54 b5 14 d0 fd 69 b8 9e 4f 4c 8a 59 43 18 ae 6f 7b 2c b7 d0 ec 5e 28 d9>,
    byteLength: 32
  },
  response: {
    ok: true,
    status: 200,
    statusText: 'OK',
    body: { status: 'replied', certificate: [Uint8Array] },
    headers: [ [Array], [Array], [Array], [Array] ]
  },
  reject_code: 5,
  reject_message: 'Error from Canister bkyz2-fmaaa-aaaaa-qaaaq-cai: Canister trapped: heap out of bounds.\n' +
    'Consider gracefully handling failures from this canister or altering the canister to handle exceptions. See documentation: http://internetcomputer.org/docs/current/references/execution-errors#trapped',
  error_code: 'IC0502'
}