Flaky tests

[alanshaw]

Needs fixing. Using this to log common flakes:

  service › blob › blob/add schedules allocation only on first blob/add

  Value is not truthy:

  false

  › Object.ok (file://test/helpers/assert.js:8:32)
  › blob/add schedules allocation only on first blob/add (file:///home/runner/work/w3infra/w3infra/node_modules/@web3-storage/upload-api/test/handlers/blob.js:148:12)
  › async file://test/service/blob.test.js:29:5

[alanshaw]

put deal on deal tracker for aggregate bafkzcibcaapbcsurqceaadkc7m2ruqt2uj6jabxra7jcmti5ih2xie2gjslwwiy

wait for aggregate/accept receipt bafyreibstdclventh56h6n4akknosgapgis7ji62iqamtf45ysv54rwgo4 ...

  ✖ filecoin › w3filecoin integration flow 

  ─

  filecoin › w3filecoin integration flow

  test/filecoin.test.js:238

   237:         const callDealerCronRes = await pRetry(() => fetch(`https://sta…

   238:         t.true(callDealerCronRes.ok)                                    

   239:                                                                         

  Value is not `true`:

  false

  › file://test/filecoin.test.js:238:15

  › async pRetry.maxTimeout (file://test/helpers/store.js:10:15)

  › async RetryOperation._fn (file://node_modules/p-retry/index.js:57:20)

  ─

  `--fail-fast` is on. 3 test files were skipped.

  1 test failed

🤞 hoping this change solved this one: https://github.com/storacha-network/w3infra/commit/b350e5c85dc8054af1e1ea71d92e317a5d7fdf96#diff-68e8d7c0a05346ab6d9138051a0b0f0c3a54a6f3bfad5996cebf925fe3856f40

[alanshaw]

wait for aggregate/accept receipt bafyreibvlofzk7em7pkpydofcxhgitlevo6uic4gm75tuezxq6bkpvwxpy ...

  ✔ filecoin › w3filecoin integration flow (4m 54.4s)

  ✖ integration › GET / 

  ─

  integration › GET /

  test/integration.test.js:42

   41:   const response = await fetch(t.context.apiEndpoint)

   42:   t.is(response.status, 200)                         

   43: })                                                   

  Difference:

  - 500

  + 200

  › file://test/integration.test.js:42:5

  ─

  `--fail-fast` is on. At least 4 tests were skipped, as well as 2 test files.

  1 test failed

ERROR: Failed to run: npm run test-integration

[alanshaw]

  ✖ metrics-aggregate-offer-total › handles a batch of single invocation with aggregate/offer Rejected promise returned by test
  ─

  metrics-aggregate-offer-total › handles a batch of single invocation with aggregate/offer

  Rejected promise returned by test. Reason:

  Error {
    $metadata: {
      attempts: 3,
      totalRetryDelay: 72,
    },
    address: '::1',
    code: 'ECONNREFUSED',
    errno: -111,
    port: 32844,
    syscall: 'connect',
    message: 'connect ECONNREFUSED ::1:32844',
  }

[alanshaw]

Debugging for service › blob › blob/add schedules allocation only on first blob/add

allocate receipts not equal:
Receipt {
  store: Map(3) {
    'bafyreihzxqroerq7hbizdx2dzfwsuyyaymvsjikqqjn5ewxlaqndckkzpi' => {
      cid: CID(bafyreihzxqroerq7hbizdx2dzfwsuyyaymvsjikqqjn5ewxlaqndckkzpi),
      bytes: [Uint8Array]
    },
    'bafyreigdmoudk2emrru7d75au7uvebtfaymf5akfrt3ysa2wa3ipyw3244' => {
      cid: CID(bafyreigdmoudk2emrru7d75au7uvebtfaymf5akfrt3ysa2wa3ipyw3244),
      bytes: [Uint8Array]
    },
    'bafyreiapylxlogpncxeutw3j37zoqzaqmupvnx6sxzun3g3tqalqbyn4xy' => {
      cid: CID(bafyreiapylxlogpncxeutw3j37zoqzaqmupvnx6sxzun3g3tqalqbyn4xy),
      bytes: [Uint8Array]
    }
  },
  root: {
    cid: CID(bafyreihzxqroerq7hbizdx2dzfwsuyyaymvsjikqqjn5ewxlaqndckkzpi),
    bytes: Uint8Array(824) [
      162,  99, 111,  99, 109, 166,  98, 102, 120, 161, 100, 102,
      111, 114, 107, 128,  99, 105, 115, 115, 120,  25, 100, 105,
      100,  58, 119, 101,  98,  58, 116, 101, 115, 116,  46, 119,
      101,  98,  51,  46, 115, 116, 111, 114,  97, 103, 101,  99,
      111, 117, 116, 161,  98, 111, 107, 162, 100, 115, 105, 122,
      101,   5, 103,  97, 100, 100, 114, 101, 115, 115, 163,  99,
      117, 114, 108, 121,   1, 230, 104, 116, 116, 112,  58,  47,
       47, 108, 111,  99,  97, 108, 104, 111, 115, 116,  58,  51,
       50,  55,  56,  53,
      ... 724 more items
    ],
    data: { ocm: [Object], sig: [Uint8Array] }
  },
  _ran: Invocation {
    root: {
      cid: CID(bafyreigdmoudk2emrru7d75au7uvebtfaymf5akfrt3ysa2wa3ipyw3244),
      bytes: [Uint8Array]
    }
  },
  _fx: undefined,
  _signature: undefined,
  _proofs: undefined,
  _issuer: undefined
}
Receipt {
  store: Map(3) {
    'bafyreig7zvr2tqtpmslz2xdrxd2iqnedvrysbualskrhhmd4kshdlbgmqe' => {
      cid: CID(bafyreig7zvr2tqtpmslz2xdrxd2iqnedvrysbualskrhhmd4kshdlbgmqe),
      bytes: [Uint8Array]
    },
    'bafyreicuxt3bdcugk7mwrlbklh4nbahu4sbvpljuceb2cvs7aorinvsv4u' => {
      cid: CID(bafyreicuxt3bdcugk7mwrlbklh4nbahu4sbvpljuceb2cvs7aorinvsv4u),
      bytes: [Uint8Array]
    },
    'bafyreih7ffktysxx472dodh45dn5hkipntxemymglhol5gqgjkunekb7ji' => {
      cid: CID(bafyreih7ffktysxx472dodh45dn5hkipntxemymglhol5gqgjkunekb7ji),
      bytes: [Uint8Array]
    }
  },
  root: {
    cid: CID(bafyreig7zvr2tqtpmslz2xdrxd2iqnedvrysbualskrhhmd4kshdlbgmqe),
    bytes: Uint8Array(824) [
      162,  99, 111,  99, 109, 166,  98, 102, 120, 161, 100, 102,
      111, 114, 107, 128,  99, 105, 115, 115, 120,  25, 100, 105,
      100,  58, 119, 101,  98,  58, 116, 101, 115, 116,  46, 119,
      101,  98,  51,  46, 115, 116, 111, 114,  97, 103, 101,  99,
      111, 117, 116, 161,  98, 111, 107, 162, 100, 115, 105, 122,
      101,   0, 103,  97, 100, 100, 114, 101, 115, 115, 163,  99,
      117, 114, 108, 121,   1, 230, 104, 116, 116, 112,  58,  47,
       47, 108, 111,  99,  97, 108, 104, 111, 115, 116,  58,  51,
       50,  55,  56,  53,
      ... 724 more items
    ],
    data: { ocm: [Object], sig: [Uint8Array] }
  },
  _ran: Invocation {
    root: {
      cid: CID(bafyreicuxt3bdcugk7mwrlbklh4nbahu4sbvpljuceb2cvs7aorinvsv4u),
      bytes: [Uint8Array]
    }
  },
  _fx: undefined,
  _signature: undefined,
  _proofs: undefined,
  _issuer: undefined
}
  ✖ service › blob › blob/add schedules allocation only on first blob/add 

[Peeja]

This is failing, but seems pretty consistent. Hard to say so far if it's flaky or if something in the world has changed to make it fail every time.

> ava --verbose --serial --timeout=660s --fail-fast test/*.test.js
Creating new File
  ✖ blob › blob integration flow with receipts validation Rejected promise returned by test
  ─
  blob › blob integration flow with receipts validation
  test/blob.test.js:121
   120:           // Given Blob client exported from client would only return m…
   121:           const res = await Blob.add(serviceProps.conf, bytes, { connec…
   122:           t.truthy(res)                                                 
  Rejected promise returned by test. Reason:
  TypeError {
    message: 'Cannot read properties of undefined (reading \'address\')',
  }
  › Module.add (file://test/helpers/blob-client.js:64:48)
  › async Object.transform (file://test/blob.test.js:121:23)
  ─
  `--fail-fast` is on. At least 1 test was skipped, as well as 5 test files.
  1 test failed

[alanshaw]

https://github.com/storacha/w3infra/issues/404#issuecomment-2244903587 is fixed in https://github.com/storacha/upload-service/commit/9deef577b8f79665e9df91d2c13e8747c86219f1