search

hyperledger-cacti / cacti

Hyperledger Cacti is a new approach to the blockchain interoperability problem
https://wiki.hyperledger.org/display/cactus
Apache License 2.0
340 stars 281 forks source link

fix(ci): tests are failing after recent package bumps #2807

Closed outSH closed 11 months ago

outSH commented 1 year ago

@petermetz @izuru0 @jagpreetsinghsasan

Describe the bug

After recent dependency bumps many tests are failing in CI - see https://github.com/hyperledger/cacti/pull/2805. I confirmed some of these errors on main branch (run locally).

TLDR I'd propose to rollback the following commits and investigate how to bump these packages without breaking the CI:

As for now I've identified the following issues:

iroha-helpers

Cannot find module 'grpc' from 'packages/cactus-plugin-ledger-connector-iroha/node_modules/iroha-helpers/lib/proto/endpoint_grpc_pb.js'

    Require stack:
      packages/cactus-plugin-ledger-connector-iroha/node_modules/iroha-helpers/lib/proto/endpoint_grpc_pb.js
      packages/cactus-plugin-ledger-connector-iroha/dist/lib/main/typescript/iroha-transaction-wrapper.js
      packages/cactus-plugin-ledger-connector-iroha/dist/lib/main/typescript/plugin-ledger-connector-iroha.js
      packages/cactus-plugin-ledger-connector-iroha/dist/lib/main/typescript/public-api.js
      packages/cactus-plugin-ledger-connector-iroha/dist/lib/main/typescript/index.js
      packages/cactus-verifier-client/dist/lib/main/typescript/get-validator-api-client.js
      packages/cactus-verifier-client/dist/lib/main/typescript/public-api.js
      packages/cactus-verifier-client/dist/lib/main/typescript/index.js
      packages/cactus-test-plugin-ledger-connector-ethereum/src/test/typescript/integration/api-client/verifier-integration-with-ethereum-connector.test.ts

      21 | } from "iroha-helpers/lib/proto/primitive_pb";
      22 |
    > 23 | import { CommandService_v1Client as CommandService } from "iroha-helpers/lib/proto/endpoint_grpc_pb";
         | ^
      24 | import { QueryService_v1Client as QueryService } from "iroha-helpers/lib/proto/endpoint_grpc_pb";
      25 |
      26 | import commands from "iroha-helpers/lib/commands/index";

      at Resolver._throwModNotFoundError (node_modules/jest-resolve/build/resolver.js:427:11)
      at Object.<anonymous> (packages/cactus-plugin-ledger-connector-iroha/node_modules/iroha-helpers/lib/proto/endpoint_grpc_pb.js:9:12)
      at Object.<anonymous> (packages/cactus-plugin-ledger-connector-iroha/src/main/typescript/iroha-transaction-wrapper.ts:23:1)
      at Object.<anonymous> (packages/cactus-plugin-ledger-connector-iroha/src/main/typescript/plugin-ledger-connector-iroha.ts:7:1)
      at Object.<anonymous> (packages/cactus-plugin-ledger-connector-iroha/src/main/typescript/public-api.ts:1:1)
      at Object.<anonymous> (packages/cactus-plugin-ledger-connector-iroha/src/main/typescript/index.ts:1:1)
      at Object.<anonymous> (packages/cactus-verifier-client/src/main/typescript/get-validator-api-client.ts:33:1)
      at Object.<anonymous> (packages/cactus-verifier-client/src/main/typescript/public-api.ts:2:1)
      at Object.<anonymous> (packages/cactus-verifier-client/src/main/typescript/index.ts:1:1)
      at Object.<anonymous> (packages/cactus-test-plugin-ledger-connector-ethereum/src/test/typescript/integration/api-client/verifier-integration-with-ethereum-connector.test.ts:42:1)

This is caused by invalid dependency in iroha-helpers package - after we upgraded to grpc-js it can't find grpc. I've opened an issue in their repository and proposed to add grpc dependency.

Axios

    Cannot find module 'axios/lib/defaults' from 'packages/cactus-test-tooling/node_modules/axios-cookiejar-support/lib/index.js'

    Require stack:
      packages/cactus-test-tooling/node_modules/axios-cookiejar-support/lib/index.js
      packages/cactus-test-tooling/node_modules/nano/lib/nano.js
      packages/cactus-test-tooling/node_modules/fabric-network/lib/impl/wallet/couchdbwalletstore.js
      packages/cactus-test-tooling/node_modules/fabric-network/lib/impl/wallet/wallets.js
      packages/cactus-test-tooling/node_modules/fabric-network/index.js
      packages/cactus-test-tooling/dist/lib/main/typescript/fabric/fabric-test-ledger-v1.js
      packages/cactus-test-tooling/dist/lib/main/typescript/public-api.js
      packages/cactus-test-tooling/dist/lib/main/typescript/index.js
      packages/cactus-plugin-ledger-connector-iroha2/src/test/typescript/test-helpers/iroha2-env-setup.ts
      packages/cactus-plugin-ledger-connector-iroha2/src/test/typescript/integration/iroha2-setup-and-basic-operations.test.ts

      10 | } from "dockerode";
      11 |
    > 12 | import { Wallets, Gateway, Wallet, X509Identity } from "fabric-network";
         | ^
      13 | import FabricCAServices from "fabric-ca-client";
      14 |
      15 | import Joi from "joi";

      at Resolver._throwModNotFoundError (node_modules/jest-resolve/build/resolver.js:427:11)
      at Object.<anonymous> (packages/cactus-test-tooling/node_modules/axios-cookiejar-support/lib/index.js:8:40)
      at Object.<anonymous> (packages/cactus-test-tooling/node_modules/nano/lib/nano.js:17:31)
      at Object.<anonymous> (packages/cactus-test-tooling/node_modules/fabric-network/src/impl/wallet/couchdbwalletstore.ts:7:1)
      at Object.<anonymous> (packages/cactus-test-tooling/node_modules/fabric-network/src/impl/wallet/wallets.ts:7:1)
      at Object.<anonymous> (packages/cactus-test-tooling/node_modules/fabric-network/index.js:283:26)
      at Object.<anonymous> (packages/cactus-test-tooling/src/main/typescript/fabric/fabric-test-ledger-v1.ts:12:1)
      at Object.<anonymous> (packages/cactus-test-tooling/src/main/typescript/public-api.ts:51:1)
      at Object.<anonymous> (packages/cactus-test-tooling/src/main/typescript/index.ts:1:1)
      at Object.<anonymous> (packages/cactus-plugin-ledger-connector-iroha2/src/test/typescript/test-helpers/iroha2-env-setup.ts:14:1)
      at Object.<anonymous> (packages/cactus-plugin-ledger-connector-iroha2/src/test/typescript/integration/iroha2-setup-and-basic-operations.test.ts:32:1)

I think this one is caused by axios-cookiejar-support that lists axios as peer depdendency "axios": ">=0.16.2", (so matches 1. we use) but it should list only 0. releases. I'll open an issue in their repo once I confirm this.

Axios 2

Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './lib/defaults' is not defined by "exports" in /home/runner/work/cacti/cacti/packages/cactus-test-tooling/node_modules/axios/package.json
    at new NodeError (node:internal/errors:371:5)
    at throwExportsNotFound (node:internal/modules/esm/resolve:453:9)
    at packageExportsResolve (node:internal/modules/esm/resolve:731:3)
    at resolveExports (node:internal/modules/cjs/loader:482:36)
    at Function.Module._findPath (node:internal/modules/cjs/loader:522:31)
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:919:27)
    at Function.Module._resolveFilename.sharedData.moduleResolveFilenameHook.installedValue [as _resolveFilename] (/home/runner/work/cacti/cacti/node_modules/@cspotcode/source-map-support/source-map-support.js:811:30)
    at Function.Module._load (node:internal/modules/cjs/loader:778:27)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/home/runner/work/cacti/cacti/packages/cactus-test-tooling/node_modules/axios-cookiejar-support/lib/index.js:8:40)
    at Module._compile (node:internal/modules/cjs/loader:1103:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
    at Object.require.extensions.<computed> [as .js] (/home/runner/work/cacti/cacti/node_modules/ts-node/src/index.ts:1608:43)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12) {
  code: 'ERR_PACKAGE_PATH_NOT_EXPORTED'
}

ipfs-http-client

    Cannot find module 'ipfs-http-client' from 'extensions/cactus-plugin-object-store-ipfs/src/main/typescript/plugin-object-store-ipfs.ts'

    Require stack:
      extensions/cactus-plugin-object-store-ipfs/src/main/typescript/plugin-object-store-ipfs.ts
      extensions/cactus-plugin-object-store-ipfs/src/main/typescript/public-api.ts
      extensions/cactus-plugin-object-store-ipfs/src/test/typescript/unit/api-surface.test.ts

      1 | import path from "path";
      2 | import type { Express } from "express";
    > 3 | import { create, IPFSHTTPClient } from "ipfs-http-client";
        | ^
      4 | import type { Options } from "ipfs-http-client";
      5 | import { RuntimeError } from "run-time-error";
      6 | import { Logger, Checks, LoggerProvider } from "@hyperledger/cactus-common";

      at Resolver._throwModNotFoundError (node_modules/jest-resolve/build/resolver.js:427:11)
      at Object.<anonymous> (extensions/cactus-plugin-object-store-ipfs/src/main/typescript/plugin-object-store-ipfs.ts:3:1)
      at Object.<anonymous> (extensions/cactus-plugin-object-store-ipfs/src/main/typescript/public-api.ts:4:1)
      at Object.<anonymous> (extensions/cactus-plugin-object-store-ipfs/src/test/typescript/unit/api-surface.test.ts:1:1)
petermetz commented 1 year ago

image

petermetz commented 1 year ago

@outSH Yeah, sorry, this is on me, going fast and breaking things (as much as I don't like to do that). Deadlines are a bit tight right now and that's been a forcing function.

Below is my step by step update on each of the issues you mentioned:

For the iroha-helpers issue: old grpc is so bad that if Iroha v1 keeps depending on it in my opinion we should just deprecate the Iroha 1 connector. With that said, I'd guess there's a good chance if we update your PR to use the new pure JS grpc package instead of the legacy grpc, it will just work and then Iroha v1 is no longer an issue.

For the axios problems: I'll submit another PR with fine-tuning my earlier changes (I figured out a way to not have the vulnerable versions and yet keep the tests functional)

ipfs-http-client: This one is the hardest. This package is also entirely deprecated and won't receive security updates (which means we shouldn't be using it at all for production code) but I haven't yet been able to find a quick and dirty solution to fix this. The only solution right now seems to be to do a full migration over to this new helia package that they point to in their readme (but that seems like a much bigger chunk of work and I'm really tight on time right now). I'm looking into options here, any help is most welcome.

outSH commented 1 year ago

@petermetz

ipfs-http-client: This one is the hardest. This package is also entirely deprecated and won't receive security updates

I think the replacement for it is https://github.com/ipfs/js-kubo-rpc-client (for HTTP API), but it claims to be Work In Progress. They didn't commit any significant change this year so maybe they are done, but I'm not sure.

Either way I didn't find any other alternative, so it seems we are between a rock and a hard place on ths one :S

outSH commented 12 months ago

@petermetz BTW If you're OK with using https://github.com/ipfs/js-kubo-rpc-client I can migrate to it

petermetz commented 12 months ago

@petermetz BTW If you're OK with using https://github.com/ipfs/js-kubo-rpc-client I can migrate to it

@outSH I'm 100% OK with that, whatever that actually works and isn't vulnerable is still better than something that is known to never again receive security updates. So the kubo RPC client is a step forward IMO even if it's right now unstable.

On the topic of the axios fix: I just submitted a PR that should take care of that problem. Between that and your kubo migration we just down to the grpc issue.

outSH commented 12 months ago

@petermetz Hi, I've prepared a draft with some comments for fixing IPFS issues, have a look - #2829