Workflow: publish Argus, Colossus and QueryNode as docker images

[mnaamani]

Running a storage-node and distributor-node currently requires

• cloning the monorepo
• building packages
• start docker services with local repo mounted as a bind volume

This is not practical for many reasons, and although is great for development cycle it is not a proper production setup. So in this PR we are making the two nodes build-able into proper independent docker images that will get published to docker hub giving operators more flexibility and deployment options in cloud, k8 clusters etc.

• proper docker images
• fixes found when running nodes as docker images.
  • missing direct dependencies
  • TS compiler was not copying openapi spec files into compiled lib/ destination (why is not needed when running from the monorepo?) Is it because of not installing dev dependencies?
• Update to use node18 -> better performance/security fixes
• query-node generated schema.graphql is now placed in the storage-node, distributor-node package so we don't require building query-node ahead of the docker images.

The images can be tested by running integration tests with the docker-compose-no-bind-volumes.yml docker compose file.

Successful run: https://github.com/Joystream/joystream/actions/runs/5437536353

you should be able to pull the images:

docker pull joystream/storage-node:3.4.0
docker pull joystream/distributor-node:1.2.0
docker pull joystream/query-node:1.2.1

I've prepared a standalone repo for running query-node from built docker image. https://github.com/mnaamani/query-node-standalone

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

| Name | Status | Preview | Updated (UTC) | | :--- | :----- | :------ | :------ | | **pioneer-testnet** | ⬜️ Ignored ([Inspect](https://vercel.com/joystream/pioneer-testnet/6GXh1WK6aUaR7WkvNtH7KGmet1gE)) | | Jul 5, 2023 3:57pm |

[mnaamani]

After updating from master, storage-node as docker image is failing:

2023-06-09 12:40:58 2023-06-09 08:40:58:4058 error: Server error: TypeError: Cannot read properties of undefined (reading 'sync')
2023-06-09 12:40:58 error Command failed with exit code 105.

Looking into it.

[mnaamani]

Resolved it by reverting to older verision of mkdirp in storage-node. As there was a conflict with with version used in query-node/codegen. I think we really need to break out the query node into its own repo as it wrecks havock with package dependencies in the yarn worksapce.

[mnaamani]

For some reason in the integration test (content-directory scenario) with the docker images built the CLI is failing to connect to distributor

2023-06-20T06:39:40.309Z integration-tests:job:manage channels and videos through CLI flow 0 failed:
Error: connect ECONNREFUSED 127.0.0.1:3334
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 3334,
  config: {
    url: 'http://127.0.0.1:3334/api/v1/assets/0',
.
.

Tested locally without issues. Something specific to github runner maybe?

[mnaamani]

Tested locally without issues. Something specific to github runner maybe?

Again it was dependency issue on building monorepo before docker images.

 (node:28) [MODULE_NOT_FOUND] Error Plugin: @joystream/distributor-cli: Cannot find module '/joystream/node_modules/@joystream/storage-node-client/lib/index.js'. Please verify that the package.json has a valid "main" entry

Will add build step for the storage-node-client in distributor dockerfile.

[mnaamani]

Looks good. Just one thing, did you see the last comment from #4780 (review)

Shouldn't we also update the compatible node versions in package.json file ...

WDYT?

Sorry I totally missed that. I had actually prepared a separate PR to update for that https://github.com/Joystream/joystream/pull/4778 which I was planning to finish after this.

Thanks for explaining the oclif/ts-node matter about the openapi spec file not being copied :+1:

Regarding checking in the schema.graphql I take your point. I'll revert for cli and integration tests.