Add examples (kubernetes) for Mirrored Storage

[pseymournutanix]

As per https://github.com/buildbarn/bb-adrs/blob/master/0002-storage.md

Setting up configuration with this doesn't seem to match (tbf that doc is a few years old) for example in my common.libsonnet I have

        contentAddressableStorage: {
          sharding: {
            hashInitialization: 11946695773637837490,
            shards: [
              {
                backend: {
                  mirrored: {
                    backendA: { grpc: { address: 'storage-a-0.storage.buildbarn:7982' } },
                    backendB: { grpc: { address: 'storage-a-1.storage.buildbarn:7982' } },
                  },
                },
                weight: 1,
              },
              {
                backend: {
                  mirrored: {
                    backendA: { grpc: { address: 'storage-b-0.storage.buildbarn:7982' } },
                    backendB: { grpc: { address: 'storage-b-1.storage.buildbarn:7982' } },
                  },
                },
                weight: 1,
              },
              {
                backend: {
                  mirrored: {
                    backendA: { grpc: { address: 'storage-c-0.storage.buildbarn:7982' } },
                    backendB: { grpc: { address: 'storage-c-1.storage.buildbarn:7982' } },
                  },
                },
                weight: 1,
              },
            ],
          },
        },

This results in errors in the frontend / browser services as

2023/12/04 17:19:16 Fatal error: rpc error: code = InvalidArgument desc = Failed to create Content Addressable Storage: Replicator configuration not specified

[stagnation]

Hi

The short answer is that there is another field we must set, the blob replicator. See the proto defintion with helpful documentation here, https://github.com/buildbarn/bb-storage/blob/master/pkg/proto/configuration/blobstore/blobstore.proto#L314 . I'll see if I can whip up a little example too.

[stagnation]

You can use a simple replicator, 'local', or use a more advanced option, described in the proto: https://github.com/buildbarn/bb-storage/blob/master/pkg/proto/configuration/blobstore/blobstore.proto#L708

    contentAddressableStorage: {
      sharding: {
        hashInitialization: 11946695773637837490,
        shards: [
          {
            backend: {
              mirrored: {
                backendA: { grpc: { address: 'storage-0:8981' } },
                backendB: { grpc: { address: 'storage-1:8981' } },
                replicatorAToB: { 'local': {} },
                replicatorBToA: { 'local': {} },
              },
            },
            weight: 1,
          },
          {
            backend: {
              mirrored: {
                backendA: { grpc: { address: 'storage-2:8981' } },
                backendB: { grpc: { address: 'storage-3:8981' } },
                replicatorAToB: { 'local': {} },
                replicatorBToA: { 'local': {} },
              },
            },
            weight: 1,
          },
        ],
      },
    },

Patch for the docker-compose deployment

``` commit 0acad45a6a964959eafcdaca8d2ae607eb90b8bb (HEAD) Author: Nils Wireklint Date: Tue Dec 5 10:59:04 2023 +0100 Mirror example diff --git a/docker-compose/config/common.libsonnet b/docker-compose/config/common.libsonnet index 804ee57..64c3327 100644 --- a/docker-compose/config/common.libsonnet +++ b/docker-compose/config/common.libsonnet @@ -5,11 +5,25 @@ hashInitialization: 11946695773637837490, shards: [ { - backend: { grpc: { address: 'storage-0:8981' } }, + backend: { + mirrored: { + backendA: { grpc: { address: 'storage-0:8981' } }, + backendB: { grpc: { address: 'storage-1:8981' } }, + replicatorAToB: { 'local': {} }, + replicatorBToA: { 'local': {} }, + }, + }, weight: 1, }, { - backend: { grpc: { address: 'storage-1:8981' } }, + backend: { + mirrored: { + backendA: { grpc: { address: 'storage-2:8981' } }, + backendB: { grpc: { address: 'storage-3:8981' } }, + replicatorAToB: { 'local': {} }, + replicatorBToA: { 'local': {} }, + }, + }, weight: 1, }, ], @@ -26,7 +40,7 @@ weight: 1, }, { - backend: { grpc: { address: 'storage-1:8981' } }, + backend: { grpc: { address: 'storage-2:8981' } }, weight: 1, }, ], diff --git a/docker-compose/docker-compose.yml b/docker-compose/docker-compose.yml index ba9075c..169297b 100644 --- a/docker-compose/docker-compose.yml +++ b/docker-compose/docker-compose.yml @@ -35,6 +35,30 @@ services: - ./volumes/storage-ac-1:/storage-ac - ./volumes/storage-cas-1:/storage-cas + storage-2: + image: ghcr.io/buildbarn/bb-storage:20231111T202247Z-ece87ab + command: + - /config/storage.jsonnet + expose: + - 8981 + - 9980 + volumes: + - ./config:/config + - ./volumes/storage-ac-2:/storage-ac + - ./volumes/storage-cas-2:/storage-cas + + storage-3: + image: ghcr.io/buildbarn/bb-storage:20231111T202247Z-ece87ab + command: + - /config/storage.jsonnet + expose: + - 8981 + - 9980 + volumes: + - ./config:/config + - ./volumes/storage-ac-3:/storage-ac + - ./volumes/storage-cas-3:/storage-cas + scheduler: image: ghcr.io/buildbarn/bb-scheduler:20231114T140735Z-33a0262 command: ```

[pseymournutanix]

Oh that's great thank you so much. I am a little bad at reading those .proto files and mapping them back to camelcase configuration items ! Now just the error with the remote_asset service but that is for another place.

Appreciate the help. Thank you.