Nim-Waku Clusters

[decanus]

Stable Cluster

This will be the first cluster, we will release to this cluster whenever a new tag is pushed indicating a stable release of WakuV2. This cluster should contain 3 nodes in geographically different regions so it is similar to the production clusters of what status is currently running.

Testing Cluster

This cluster will be trailing the master branch. It is the less stable testing cluster of WakuV2. The geographical location of this cluster is probably not as important as in the stable cluster, but there should also be multiple nodes.

[decanus]

cc @oskarth @jakubgs

[jakubgs]

Questions:

• Should the testing cluster be updated after every commit pushed to master? Or on a hourly/daily interval?
• Do these nodes require any specific configuration to isolate them from one another? Or is it fine if they find each other?
• Are there any special hardware requirements for these clusters, or is same as already existing hosts fine?

[decanus]

@jakubgs hourly or daily works, but I think commits to master may be fewer so pushes could be better?

The nodes of each network should find eachother but networks should not overlap. There are no specific hardware requirements

[jakubgs]

What does "networks should not overlap" mean?

[decanus]

Stable and Testing should not peer.

[jakubgs]

Do we have some built-in way of preventing that? Or should it be done via firewall?

[decanus]

I guess firewall, maybe @oskarth or @kdeme know of some built in method.

[jakubgs]

I did some cleanup today and renamed old fleet to wakuv1.test to match the schema: e19fa054

Also in the process I cleaned up a few smaller issues:

• https://github.com/status-im/infra-role-bootstrap/commit/be545088 - Duplicate fstype when binding volume
• https://github.com/status-im/infra-role-bootstrap/commit/6455dc9b - Missing permissions for Docker config
• https://github.com/status-im/infra-role-bootstrap/commit/eaf80936 - Volume binding bug when no volume exists
• https://github.com/status-im/infra-nim-waku/commit/cead2f15 - Broken JSON templateing due to jinja2_native=true

[jakubgs]

I deployed a 3 host wakuv2.test fleet in https://github.com/status-im/infra-nim-waku/commit/367fa88d.\

 > ansible localhost -m debug -a 'var=groups["wakuv2"]'
localhost | SUCCESS => {
    "groups[\"wakuv2\"]": [
        "node-01.ac-cn-hongkong-c.wakuv2.test",
        "node-01.do-ams3.wakuv2.test",
        "node-01.gc-us-central1-a.wakuv2.test"
    ]
}

[jakubgs]

The do-ams3 and ac-cn-hongkong-c nodes are up, but for some reason gc-us-central1-a is having issues:

admin@node-01.gc-us-central1-a.wakuv2.test:~ % d ps -a
CONTAINER ID        NAMES               IMAGE                           CREATED             STATUS
f6c9e1e91636        nim-waku-v2         statusteam/nim-waku:wakunode2   5 minutes ago       Exited (132) About a minute ago

Seems to be related to native compilation and lack of some opcodes:

• https://github.com/openresty/docker-openresty/issues/39
• https://github.com/Kong/docker-kong/issues/138
• https://medium.com/@nprch_12/docker-exited-132-e38f9dd2cd0d

[jakubgs]

Added a Jenkinsfile in https://github.com/status-im/nim-waku/pull/302 so we can auto-update the wakuv2.test fleet.

[jakubgs]

Okay, I built the image with -d:disableMarchNative but it's still failing with 132.

@decanus am I using a wrong flag to disable native compilation flags?

[decanus]

ping @kdeme or @oskarth

[oskarth]

No idea, never seen this tbh. @zah maybe?

[zah]

@jakubgs, how have you tried to disable it?

As far as I can tell, -d:disableMarchNative affects the config file here:

https://github.com/status-im/nim-waku/blob/f38bdf290088be7fd8381615eaee611e42df5c6f/config.nims#L26-L33

If you use the Dockerfile from this repo, you are supposed to either add that flag here: https://github.com/status-im/nim-waku/blob/f38bdf290088be7fd8381615eaee611e42df5c6f/Dockerfile#L17

... or pass it to the docker build with NIM_PARAMS.

[jakubgs]

Yeah, and that's what I'm doing by adding -d:disableMarchNative to parameter NIM_PARAMS: https://github.com/status-im/nim-waku/blob/35f9e52d499f7818f21e92b7d7dc708b8da4529f/Jenkinsfile#L21-L22 And then using it in the docker build: https://github.com/status-im/nim-waku/blob/35f9e52d499f7818f21e92b7d7dc708b8da4529f/Jenkinsfile#L39-L42 And based on the Dockerfile they should be used: https://github.com/status-im/nim-waku/blob/70e19435c3e26b1e4513b4c058c86958ec11df68/Dockerfile#L20-L21 And based on build logs it does seem to be used when calling docker build:

13:13:26  + docker build -t statusteam/nim-waku:35f9e5 \
                 --build-arg=MAKE_TARGET=wakunode2 \
                 --build-arg=NIM_PARAMS=-d:disableMarchNative -d:chronicles_colors:none -d:insecure \
                 .

As well as when make is called:

13:18:58  Step 9/21 : RUN make -j$(nproc) $MAKE_TARGET NIM_PARAMS="$NIM_PARAMS"
13:18:58   ---> Running in 040da4b08a22
13:18:59  Building: build/wakunode2
13:18:59  Hint: used config file '/app/vendor/nimbus-build-system/vendor/Nim/config/nim.cfg' [Conf]
13:18:59  Hint: used config file '/app/vendor/nimbus-build-system/vendor/Nim/config/config.nims' [Conf]
13:18:59  Hint: used config file '/app/config.nims' [Conf]
13:18:59  [NimScript] exec: nim c --out:build/wakunode2 -d:chronicles_log_level=TRACE -d:disableMarchNative -d:chronicles_colors:none -d:insecure waku/v2/node/wakunode2.nim

https://ci.status.im/job/nim-waku/job/master/15/console

So not sure what's missing.

[jakubgs]

Ok wait, I think it did work. I re-created the container on node-01.gc-us-central1-a.wakuv2.test by hand and it looks like it's up:

admin@node-01.gc-us-central1-a.wakuv2.test:~ % d
CONTAINER ID        NAMES               IMAGE                           CREATED              STATUS
2fca95e2a90e        nim-waku-v2         statusteam/nim-waku:wakunode2   About a minute ago   Up 58 seconds

It seems like -d:disableMarchNative did help. The image just wasn't propagated properly. Thanks @zah for clarifying.

[jakubgs]

In order to re-arrange the layout to make use of the Terraform workspaces I had to do a bunch of changes to TF modules:

• https://github.com/status-im/infra-tf-alibaba-cloud/commit/db2434b0
• https://github.com/status-im/infra-tf-google-cloud/commit/79602fbf
• https://github.com/status-im/infra-tf-digital-ocean/commit/d5d9884d
• https://github.com/status-im/infra-tf-multi-provider/commit/13321b1a

With this I was able to move wakuv1.test fleet into a v1 workspace and wakuv2.test into test workspace: f7331d6a760369294254aa9cad1bd69ec007a94a

 > terraform workspace list
  default
* v1
  test

Now I can just do terraform workspace new prod and terraform apply to get a wakuv2.prod fleet.

[jakubgs]

And added the wakuv2.prod fleet: https://github.com/status-im/infra-nim-waku/commit/f3f56dee

[jakubgs]

Adjustment to Jenkinsfile to allow definition of multiple different builds: https://github.com/status-im/nim-waku/pull/310

[jakubgs]

Since https://github.com/status-im/nim-waku/pull/310 was merged I've updated branches in https://ci.status.im/job/nim-waku/ jobs to master.

I consider this done. If there's anything you want adjusted/changed please reopen this.