sbrl
commented
2 years ago Ok, I have some more information. This only seems to happen if there's more than 1 backend Consul service.
Closed sbrl closed 1 year ago
sbrl
commented
2 years ago Ok, I have some more information. This only seems to happen if there's more than 1 backend Consul service.
tristanmorgan
commented
2 years ago Could it be two paths pointing to different services? some sort of duplication?
sbrl
commented
2 years ago I'm not sure what you mean, @tristanmorgan?
But what I do know is that if I change count = 1 to count = 2 in a Nomad job spec, then I get this crash in Fabio.
nathanejohnson
commented
2 years ago I'm not sure what you mean, @tristanmorgan?
But what I do know is that if I change
count = 1tocount = 2in a Nomad job spec, then I get this crash in Fabio.
Can you post this job spec? Can you post a little more information about how you're running fabio and consul in general?
sbrl
commented
1 year ago Sure thing, @nathanejohnson. Here's an example job spec:
job "files-public" {
datacenters = ["dc1"]
priority = 35
# Spread allocations over all nodes
spread {
attribute = "${node.unique.name}"
}
group "a" {
count = 2
network {
port "main" { to = 5000 }
}
task "files-public" {
driver = "docker"
config {
image = "myregistry.mydomain.tld:5000/node-serve"
labels { group = "services" }
ports = [ "main" ]
mount {
type = "bind"
target = "/srv"
source = "/mnt/elfstone/Public"
readonly = true
}
}
resources {
# Increased from 25MiB on 2021-05-12 because it ran out of memory serving a 50MB file
memory = 100 # MiB
}
}
service {
name = "${JOB}"
tags = [
"service",
"urlprefix-sub.domain.com/"
]
address_mode = "host"
port = "main"
check {
type = "http"
port = "main"
interval = "60s"
timeout = "30s"
path = "/"
}
}
}
}Simply change count = 2 to count = 1 to fix the bug.
....the Dockerfile for that is really simple:
FROM node:latest
RUN npm install --global serve && rm -rf "$(npm get cache)";
VOLUME [ "/srv" ]
USER 80:80
ENV NODE_ENV production
WORKDIR /srv
ENTRYPOINT [ "serve", "-l", "5000" ]Some details of my setup:
nathanejohnson
commented
1 year ago You getting that stack trace with "index out of range -1" almost seems to me some sort of golang runtime bug on armv7. The only pi I have going right now i a pi4 running on ubuntu 22.04 arm64, so I don't have an easy way to verify this.
This is the line from the stack trace.
For some reason randIntn is returning -1. Ideally this rand method would probably work differently, maybe I will refactor with math.Rand provided it doesn't benchmark significantly slower?
By the way,is your clock set correctly on the Pi?
nathanejohnson
commented
1 year ago One other thing, can you try configuring the proxy.strategy to "rr" and see if the problem goes away?
nathanejohnson
commented
1 year ago So the problem ended up being on 32 bit platforms, the random picker function was overflowing a 32 bit int, causing it to go negative. I ended up benchmarking the homegrown rand picker function against golang's math/rand rand.Intn function, and as it turns out, the math/rand function is 4x faster than the homegrown implementation. According to the comments above the homegrown implementation, at one point math/rand.Intn was fairly slow, but this it appears it's now quite fast. Faster than our broken version at any rate. I have a PR to fix this, I'll look at it more on Monday and merge this in with the next release. In the mean time, setting the proxy.strategy to "rr" should be a workaround.
I'm getting a reproducible crash in Fabio:
Client details
uname -a:Linux DEVICE_NAME 5.13.0-40-generic #45-Ubuntu SMP Tue Mar 29 14:48:14 UTC 2022 x86_64 x86_64 x86_64 GNU/LinuxServer details
uname -a:Linux SERVER_NAME 5.10.103-v7l+ #1529 SMP Tue Mar 8 12:24:00 GMT 2022 armv7l GNU/Linux