akerouanton
commented
2 years ago Hello @duktig-dev, could you try to share a minimal example of a Golang app and a Dockerfile that reproduces this issue please?
Closed duktig-dev closed 2 years ago
akerouanton
commented
2 years ago Hello @duktig-dev, could you try to share a minimal example of a Golang app and a Dockerfile that reproduces this issue please?
duktig-dev
commented
2 years ago Sure!
Dockerfile
FROM golang:1.17
WORKDIR /app
COPY ./src/go.mod ./
COPY ./src/go.sum ./
COPY ./src /app
RUN go mod download
# Exampe template pf build command:
# go build -o {output-bin-file} {source-files-path}
RUN go build -o /events-go-publisher /app/cmd/
# RUN go get -d -v ./...
# RUN go install -v ./...
CMD ["/events-go-publisher"]
golang application main file ( other files not included, cos they are not mater)
package main
import (
"context"
"fmt"
"math/rand"
"os"
"redis_publisher/config"
"redis_publisher/lib"
"sync"
"time"
"github.com/go-redis/redis/v8"
)
var ctx = context.Background()
var wg = sync.WaitGroup{}
var logger lib.Logger
var cnf = config.Values
var letters = []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ")
var mut sync.Mutex
func main() {
publish()
}
func publish() {
rand.Seed(time.Now().Unix())
fmt.Println("Connecting to Redis Server")
logger.LogINFO("Connecting to Redis Server")
var redisClient *redis.Client
redisClient = redis.NewClient(&redis.Options{
Addr: cnf.RedisHostPort,
Password: cnf.RedisPassword,
DB: cnf.RedisDb,
})
if _, err := redisClient.Ping(ctx).Result(); err != nil {
logger.LogError(fmt.Sprintf("[redis] %#v", err))
panic(fmt.Errorf("[redis] %#v", err))
}
i := 1
f, err := os.OpenFile(fmt.Sprintf("/log/%s.log", cnf.PublisherId), os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0666)
if err != nil {
panic(err)
}
defer f.Close()
for {
select {
case <-time.After(time.Millisecond * cnf.PublishInterval):
mut.Lock()
randomWord := randSeq(10)
randomNumber := rand.Intn(100) + rand.Intn(100)
//randomNumber2 := rand.Intn(100)
randomIdNumber := rand.Intn(100) + rand.Intn(101) + rand.Intn(104)
randomWordId := randSeq(5)
payload := fmt.Sprintf(`{"event":"Golang-testing","service":"events.go.publisher.tests","published_time":"%s","data":{"go-publisher":"%s","random-number":%d, "number":%d,"strltr":"test-%s"},"appendix":{"event_id":"EVENT-%s%d"}}`,
time.Now().Format(time.RFC3339),
cnf.PublisherId,
randomNumber,
i,
randomWord,
randomWordId,
randomIdNumber,
)
f.WriteString(payload + "\n")
if err := redisClient.Publish(ctx, cnf.RedisChannel, payload).Err(); err != nil {
panic(err)
}
mut.Unlock()
if cnf.PublishAmount > 0 && cnf.PublishAmount == i {
logger.LogINFO("Finished Publishing from golang")
os.Exit(0)
}
i++
}
}
}
func randSeq(n int) string {
b := make([]rune, n)
for i := range b {
b[i] = letters[rand.Intn(len(letters))]
}
return string(b)
}
I'm open for any details.
And saying again, I tested with PHP, Node.js and Golang.
Thank you!
duktig-dev
commented
2 years ago @akerouanton thank you for your attention. as you can see in the golang code, I do publish into Redis and also write content to file for analyzing.
I have written a script to analyze log file content which returning exactly the same values as I do query in PostgreSQL table (where data inserted from Redis subscriber).
In other words, I do the testing with more than one way and results is the same.
With my opinion, there is some issue with Docker containers memory management, with which I'm not familiar with.
Thank you.
duktig-dev
commented
2 years ago @akerouanton to do real reproduction,
Just visit to this repo for more details or run:
git clone https://github.com/duktig-dev/docker-memory-issue-reproduction.git
cd docker-memory-issue-reproduction
docker-compose up -dThanks!
duktig-dev
commented
2 years ago The interest thing is that when we trying to generate 10,000 values, the result comes as:
Total: 10000
Unique: 2500
Duplicate: 7500And we did this try many times.
Also, we tried to add more time interval between data creation. more than 2-3 second.
akerouanton
commented
2 years ago Your code is a bit complex since it involves Redis (it's not a "minimal bug reproducer"). Anyway, I managed to remove these parts and just fmt.Printf random values.
Whether I run this code on my machine or in a container, it produces the same result: if two processes are started at the same time (ie. same unix timestamp), they produce the same output. Actually that's expected: your code relies on rand.Seed to seed the random number generator. If you go look at rand.Seed doc, you can read the following:
Seed uses the provided seed value to initialize the default Source to a deterministic state. [...] Seed values that have the same remainder when divided by 2³¹-1 generate the same pseudo-random sequence.
To circumvent that, you could use time.Now().UnixNano() to seed the PRNG. You could also use crypto/rand.Int() if you need a cryptographically secure RNG and don't mind doing an extra syscall (see man 2 getrandom).
Also note that how rand.Seed()/rand.Intn() behave is not specific to Go. For instance, in PHP, if you seed the PRNG with a predetermined value, you got the exact same values every time you run the script (and that's sometimes useful for testing purposes). PHP also has a cryptographically secure RNG (see random_int).
duktig-dev
commented
2 years ago Hi @akerouanton ! Thank you for response.
For the first, of course I have created a clean/pure code for testing some days ago and commented here: (if you scroll top you will see: Just visit to this repo for more details or run:
git clone https://github.com/duktig-dev/docker-memory-issue-reproduction.git
cd docker-memory-issue-reproduction
docker-compose up -d)
Anyways,
Yes, because the Docker starts/deploys containers in same time, assuming the random functionality will generate the same values.
I will change the code with your recommendations and report to you.
Thanks again for your attention.
duktig-dev
commented
2 years ago @akerouanton it worked !
Generated 1,000,000 random data and it works perfect!
Starting...
Get file: Golang-Publisher-A.log lines 250000
Get file: Golang-Publisher-B.log lines 250000
Get file: Golang-Publisher-C.log lines 250000
Get file: Golang-Publisher-D.log lines 250000
Total: 1000000
Unique: 1000000
Duplicate: 0And this a part of improved code:
func getRandomInt64(limit int64) int64 {
nBig, err := rand.Int(rand.Reader, big.NewInt(limit))
if err != nil {
panic(err)
}
n := nBig.Int64()
//fmt.Printf("Here is a random %T in [0,%d) : %d\n", n, limit, n)
return n
}
func getSmallInt() byte {
b := []byte{0}
if _, err := rand.Reader.Read(b); err != nil {
panic(err)
}
return b[0]
}
func getToken(length int) string {
randomBytes := make([]byte, 32)
_, err := rand.Read(randomBytes)
if err != nil {
panic(err)
}
return base32.StdEncoding.EncodeToString(randomBytes)[:length]
}Thank you very much !!!
thaJeztah
commented
2 years ago Thanks for helping out, @akerouanton !
Expected behavior
Running 4 containers from same image Application running in docker container generates random data and creates json string. Each application running inside containers should generate/create unique values inside docker containers.
Actual behavior
Running 4 containers from same image Applications running inside docker containers mixing a values and almost 70% is the same value. Assuming, issue with containers "shared" memory. Same application(s) running outside of docker container works perfect and generates unique values.
Tested with PHP, Node.js, Golang. The issue is the same.
Steps to reproduce the behavior
Just visit to this repo for more details or run:
In some words you can imagine:
Without docker containers creation, running 4 copies of the same application: Generates unique random values.
Inside docker containers, applications running and 70% of values is the same. Assuming, a problem comes from container's memory stack.
Using docker-compose to deploy containers:
all 4 containers using the same image
Output of
docker version:Output of
docker info:Running the docker in: Ubuntu 21.10 ( bare metal )