search

timescale / timescaledb-tune

A tool for tuning TimescaleDB for better performance by adjusting settings to match your system's CPU and memory resources.
https://www.timescale.com/
Apache License 2.0
450 stars 41 forks source link

"panic: bytes must be at least 1 byte" when running docker image timescaledb:1.2.2-pg11 #48

Closed corytu closed 3 years ago

corytu commented 5 years ago

I am trying to install TimescaleDB with Docker. It works perfectly on MacOS Mojave 10.14.4, but on Ubuntu 14.04.6 LTS, Trusty Tahr, an error was raised. I found out how it happened, but have no idea why it happened and how to fix it.

docker pull timescale/timescaledb:1.2.2-pg11
docker run -d --name timescaledb -p 5432:5432 -e POSTGRES_PASSWORD=password timescale/timescaledb:1.2.2-pg11

However, the container was exited immediately afterwards. It seems like that error from this line of timescaledb-tune was triggered in the initiation process. I used docker logs timescaledb and found error logs like this:

Using postgresql.conf at this path:
/var/lib/postgresql/data/postgresql.conf
Writing backup to:
/tmp/timescaledb_tune.backup201904180746
panic: bytes must be at least 1 byte
goroutine 1 [running]:
github.com/timescale/timescaledb-tune/internal/parse.parseIntToFloatUnits(...)
    /go/src/github.com/timescale/timescaledb-tune/internal/parse/parse.go:43
github.com/timescale/timescaledb-tune/internal/parse.BytesToDecimalFormat(0x0, 0x2a, 0xc00003fd40)
    /go/src/github.com/timescale/timescaledb-tune/internal/parse/parse.go:66 +0x1ff
github.com/timescale/timescaledb-tune/pkg/tstune.(*Tuner).processQuiet(0xc00000c860, 0xc000070ff0, 0x0, 0x0)
    /go/src/github.com/timescale/timescaledb-tune/pkg/tstune/tuner.go:601 +0x51
github.com/timescale/timescaledb-tune/pkg/tstune.(*Tuner).Run(0xc00000c860, 0x61a020, 0x550800, 0xc00000e010, 0x550820, 0xc00000e018, 0x550820, 0xc00000e020)
    /go/src/github.com/timescale/timescaledb-tune/pkg/tstune/tuner.go:261 +0x3b6
main.main()
    /go/src/github.com/timescale/timescaledb-tune/cmd/timescaledb-tune/main.go:55 +0x170
RobAtticus commented 5 years ago

It seems to be reading a bad value from your memory settings. So the question would be, what are the values of the following things if you exec into the docker container:

cat /sys/fs/cgroup/memory/memory.limit_in_bytes

and

free -m | grep 'Mem' | awk '{print $2}'
corytu commented 5 years ago

@RobAtticus thanks, but like I mentioned earlier, the container was exited immediately after docker run, which makes me hardly possible to docker exec into the container because it's not running.

Is this relevant to any docker settings that I can check?

RobAtticus commented 5 years ago

You can pass an environment variable TS_TUNE_MEMORY to what the memory should be using the -e flag: docker run -e TS_TUNE_MEMORY="1 GB" ...

Then you should be able to get into it and get those values

corytu commented 5 years ago

@RobAtticus I did docker run -d --name timescaledb -p 5432:5432 -e TS_TUNE_MEMORY="1 GB" -e POSTGRES_PASSWORD=password timescale/timescaledb:1.2.2-pg11, and the container is still exited right away.

However this time the logs are different:

2019-04-19 03:00:57.740 UTC [46] CONTEXT:  PL/pgSQL function inline_code_block line 15 at RAISE
2019-04-19 03:00:57.745 UTC [48] ERROR:  TimescaleDB background worker connected to template database, exiting
CREATE EXTENSION

/usr/local/bin/docker-entrypoint.sh: sourcing /docker-entrypoint-initdb.d/001_reenable_auth.sh
2019-04-19 03:00:57.746 UTC [32] LOG:  background worker "TimescaleDB Background Worker Scheduler" (PID 48) exited with exit code 1

/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/002_timescaledb_tune.sh
exit: incorrect PostgreSQL bytes format: '1'
RobAtticus commented 5 years ago

Sorry that's my bad, it should have been -e TS_TUNE_MEMORY="1GB" (you can change 1GB to whatever is a suitable resource value too).

corytu commented 5 years ago

@RobAtticus thank you. I now get into the running container with -e TS_TUNE_MEMORY="1GB" when docker run. For the two mention values, I listed them below.

cat /sys/fs/cgroup/memory/memory.limit_in_bytes: 18446744073709551615 free -m | grep 'Mem' | awk '{print $2}': 7479

RobAtticus commented 5 years ago

Thanks @corytu , I think I found the root cause of the problem and with our next release it should be fixed. For now, you can pass in -e TS_TUNE_MEMORY="7GB" or however much memory you want PG to have.

thalesmaoanz commented 2 months ago

I'm still facing this problem in a raspberry pi 4. Maybe re-open it? The TS_TUNE_MEMORY definition solves the problem.

$ docker compose up
[+] Running 2/2
 ✔ Network timescale_default  Created                                                             0.2s 
 ✔ Container timescale-pg16   Created                                                             0.1s 
Attaching to timescale-pg16
timescale-pg16  | The files belonging to this database system will be owned by user "postgres".
timescale-pg16  | This user must also own the server process.
timescale-pg16  | 
timescale-pg16  | The database cluster will be initialized with locale "en_US.utf8".
timescale-pg16  | The default database encoding has accordingly been set to "UTF8".
timescale-pg16  | The default text search configuration will be set to "english".
timescale-pg16  | 
timescale-pg16  | Data page checksums are disabled.
timescale-pg16  | 
timescale-pg16  | fixing permissions on existing directory /var/lib/postgresql/data/pgdata ... ok
timescale-pg16  | creating subdirectories ... ok
timescale-pg16  | selecting dynamic shared memory implementation ... posix
timescale-pg16  | selecting default max_connections ... 100
timescale-pg16  | selecting default shared_buffers ... 128MB
timescale-pg16  | selecting default time zone ... America/Sao_Paulo
timescale-pg16  | creating configuration files ... ok
timescale-pg16  | running bootstrap script ... ok
timescale-pg16  | sh: locale: not found
timescale-pg16  | 2024-08-19 18:38:53.421 -03 [36] WARNING:  no usable system locales were found
timescale-pg16  | performing post-bootstrap initialization ... ok
timescale-pg16  | syncing data to disk ... ok
timescale-pg16  | 
timescale-pg16  | 
timescale-pg16  | Success. You can now start the database server using:
timescale-pg16  | 
timescale-pg16  |     pg_ctl -D /var/lib/postgresql/data/pgdata -l logfile start
timescale-pg16  | 
timescale-pg16  | initdb: warning: enabling "trust" authentication for local connections
timescale-pg16  | initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
timescale-pg16  | waiting for server to start....2024-08-19 18:38:54.855 -03 [42] LOG:  starting PostgreSQL 16.3 on aarch64-unknown-linux-musl, compiled by gcc (Alpine 13.2.1_git20240309) 13.2.1 20240309, 64-bit
timescale-pg16  | 2024-08-19 18:38:54.857 -03 [42] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
timescale-pg16  | 2024-08-19 18:38:54.864 -03 [45] LOG:  database system was shut down at 2024-08-19 18:38:54 -03
timescale-pg16  | 2024-08-19 18:38:54.874 -03 [42] LOG:  database system is ready to accept connections
timescale-pg16  | 2024-08-19 18:38:54.877 -03 [48] LOG:  TimescaleDB background worker launcher connected to shared catalogs
timescale-pg16  |  done
timescale-pg16  | server started
timescale-pg16  | CREATE DATABASE
timescale-pg16  | 
timescale-pg16  | 
timescale-pg16  | /usr/local/bin/docker-entrypoint.sh: sourcing /docker-entrypoint-initdb.d/000_install_timescaledb.sh
timescale-pg16  | CREATE EXTENSION
timescale-pg16  | 2024-08-19 18:38:56.356 -03 [66] ERROR:  TimescaleDB background worker connected to template database, exiting
timescale-pg16  | CREATE EXTENSION
timescale-pg16  | 2024-08-19 18:38:56.365 -03 [42] LOG:  background worker "TimescaleDB Background Worker Scheduler" (PID 66) exited with exit code 1
timescale-pg16  | CREATE EXTENSION
timescale-pg16  | 
timescale-pg16  | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/001_timescaledb_tune.sh
timescale-pg16  | cat: can't open '/sys/fs/cgroup/memory.max': No such file or directory
timescale-pg16  | /docker-entrypoint-initdb.d/001_timescaledb_tune.sh: line 49: [: -gt: unary operator expected
timescale-pg16  | Using postgresql.conf at this path:
timescale-pg16  | /var/lib/postgresql/data/pgdata/postgresql.conf
timescale-pg16  | 
timescale-pg16  | Writing backup to:
timescale-pg16  | /tmp/timescaledb_tune.backup202408191838
timescale-pg16  | 
timescale-pg16  | panic: bytes must be at least 1 byte (got 0)
timescale-pg16  | 
timescale-pg16  | goroutine 1 [running]:
timescale-pg16  | github.com/timescale/timescaledb-tune/internal/parse.parseIntToFloatUnits(0x40000cfc08?)
timescale-pg16  |   /go/pkg/mod/github.com/timescale/timescaledb-tune@v0.16.0/internal/parse/parse.go:115 +0x114
timescale-pg16  | github.com/timescale/timescaledb-tune/internal/parse.BytesToDecimalFormat(0x0?)
timescale-pg16  |   /go/pkg/mod/github.com/timescale/timescaledb-tune@v0.16.0/internal/parse/parse.go:138 +0x1c
timescale-pg16  | github.com/timescale/timescaledb-tune/pkg/tstune.(*Tuner).processQuiet(0x40000cfee0, 0x40000af1c0, 0x0)
timescale-pg16  |   /go/pkg/mod/github.com/timescale/timescaledb-tune@v0.16.0/pkg/tstune/tuner.go:659 +0x44
timescale-pg16  | github.com/timescale/timescaledb-tune/pkg/tstune.(*Tuner).Run(0x40000cfee0, 0x0?, {0x15f198, 0x40000a0040}, {0x15f1b8, 0x40000a0048}, {0x15f1b8, 0x40000a0050})
timescale-pg16  |   /go/pkg/mod/github.com/timescale/timescaledb-tune@v0.16.0/pkg/tstune/tuner.go:300 +0x42c
timescale-pg16  | main.main()
timescale-pg16  |   /go/pkg/mod/github.com/timescale/timescaledb-tune@v0.16.0/cmd/timescaledb-tune/main.go:65 +0xf4
timescale-pg16 exited with code 0
$ cat docker-compose.yml 
services:
  timescale-pg16:
#    image: timescale/timescaledb-ha:pg14-latest
#    image: timescale/timescaledb-ha:pg16
#    image: timescale/timescaledb:latest-pg14
    image: timescale/timescaledb:latest-pg16
    container_name: timescale-pg16
    restart: unless-stopped
    ports:
      - '5432:5432'
    volumes:
      - ./data:/var/lib/postgresql/data/pgdata
    environment:
      - POSTGRES_USER=admin
      - POSTGRES_PASSWORD=mysuperstrongdatabasepassword
      - PGDATA=/var/lib/postgresql/data/pgdata
      - TZ=America/Sao_Paulo
#      - TS_TUNE_MEMORY=8GB # free -m | grep 'Mem' | awk '{print $2}'
    logging:
      options:
        max-size: "100m"
        max-file: "3"

$ cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ uname -a
Linux simoi 6.6.31+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.31-1+rpt1 (2024-05-29) aarch64 GNU/Linux