Unable to init database

[technicallyty]

unable to init database using '/var/folders/70/rld0rwh92bg781zkxt1x6szh0000gn/T/TestProcessEvmEvent_IRIsExist1738323574/001/bin/initdb -A password -U postgres -D /var/folders/70/rld0rwh92bg781zkxt1x6szh0000gn/T/TestProcessEvmEvent_IRIsExist1738323574/001/data --pwfile=/var/folders/70/rld0rwh92bg781zkxt1x6szh0000gn/T/TestProcessEvmEvent_IRIsExist1738323574/001/pwfile': exit status 1

System Specs:

Apple M1 Pro MacOS Ventura 13.0

using github.com/fergusstrange/embedded-postgres v1.19.0

Only way i've been able to reconcile this is by restarting my computer.

[fergusstrange]

Hey @technicallyty. I've not seen this before but I am aware of some issues running with Apple M1.

Could you post the Go code you used to produce this error?

Have you also been able to try this on a non Apple M1 device?

[technicallyty]

here's the code which setups up the embedded pg

package dbtest

import (
    "fmt"
    "io"
    "testing"

    embeddedpostgres "github.com/fergusstrange/embedded-postgres"
    "github.com/phayes/freeport"
    "github.com/rs/zerolog"
    "github.com/stretchr/testify/require"

    "github.com/regen-network/toucan-bridge-service/db"
)

// NewTestDatabase creates an embedded postgres database for testing purposes.
func NewTestDatabase(t *testing.T) db.Database {
    port, err := freeport.GetFreePort()
    require.NoError(t, err)

    logger := zerolog.New(zerolog.NewConsoleWriter())
    cfg := embeddedpostgres.DefaultConfig().
        Port(uint32(port)).
        Database("postgres").
        Logger(logger).
        RuntimePath(t.TempDir())

    postgres := embeddedpostgres.NewDatabase(cfg)
    require.NoError(t, postgres.Start())

    t.Cleanup(func() {
        require.NoError(t, postgres.Stop())
    })

    customDbWrapper, err := db.NewDatabase(
        fmt.Sprintf("host=localhost port=%d user=postgres password=postgres dbname=postgres sslmode=disable", port),
        logger,
    )
    require.NoError(t, err)
    return customDbWrapper
}

Have you also been able to try this on a non Apple M1 device?

I have not

edit: i have never seen this in our CI system / github actions, though. i think it runs it in a linux container.

[mchusovlianov]

I have similar issue on M2 chip:

unable to init database using '/var/folders/j2/_f8t7nhd1zx2ycjy6nymcddc0000gn/T/embedded_postgres_test_lib_2003912407/bin/initdb -A password -U postgres -D /var/folders/j2/_f8t7nhd1zx2ycjy6nymcddc0000gn/T/embedded_postgres_test_data_2418798707 --pwfile=/var/folders/j2/_f8t7nhd1zx2ycjy6nymcddc0000gn/T/embedded_postgres_test_runtime_1981438580/pwfile': exit status 1

same code on x86_64 platform works without any issues

[technicallyty]

I have similar issue on M2 chip:

unable to init database using '/var/folders/j2/_f8t7nhd1zx2ycjy6nymcddc0000gn/T/embedded_postgres_test_lib_2003912407/bin/initdb -A password -U postgres -D /var/folders/j2/_f8t7nhd1zx2ycjy6nymcddc0000gn/T/embedded_postgres_test_data_2418798707 --pwfile=/var/folders/j2/_f8t7nhd1zx2ycjy6nymcddc0000gn/T/embedded_postgres_test_runtime_1981438580/pwfile': exit status 1

same code on x86_64 platform works without any issues

how do you usually resolve this issue? Do you just restart your laptop as well? Hoping you may have found another way so I don't have to keep restarting my machine to run tests 😆

[mchusovlianov]

I don't resolve it. Just found this issue yesterday when moving to a new laptop. Decided to continue work in my old environment so far.

[sol-mocha]

+1, having issues on the M1 Max as well as in github action CI using ubuntu-latest. Any logs I can provide to help debug?

embeddedDB := embeddedpostgres.NewDatabase(embeddedpostgres.DefaultConfig().
        Username(dbConfig.GetUser()).
        Password(dbConfig.GetPassword()).
        Database(dbConfig.GetDBName()).
        Port(uint32(dbConfig.GetPort())),
    )
    if err := embeddedDB.Start(); err != nil {
        return nil, err
    }

[Minsoo-Shin]

/Users/ms/.embedded-postgres-go/extracted/bin/initdb -A password -U acloset -D /Users/ms/.embedded-postgres-go/extracted/data --pwfile=/Users/ms/.embedded-postgres-go/extracted/pwfile
The files belonging to this database system will be owned by user "ms".
This user must also own the server process.

The database cluster will be initialized with locale "ko_KR.UTF-8".
The default database encoding has accordingly been set to "UTF8".
initdb: could not find suitable text search configuration for locale "ko_KR.UTF-8"
The default text search configuration will be set to "simple".

Data page checksums are disabled.

creating directory /Users/ms/.embedded-postgres-go/extracted/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 20
selecting default shared_buffers ... 400kB
selecting default time zone ... Asia/Seoul
creating configuration files ... ok
running bootstrap script ... 2022-11-25 16:12:44.266 KST [4737] FATAL:  could not create shared memory segment: Cannot allocate memory
2022-11-25 16:12:44.266 KST [4737] DETAIL:  Failed system call was shmget(key=16783864, size=56, 03600).
2022-11-25 16:12:44.266 KST [4737] HINT:  This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMALL parameter.  You might need to reconfigure the kernel with larger SHMALL.
    The PostgreSQL documentation contains more information about shared memory configuration.
child process exited with exit code 1
initdb: removing data directory "/Users/ms/.embedded-postgres-go/extracted/data"

i think this is because that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMALL parameter

we have to configuring shared memory in macOS is to create a file named /etc/sysctl.conf, https://www.postgresql.org/docs/13/kernel-resources.html

[jacques-n]

When you see this message, you should do waht @Minsoo-Shin did just above. You should hand run the same initdb command and see what the actual error message is. We also saw the shmget failure. We've seen this error after successive builds using this package. In all cases, we were failing to shutdown the postgres embedded instance (calling stop). You can check ps to see if you have a bunch of zombie postgres instances running and kill them. You can also use ipcs -a on OSX to see what shmem segments might be outstanding. If that is your problem, you need to fix the places where you aren't shutting this down.

Note that this is not a problem in container build envs because zombie processes are just thrown away when the container is. However, if you're running on a machine without a container (e.g. OSX), failing to shutdown the Postgres instance will cause many times will ultimately result in depletion of available shared memory and shmem failures. A reboot fixing the problem is also a good indicator of this.

[SouzaGustavoGG]

Facing the same issue on Windows 11: unable to init database using: <MY_PATH>\.embedded-postgres-go\extracted\bin\initdb.exe -A password -U postgres -D <MY_PATH>\.embedded-postgres-go\extracted\data --pwfile=<MY_PATH>\.embedded-postgres-go\extracted\pwfile exit status 1

On Windows 10 and Linux still working.

UPDATE

Trying to debug: Executing the bin\initdb.exe under the compressed folder I got this error: VCRUNTIME140.dll not found.

So I download this Visual C++ package from Microsoft: https://www.microsoft.com/pt-BR/download/details.aspx?id=48145

Then my problem on Windows 11 was solved.

[fergusstrange]

Hey @technicallyty we've just cut a release v1.20.0 which adds better support for Apple M1/2. I'd be interested to see if you fair any better with this and Postgres v14.2+ which should offer native support.

We also have some upcoming changes to improve logging, including displaying errors a little more clearly in this case.

[fergusstrange]

Release now made for improved logging at v1.21.0. I'm going to close this now. Please reopen should you continue to have issues.

[sol-mocha]

Release now made for improved logging at v1.21.0. I'm going to close this now. Please reopen should you continue to have issues.

late to reply, but this version works for me on m1 👍🏾