embedded-postgres on Apple Silicon (M1) issue

[drej1]

Hi there,

thanks for the great work in the first place, we were using embedded Postgres for a while now.

Now I switched to a MacBook Pro with the new M1 SoC which has arm64 architecture. I debugged DefaultPostgresBinaryResolver#getPgBinary and the line

Resource resource = findPgBinary(normalize(format("postgres-%s-%s.txz", system, architecture)));

does not find any binary because it is searching for postgres-darwin-arm_64.txzwhich I didn't find in https://mvnrepository.com/artifact/io.zonky.test.postgres

I assume there will be no release of Postgres for Darwin arm64 soon so is there a way to load the Darwin-amd64 embedded Postgres?

Dependencies we use:

compile("io.zonky.test:embedded-database-spring-test:1.6.1")
dependencyManagement {
        imports {
            mavenBom 'io.zonky.test.postgres:embedded-postgres-binaries-bom:9.6.16'
        }
    }

Thanks

[drej1]

@tomix26 ? :)

[tomix26]

Hi @drej1, thanks for the report. The mac os binaries are actually EnterpriseDB binaries but in reduced and repacked form. So now we are waiting for EnterpriseDB to release binaries for M1 architecture.

[drej1]

@tomix26 thanks for the info. Is there any roadmap on EnterpriseDB for supporting M1? If not, won't there be any workaround to support M1 in the meantime? I did a local workaround for our project. It's very primitive :D I duplicated io.zonky.test.db.postgres.embedded.DefaultPostgresBinaryResolver and the only change is in #getPgBinary as follows:

@Override
public InputStream getPgBinary(String system, String machineHardware) throws IOException {
    String architecture = ArchUtils.normalize(machineHardware);
    String distribution = LinuxUtils.getDistributionName();

    log.info("Detected distribution: '{}'", Optional.ofNullable(distribution).orElse("Unknown"));

    if (distribution != null) {
        Resource resource = findPgBinary(normalize(format("postgres-%s-%s-%s.txz", system, architecture, distribution)));
        if (resource != null) {
            log.info("Distribution specific postgres binaries found: {}", resource.getFilename());
            return resource.getInputStream();
        } else {
            log.debug("Distribution specific postgres binaries not found");
        }
    }

    // START CHANGES
    if (system.equals("Darwin") && architecture.equals("arm_64")) {
        architecture = "x86_64";
    }
    // END CHANGES

    Resource resource = findPgBinary(normalize(format("postgres-%s-%s.txz", system, architecture)));
    if (resource != null) {
        log.info("System specific postgres binaries found: {}", resource.getFilename());
        return resource.getInputStream();
    }

    log.error("No postgres binaries were found, you must add an appropriate maven dependency "
              + "that meets the following parameters - system: {}, architecture: {}", system, architecture);
    throw new IllegalStateException("Missing postgres binaries");
}

This downloads and runs intel architecture based embedded DB which works fine on the M1.

[tomix26]

Unfortunately, I don't know about any roadmap. At least not yet 😞

Nevertheless, your solution seems great to me. Using rosetta to translate x86_64 binaries to arm64 architecture is good enough. It should also be possible to avoid implementing the custom resolver by setting the architecture via a system property: -Dos.arch=x86_64

[drej1]

Hmm it sounds interesting, I didn't know about this option. I will try to play with this. Other people on the team don't use M1 so I want to find some global solution for the project. Thanks for the tip.

I leave this open for the native support.

[okopok]

@tomix26 still didn't work, even with duplicated DefaultPostgresBinaryResolver

[jedvardsson]

@okopok Here is what I have working on my M1.

@Bean
@ConditionalOnProperty(name = "spring.datasource.embedded", havingValue = "true")
public EmbeddedPostgres embeddedPostgres() throws IOException {
    return EmbeddedPostgres.builder()
            .setPgBinaryResolver((system, machineHardware) -> {
                var arch = system.equals("Darwin") && machineHardware.equals("aarch64") ? "x86_64" : machineHardware;
                return DefaultPostgresBinaryResolver.INSTANCE.getPgBinary(system, arch);
            })
            .start();
}

[mrioan]

Taking the response from @jedvardsson and modifying it for my case:

builder.setPgBinaryResolver((system, machineHardware) -> {
                            String arch = system.equals("Darwin") && machineHardware.equals("aarch64") ? "x86_64" : machineHardware;
                            return EmbeddedPostgres.class.getResourceAsStream(format("/postgresql-%s-%s.txz", system, arch));
                        }).start();

I also had to do the following:

To solve: dyld: Library not loaded: @loader_path/../lib/libpq.5.dylib

brew update
brew link libpq --force

And to solve: This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMALL parameter. You might need to reconfigure the kernel with larger SHMALL.

sudo vi /etc/sysctl.conf

and pasted:

kern.sysv.shmmax=4194304
kern.sysv.shmmin=1
kern.sysv.shmmni=10240
kern.sysv.shmseg=4096
kern.sysv.shmall=33554432

[tomix26]

I've just released version 1.3.0 that includes support for Rosetta 2 emulation out of the box. So it's no longer necessary to implement a custom binary resolver. Nevertheless, this is only a temporary solution and I'm still waiting for native binaries to become available.

[drej1]

Hi @tomix26 , how does it look with the native binaries? I have postgres installed via homebrew and it runs natively on Apple Silicon.

BTW the workaround works like a charm ;)

[jedvardsson]

@tomix26 What do you think about allowing the resolver to piggy-back on a locally installed Postgres or perhaps even a locally running Postgres.

[tomix26]

@drej1 Unfortunately, I have no new information, I'm still waiting for EnterpriseDB to release binaries for M1 architecture.

@jedvardsson I don't like it, because the main idea of this project is to have platform independent tests that can be easily run without any additional setup. If the tests were dependent on a local database installation, it would break this idea.

[tazle]

Looking at the EnterpriseDB binaries, they appear to be universal binaries and do include native arm64 variant:

% file psql 
psql: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64:Mach-O 64-bit executable arm64]
psql (for architecture x86_64): Mach-O 64-bit executable x86_64
psql (for architecture arm64):  Mach-O 64-bit executable arm64

(sql came from https://www.enterprisedb.com/download-postgresql-binaries or https://sbp.enterprisedb.com/getfile.jsp?fileid=1257912 to be precise)

[timt]

I've just set up an M1 mac mini as a build server, and the Rosetta 2 emulation works really well, but only for a short while, then it seems like it stops working. (usually after about 10 builds have completed)

Has anyone else experienced this?

We are using the following dependencies/versions:

 "io.zonky.test" % "embedded-postgres" % "1.3.1"
 "io.zonky.test.postgres" % "embedded-postgres-binaries-darwin-amd64" % "11.13.0"

We see this warning just before it starts failing:

[WARN] io.zonky.test.db.postgres.embedded.DefaultPostgresBinaryResolver pool-1-thread-1 - No native binaries supporting aarch64 architecture found. Trying to use binaries for amd64 architecture instead: 'postgres-darwin-x86_64.txz'. Make sure you have Rosetta 2 emulation enabled. Note that performance may be degraded.

[GPJohnston]

I've just set up an M1 mac mini as a build server, and the Rosetta 2 emulation works really well, but only for a short while, then it seems like it stops working. (usually after about 10 builds have completed)

Has anyone else experienced this?

We are using the following dependencies/versions:

 "io.zonky.test" % "embedded-postgres" % "1.3.1"
 "io.zonky.test.postgres" % "embedded-postgres-binaries-darwin-amd64" % "11.13.0"

We see this warning just before it starts failing:

[WARN] io.zonky.test.db.postgres.embedded.DefaultPostgresBinaryResolver pool-1-thread-1 - No native binaries supporting aarch64 architecture found. Trying to use binaries for amd64 architecture instead: 'postgres-darwin-x86_64.txz'. Make sure you have Rosetta 2 emulation enabled. Note that performance may be degraded.

@timt

I have been running into a similar issue as you today. It has built maybe 10-15 times before failing.

I believe that log warning you have seen is just part of the process of selecting the binaries for the M1 architecture.

After running the failing command In my shell I got this.

running bootstrap script ... 2022-01-27 15:08:54.266 GMT [58511] FATAL:  could not create shared memory segment: Cannot allocate memory
2022-01-27 15:08:54.266 GMT [58511] DETAIL:  Failed system call was shmget(key=5432001, size=56, 03600).
2022-01-27 15:08:54.266 GMT [58511] HINT:  This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMALL parameter.  You might need to reconfigure the kernel with larger SHMALL.
    The PostgreSQL documentation contains more information about shared memory configuration.
child process exited with exit code 1

Given that I have allocated more shared memory by executing the commands below (editing /etc/sysctl.conf did not help). I have seen an improvement, albeit likely temporary.

sudo sysctl kern.sysv.shmmax=104857600
sudo sysctl kern.sysv.shmall=2560

[tomix26]

@timt Could you please test your build with version 11.14.0? This version should already contain native arm binaries, so maybe it could solve the problem. Note that the name of the dependency still refers to amd64 platform and when the build starts the library is still printing out the warning message. Don't worry, it's because the package contains binaries in Apple's universal format and the library doesn't take it into account.

[tomix26]

@GPJohnston I was also facing similar issue and in my case it was caused by many unfinished "zombie" processes that overwhelmed my machine. So please go through processes in activity monitor and check that there is no postgres processes when the build is not running.

[GPJohnston]

@tomix26 I discovered the same when shared memory increase eventually bled out -> many processes running. Thank you.

[timt]

@tomix26 I've tried upgrading to 11.14.0 as you suggested, but still have the same issue.

On deeper investigation it is the same issue that @GPJohnston describes and his solution does work, although playing with the values, kern.sysv.shmmax, kern.sysv.shmall incrementally it seems they do not entirely fix this issue rather delays it. By setting the shmmax suitably high as suggested by @GPJohnston does buy us some time until this issue is properly resolved.

@tomix26 I've searched activity monitor for Zombie processes but can't see anything obvious - what name do these zombies have?

[tomix26]

I guess the name of the process is just "postgres" or something similar to this.

[michalmatovcik]

For me it started working after updating deps:

embedded-database-spring-test to version 2.1.1 embedded-postgres-binaries-bom to version 14.2.0

(current latest versions)

[tomix26]

Unfortunately I just realized that only the latest major version of EnterpriseDB binaries includes native support for M1 architecture. So to fix the problem, try to upgrade to postgres 14 or later.

Related issue: https://github.com/zonkyio/embedded-postgres/issues/86

[rpoitras]

Thanks folks 👍. Upgrading from 1.3.1 to 2.0.0 fixed this for me.

In Gradle:

testImplementation 'io.zonky.test:embedded-postgres:2.0.0'

BTW, I'm using Quarkus. Not sure if this is the same for Spring or SpringBoot.