PostgreSQL Semaphore exhaustion

[DavidCookBrite]

We use testing.postgresql in our automated tests that fire-up a local instance of PostgreSQL using initdb and then run our schema creation and data seeding tasks.

For the past few months we've been getting test failures due with this error:

initdb failed: 'FATAL: could not create semaphores: No space left on device DETAIL: Failed system call was semget(1, 17, 03600). HINT: This error does not mean that you have run out of disk space. It occurs when either the system limit for the maximum number of semaphore sets (SEMMNI), or the system wide maximum number of semaphores (SEMMNS), would be exceeded. You need to raise the respective kernel parameter. Alternatively, reduce PostgreSQL's consumption of semaphores by reducing its max_connections parameter. The PostgreSQL documentation contains more information about configuring your system for PostgreSQL. child process exited with exit code 1 initdb: removing contents of data directory "/tmp/tmpctGr9I/data"

These tests run on an Amazon EC2 instance running Amazon Linux 2015.09 x64. Here's the output from ipcs -ls showing semaphore configuration:

[ec2-user@ip-127-0-0-1 etc]$ ipcs -ls ------ Semaphore Limits -------- max number of arrays = 128 max semaphores per array = 250 max semaphores system wide = 32000 max ops per semop call = 32 semaphore max value = 32767 [ec2-user@ip-127-0-0-1 etc]$

We see the same issue on both Mac OS X with PostgreSQL 9.4 as well as on our Amazon EC2 instances running PostgreSQL 9.3. PostgreSQL is installed via yum on the EC2 instances via our build script, and brew on Mac.

We're unsure if it's a bug in PostgreSQL or testing.postgresql, but we thought we'd start here.

We're currently using testing.postgresql 1.1.2 (we're in the process of upgrading to the latest version - it is currently incompatible with our builds).

[georgevreilly]

When @DavidCookBrite and I upgraded to testing.postgresql 1.3.0, we ran into https://github.com/tk0miya/testing.common.database/issues/2. We had to revert to 1.1.2, as we could no longer install our app on ElasticBeanstalk.

Based on some quick Google searches I did, I think this may be due to Postgres opening a fairly large number of connections (MAX_CONNECTIONS). Obviously, a server that exists only for the lifetime of a single test does not need to open a lot of connections.

Note: we are running approximately 1,000 tests in a full test run, split across multiple cores using the multiprocessing feature of Nose.

[tk0miya]

Sorry for late response. It might be caused by many postgresql processes. Please reduce concurrency or change your postgresql settings to lesser memory usage. For example, following code makes connections lesser:

pgsql = testing.postgresql.Postgresql(
    postgres_args='-h 127.0.0.1 -F -c logging_collector=off -c max_connections=10'
)

[tk0miya]

Now I close this. But please reopen if you still have problem. Thanks,

[rubenflamshepherd]

If anyone else is running into a similar issue and touching the max_connections parameter doesn't help: I fixed this as I hadn't been calling postgresql.stop() per the recommended usage here -_-.