OSError when executing test_message_queues.py unittest

[Rakagami]

When I run the test_message_queues.py tests I get the error OSError: This process already has the maximum number of files open for various cases. I am executing it on the commit with the tag rel1.0.5. Why is this happening?

The output looks as follows:

 ❯ python tests/test_message_queues.py
...EE..E.E.E.E.......E......................
======================================================================
ERROR: test_name_as_bytes (__main__.TestMessageQueueCreation)
Test that the name can be bytes.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/raka/Workspace/idp/posix_ipc/tests/test_message_queues.py", line 135, in test_name_as_bytes
    mq = posix_ipc.MessageQueue(name, posix_ipc.O_CREX)
OSError: This process already has the maximum number of files open

======================================================================
ERROR: test_name_as_unicode (__main__.TestMessageQueueCreation)
Test that the name can be unicode.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/raka/Workspace/idp/posix_ipc/tests/test_message_queues.py", line 154, in test_name_as_unicode
    mq = posix_ipc.MessageQueue(name, posix_ipc.O_CREX)
OSError: This process already has the maximum number of files open

======================================================================
ERROR: test_o_creat_new (__main__.TestMessageQueueCreation)
tests posix_ipc.O_CREAT to create a new MessageQueue without
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/raka/Workspace/idp/posix_ipc/tests/test_message_queues.py", line 105, in test_o_creat_new
    mq = posix_ipc.MessageQueue(name, posix_ipc.O_CREAT)
OSError: This process already has the maximum number of files open

======================================================================
ERROR: test_randomly_generated_name (__main__.TestMessageQueueCreation)
tests that the randomly-generated name works
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/raka/Workspace/idp/posix_ipc/tests/test_message_queues.py", line 118, in test_randomly_generated_name
    mq = posix_ipc.MessageQueue(None, posix_ipc.O_CREX)
OSError: This process already has the maximum number of files open

======================================================================
ERROR: test_read_flag_new_queue (__main__.TestMessageQueueCreation)
test that the read flag is respected on a new queue
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/raka/Workspace/idp/posix_ipc/tests/test_message_queues.py", line 179, in test_read_flag_new_queue
    mq = posix_ipc.MessageQueue(None, posix_ipc.O_CREX, read=False)
OSError: This process already has the maximum number of files open

======================================================================
ERROR: test_write_flag_new_queue (__main__.TestMessageQueueCreation)
test that the write flag is respected on a new queue
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/raka/Workspace/idp/posix_ipc/tests/test_message_queues.py", line 194, in test_write_flag_new_queue
    mq = posix_ipc.MessageQueue(None, posix_ipc.O_CREX, write=False)
OSError: This process already has the maximum number of files open

======================================================================
ERROR: test_block_flag_default_value_and_writability (__main__.TestMessageQueuePropertiesAndAttributes)
test that the block flag is True by default and can be changed
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/raka/Workspace/idp/posix_ipc/tests/test_message_queues.py", line 548, in test_block_flag_default_value_and_writability
    mq = posix_ipc.MessageQueue(None, posix_ipc.O_CREX)
OSError: This process already has the maximum number of files open

----------------------------------------------------------------------
Ran 44 tests in 2.006s

FAILED (errors=7)

[osvenskan]

Can you provide more details about your environment? Operating system and Python version would be a good start.

If you're on Linux, some message queue limits (including the maximum number of msg queues) is described under "/proc interfaces" on this page: https://linux.die.net/man/7/mq_overview

On that same page under the heading "Mounting the message queue file system", you can read about how to see how many message queues are defined on your system.

My guess is that you're hitting the limit of message queues that your user is allowed to create.

[xhuaoe]

Same here. Ubuntu 20, python 3.8, posix_ipc 1.1.0

It's easy to reproduce: sudo python3 import posix_ipc posix_ipc.MessageQueue("/a", posix_ipc.O_CREAT)

Meanwhile it works if you use c code: mq_open

The same commands succeed on Ubuntu 18, python 3.6, posix_ioc 1.1.0

[xhuaoe]

I think the reason is the change of /proc/sys/fs/mqueue/msg_max. It changes from 10 to 64 between my two versions of Ubuntu.

ulimit -a shows that the limit for queues is 819200B. With msg max = 64, we can only allocate 1 queue.

[osvenskan]

Thanks @xhuaoe for teaching me about ulimit -a. I don't know Linux very well, and it reveals important information in this case.

I'm curious how msg_max got set to 64 on your system. I just installed Ubuntu 20.04.5 LTS and msg_max=10. That's the value I have seen on many Linux systems over the years. I even made a note of it 9 years ago (https://github.com/osvenskan/posix_ipc/blame/develop/prober.py#L236). And https://man7.org/linux/man-pages/man7/mq_overview.7.html says "The default value for msg_max is 10."

Since my Ubuntu uses the standard value, I'm able to run your test just fine, and I don't see the problem @Rakagami reported either, although I'm not convinced that you're both experiencing the same problem. Here's the output from my machine.

(venv) ubuntu@ip-172-31-85-1:~/posix_ipc$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.5 LTS
Release:    20.04
Codename:   focal
(venv) ubuntu@ip-172-31-85-1:~/posix_ipc$ python -V
Python 3.8.10
(venv) ubuntu@ip-172-31-85-1:~/posix_ipc$ python -c "import posix_ipc; print(posix_ipc.VERSION)"
1.1.0
(venv) ubuntu@ip-172-31-85-1:~/posix_ipc$ python tests/test_message_queues.py
...........................................
----------------------------------------------------------------------
Ran 43 tests in 2.009s

OK
(venv) ubuntu@ip-172-31-85-1:~/posix_ipc$ python
Python 3.8.10 (default, Nov 14 2022, 12:59:47)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import posix_ipc
>>> x = posix_ipc.MessageQueue("/a", posix_ipc.O_CREAT)
>>> x.name
'/a'
>>> x.close()
>>> x.unlink()
>>>
(venv) ubuntu@ip-172-31-85-1:~/posix_ipc$

Detailed Notes

For anyone else reading along (and for Future Me), here's what @xhuaoe is saying. On xhuaoe's system, ulimit -a reports a max of 819200 bytes for POSIX message queues. It's the same on my system (Ubuntu 20.04.5 LTS running on an AWS EC2 instance) --

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 3811
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 3811
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

xhuaoe's system is also configured to allow a max of 64 messages per queue according to /proc/sys/fs/mqueue/msg_max. That's different from my system where it's set to 10 which has been a common value for msg_max on Linux since I wrote a comment about 9 years ago (https://github.com/osvenskan/posix_ipc/blame/develop/prober.py#L236).

When msg_max=64 and /proc/sys/fs/mqueue/msgsize_max=8192 (a typical value), then a single queue created with these defaults will use 64*8192=524288 bytes. Since there's only 819200 bytes allowed for message queues, and 819200/524288=1.5625, there's not enough room for a second message queue on the system.

[osvenskan]

@xhuaoe this should fix the problem on your system: https://github.com/osvenskan/posix_ipc/commit/764d2a374f06e3e489b212d3915e5a7d7541d864

It's in the develop branch if you want to try that out. @Rakagami please let me know if that fixes your problem too.

[xhuaoe]

Thanks @osvenskan, the code change makes sense. I did the same in my code when calling MessageQueue.

[osvenskan]

@xhuaoe This is fixed in version 1.1.1. @Rakagami I am going to assume you experienced the same problem as @xhuaoe described, so I think this is fixed for you too. If not, feel free to re-open this and provide more detail.