andrzej-at-mazurkiewicz-org commented 2 years ago

"OSError: [Errno 24] Too many open files" after 508 iterations of with PrologMQI() as mqi:
with mqi.create_thread() as pth:

my code.

Hello.

Yesterday I tried to use swiplserver in a loop to run prolog query for a set of about 4000 identical queries with different data. I got " with PrologMQI() as mqi: File "/usr/local/lib/python3.9/dist-packages/swiplserver/prologmqi.py", line 326, in enter self.start() File "/usr/local/lib/python3.9/dist-packages/swiplserver/prologmqi.py", line 412, in start self._process = subprocess.Popen( File "/usr/lib/python3.9/subprocess.py", line 951, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.9/subprocess.py", line 1722, in _execute_child errpipe_read, errpipewrite = os.pipe() OSError: [Errno 24] Too many open files" It seems that '__exit_\' (if I am correct) does not close file descriptors. It does not affect single runs but the problem emerges during a longer iteration.
Since file uploader does not support .pl and .py suffixes I added additional .txt suffix to them.
Files: dolki-swiplserver.py.txt max_min.pl.txt log.dolki-swiplserver.py.log Regards Andrzej Mazurkiewicz. P.S.
My environment is on local computer and uses postgresql 13. If you need access to it, please let me know. I will prepare an account and a port forwarding from a fixed ip.
Another option is to modify source code, so that it does not use the database. If you need it - please let me know.

EricZinda commented 2 years ago

Hi @andrzej-at-mazurkiewicz-org ! Sorry I was out yesterday I'll take a look at this today.

andrzej-at-mazurkiewicz-org commented 2 years ago

Hi @andrzej-at-mazurkiewicz-org ! Sorry I was out yesterday I'll take a look at this today. Hello. No problem. Currently pyswip is OK for me and seems to be faster. So the problem is not urgent. However, it would be good to put everything in order. Regards Andrzej Mazurkiewicz

EricZinda commented 2 years ago

There was indeed a leak. I didn't realize that Popen wouldn't close its stdout/stderr streams. I was able to repro with this test on my side:

    def skip_test_launch_stress(self):
        for index in range(0, 10000):
            print(index)
            with PrologMQI(
                self.launchServer,
                self.serverPort,
                self.password,
                self.useUnixDomainSocket,
                prolog_path=self.prologPath,
            ) as server:
                with PrologThread(server) as prolog_thread:
                    prolog_thread.query("true")

I'll be checking in a fix shortly

EricZinda commented 2 years ago

OK, this fixed the leak I found, if you get a chance, replace your prologmqi.py file with this one and see if it fixes the leak you are hitting: prologmqi.py.txt

andrzej-at-mazurkiewicz-org commented 2 years ago

Works OK. I am closing the issue. Thanks. Regards, Andrzej Mazurkiewicz

JanWielemaker commented 2 years ago

This issue has been mentioned on SWI-Prolog. There might be relevant details there:

https://swi-prolog.discourse.group/t/ann-swi-prolog-8-4-1-stable/4661/1

EricZinda commented 2 years ago

I have posted a new official version of swiplserver on pypi.org that includes this fix: https://pypi.org/project/swiplserver/1.0.1/

JanWielemaker commented 2 years ago

This issue has been mentioned on SWI-Prolog. There might be relevant details there:

https://swi-prolog.discourse.group/t/ann-swi-prolog-8-5-3/4688/1

SWI-Prolog / packages-mqi

OSError: [Errno 24] Too many open files after 508 iterations of with PrologMQI() as mqi: with mqi.create_thread() as pth: #13

my code.