"RuntimeError: fdopen() failed unexpectedly" when reading too many gml files

[RogerH2018]

Describe the bug When I use the "ig.Graph.Read_GraphML" in the for loop, the following error occurs. Traceback (most recent call last): g = ig.Graph.Read_GraphML(os.path.join(path, item)) RuntimeError: fdopen() failed unexpectedly

[ntamas]

How many iterations do you need to have in the for loop to trigger the problem? Can you post a short, self-contained example that reproduces the problem on your computer? (By "self-contained", I mean that something I can run on my own machine without further modifications, assuming that igraph itself is installed).

Also, the title of the issue mentions GML as the problematic file format, while the code example uses Read_GraphML. Which format are you having problems with?

[RogerH2018]

The code is very simple. There are eight folders, and each contains about 200 graphml files. path = '' for item in os.listdir(path): g = ig.Graph.Read_GraphML(os.path.join(path, item)) The problem is on Windows. Does Windows have a limit on the number of files that can be opened? I think the bug on Windows should be fixed.

[szhorvat]

Can you please share a complete reproducible example that we can run verbatim, with no modifications? Please don't just give a rough idea of the code you ran, but show a complete script, which, if run, produces the error.

[ntamas]

For the record, I have prepared a folder containing 2000 GraphML files and then used the following script in the same repository in an attempt to try and reproduce this on Windows:

#!/usr/bin/env python3

import psutil

import igraph as ig
import os

for item in os.listdir("."):
    if item.endswith(".graphml"):
        g = ig.Graph.Read_GraphML(item)

proc = psutil.Process()
print(proc.open_files())

The psutil part at the bottom is meant to print the list of open files as I suspected that somehow igraph forgets to close the files. The script ran to completion without problems, and the list was empty at the end. So this may be dependent on the exact Windows version being used.

I agree with @szhorvat, we need a minimal reproducible example here to proceed with any kind of fix.

[szhorvat]

@RogerH2018 There is indeed a limit on concurrently open files but if you indeed used the code you show, files get closed as soon as they're read. This is why I asked for the example--to see what you did exactly.

[RogerH2018]

@ntamas Thank you for your patient reply. I guess it is a Windows version issue or a Windows parameter issue. I try the script again. Then I found the maximum number of files it can read is 509. Once you add another one, the program will report the following error. RuntimeError: fdopen() failed unexpectedly Each GraphML file is about 100Kb in size.

[ntamas]

509 is strangely close to 512, which is the allowed maximum number of open files in Windows for programs linked with the C library. (509 + stdin + stdout + stderr = 512). Which igraph version are you using?

[RogerH2018]

@ntamas python-igraph 0.9.11

[szhorvat]

The obvious question, which shouldn't need to be asked, is have you tried the latest version?

Sorry to be blunt, but by having deleted parts of the bug report form instead of filling it out (version information, reproducer), not doing due diligence for a bug report (use the latest version, provide a complete mininimal example), you are wasting the time of people who are already overwhelmed.