Unable to use multiple HDF files in separate threads (Pandas + Tables)

pjurgielewicz commented 1 year ago

Hi, I have stepped into the problem while trying to use multiple pandas.HDFStore() objects in the same interpreter but in separate nogil threads. The use case is pretty simple: each thread operates on its own file. The minimum reproduction example is as follows:

from threading import Thread
from multiprocessing import Process
import pandas as pd
import time
import numpy as np

def f1(n: str, t: float):
    fObj = pd.HDFStore(n)
    fObj.append("a",
                pd.DataFrame({"id" : 1, "time": np.zeros(int(10 * t))}),
                data_columns=["id"])
    time.sleep(t)

    fObj.close()

def main():
    t1 = Thread(target=f1, args=("f1.hdf", 1.0))
    t2 = Thread(target=f1, args=("f2.hdf", 2.0))

    # t1 = Process(target=f1, args=("f1.hdf", 1.0))
    # t2 = Process(target=f1, args=("f2.hdf", 2.0))

    t1.start()
    t2.start()

    t1.join()
    t2.join()

if __name__ == "__main__":
    main()

I tested that on Windows and Ubuntu 22.04. On Linux it always segfaults while on Windows the script may fail silently or end up with whole range of different exceptions/outcomes:

childCID = self._g_get_lchild_attr(childname, 'CLASS')
  File "tables\hdf5extension.pyx", line 1072, in tables.hdf5extension.Group._g_get_lchild_attr
tables.exceptions.HDF5ExtError: HDF5 error back trace

  File "..\..\src\H5D.c", line 369, in H5Dopen2
    can't open dataset
  File "..\..\src\H5Dint.c", line 1156, in H5D_open
    not found
  File "..\..\src\H5Dint.c", line 1264, in H5D__open_oid
    can't get layout/pline/efl info
  File "..\..\src\H5Dlayout.c", line 436, in H5D__layout_oh_read
    can't initialize chunk cache
  File "..\..\src\H5Dchunk.c", line 608, in H5D__chunk_init
    can't find object for fapl ID

End of HDF5 error back trace

or

  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\tables-3.6.1-py3.9-win-amd64.egg\tables\attributeset.py", line 248, in __init__
    self.__getattr__(attr)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\tables-3.6.1-py3.9-win-amd64.egg\tables\attributeset.py", line 308, in __getattr__
    value = self._g_getattr(self._v_node, name)
  File "tables\hdf5extension.pyx", line 756, in tables.hdf5extension.AttributeSet._g_getattr
tables.exceptions.HDF5ExtError: Can't get type info on attribute PYTABLES_FORMAT_VERSION in node /.

Or one of the files might end up filled with data or even both of them are empty.

It looks to me like a C API issue of HDF package.

As cross-check I tested the script above in different interpreter & threading/MP configurations:

(nogil) python3.9 + threading -> fails
(nogil) python3.9 + MP -> passes
python3.10 + threading -> passes
python3.10 + MP -> passes

On Ubuntu I tested this with such a set of packages:

autopep8          2.0.0
bitarray          2.6.0
crccheck          1.3.0
dataclasses-json  0.5.7
marshmallow       3.19.0
marshmallow-enum  1.5.1
mypy-extensions   0.4.3
numexpr           2.8.4
numpy             1.22.3
packaging         21.3
pandas            1.3.3
pip               21.3.1
psutil            5.9.4
pycodestyle       2.10.0
pyparsing         3.0.9
python-dateutil   2.8.2
python-vxi11      0.9
pytz              2022.6
PyVISA            1.12.0
PyYAML            6.0
setuptools        58.1.0
six               1.16.0
tables            3.7.0
tomli             2.0.1
typing_extensions 4.4.0
typing-inspect    0.8.0

While on Windows:

bitarray          2.6.0
cffi              1.14.6
crccheck          1.3.0
Cython            0.29.29
dataclasses-json  0.5.7
Jinja2            3.1.2
MarkupSafe        2.1.1
marshmallow       3.18.0
marshmallow-enum  1.5.1
mock              4.0.3
mypy-extensions   0.4.3
numexpr           2.8.0
numpy             1.19.4
packaging         21.3
pandas            1.3.1+1.gafcbd716a9
pdoc              12.2.0
pip               21.3.1
psutil            5.9.3
py-cpuinfo        9.0.0
pybind11          2.6.2
pycparser         2.21
Pygments          2.13.0
pyparsing         3.0.9
python-dateutil   2.8.2
python-vxi11      0.9
pytz              2022.5
PyVISA            1.12.0
PyYAML            5.3.1
pyzmq             24.0.1
setuptools        58.1.0
six               1.16.0
sniffio           1.3.0
tables            3.6.1 AND ALSO 3.5.2 (3.7.0 -> yet unsolved runtime error on package import)
typing_extensions 4.4.0
typing-inspect    0.8.0
wheel             0.37.1

It would be awesome if you (@colesbury) could look into that.

colesbury commented 1 year ago

Thanks for the repro case. I'll take a look at it.

colesbury commented 1 year ago

I'm reading the PyTables documentation (used by pandas HDFStore) and it looks like open & close are not thread-safe even with the GIL. I know your examples passes in Python 3.10 (with the GIL), but the documentation suggest that it's still not thread-safe. Most likely running without the GIL allows many more interleavings and therefore more opportunities for the thread-safety issues to actually occur in a given run.

https://www.pytables.org/cookbook/threading.html

With a lock around the open/close, the repro passes (example here), but I think there are also other thread-safety issues lurking that are related to nogil. For example, PyTables is being installed from the source distribution that includes pre-Cythonized files that aren't thread-safe without the GIL [1]. I'll upload a PyTables wheel built with a compatible version of Cython, but I don't think that will fix the issues with open/close. I'll look into it a bit more and see if I can understand the issues further.

It wasn't clear to me from the issue how you are using Windows. Are you using "nogil" Python on Windows? In general, I would not recommend doing so because I haven't built compatible packages, and some packages require patches to source code.

[1] There is a patched version of Cython that is thread-safe without the GIL. You get this version when you run pip install Cython, but that doesn't help with source distributions that include pre-Cythonized files.

pjurgielewicz commented 1 year ago

@colesbury Thank you for the time you spent on that. As you found out the whole operation of threads will probably break the system or data due to thread-safety issues... But I will try your suggestions anyway since I do not have any better idea than the creation of a localhost server using ZMQ and distributing the data to dedicated archiver processes ;)

The case I presented easily shows that there are a lot of places in Python code that heavily rely on GIL/are far from being thread-safe. It is a long way for nogil accommodation but it is certainly worth the effort of the community.

Regarding the nogil fork on Windows: Yes, I managed to build nogil Python directly from the source along with the required packages (https://github.com/pjurgielewicz/nogil-support-installers, step-by-step installation guide is not included there yet). I searched your repositories looking for packages with nogil tags and tried to build them. In the case of some other packages - I had to either get them directly from PIP or specify the version to be installed by PIP or build them from the original source (fiddling a lot with tags). It was certainly not bread and butter but I established the build pipeline which works for us. The added value is that in the lab we are running simultaneously Linux and Windows machines and can see whether our Dose-3D DAQ behaviour is any different (there are a few Windows quirks but they are easily handled with simple if statements).

colesbury commented 1 year ago

@pjurgielewicz I've looked a bit more and the problems go beyond PyTables and to the underlying HDF5 C library:

At present, the HDF5 library is not thread safe. To allow its use by multi-threaded applications, in the “thread safe” build, the library is equipped with a global lock that
allows only one thread into the library at a time – effectively making the entire HDF5 library a giant critical region.

From https://docs.hdfgroup.org/hdf5/rfc/RFC_multi_thread.pdf

So there's no real parallelism to be had by having multiple threads (in a single process) read or write HDF5 files, even if you were writing your whole application in C or C++. So I think something like the ZMQ strategy might be necessary if you need parallelism in the reading or writing HDF5 files (and not just parallelism in other parts of your system).

Regarding Windows: Can you make the linked GitHub project public (it's currently private)? For context, the scripts to build wheels for Linux (and occasionally macOS) are here. It's probably not terribly useful for Windows, except that it contains links to nogil forks when necessary.

A few months ago, I had experimented with using GitHub actions to build wheels, with the intention of better supporting macOS and maybe Windows. I may revisit that again and having your installers for Windows might help.

pjurgielewicz commented 1 year ago

@colesbury

So there's no real parallelism to be had by having multiple threads (in a single process) read or write HDF5 files, even if you were writing your whole application in C or C++. So I think something like the ZMQ strategy might be necessary if you need parallelism in the reading or writing HDF5 files (and not just parallelism in other parts of your system).

It is not that terrible, we were planning the usage of ZMQ for the dynamic data distribution either way. On the other hand, this will drag our system measurements (I can now start parallel measurements but without data saving ;) ).

Sorry, I forgot that I set this repository to private. It is now public but there is nothing more than a bunch of installers and pre-built site-packages to be copy-pasted into the right directory. As I mentioned, I have an instruction on how to build everything from scratch (not published yet) but I can give no guarantee it would always work or that this is the optimal way.

colesbury / nogil

Unable to use multiple HDF files in separate threads (Pandas + Tables) #95