Open qezlou opened 3 months ago
After a bit of digging in the code:
open()
and Dataset()
don't use the MPI routines implemented for C (crc/bigfile-mpi.c
) instead invoke the serial version, i.e. src/bigfile.c
. It is already mentioned in the poster [https://github.com/rainwoodman/bigfile/tree/documents] that
Python binding is based on the Non-MPI API, with its own MPI interface;
However, in contrary to the C-MPI, python wrapper opens the file on all processes for read. After a dataset is indexed, the flow is :
Dataset()
in bigfile.__init__.py
---> Dataset.read()
in pyxbigfile.pyx
---> big_file_read_records()
in src/bigfile.h
and not the MPI version , big_file_mpi_read_records
in src/bigile-mpi.h
.
This might be why the I/O is overwhelmed with large number of processes. I am planning to try mpi4py.File.Open
and Read()
and check the I/O.
Yup, that would do it!
One reason the python api did not use the c MPI api was that back then it was hard to create a python surrogate for the c MPI communicator object. Recent mpi4py offers a way ( I recall still not a public api but it is intended to be supported) of creating a python surrogate from a c pointer, so a rewrite based on that could be nice.
On Fri, Aug 16, 2024 at 4:02 PM Simeon Bird @.***> wrote:
Yup, that would do it!
— Reply to this email directly, view it on GitHub https://github.com/MP-Gadget/bigfile/issues/45#issuecomment-2294423954, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABBWTC6ISZWQBFVEWBCPMTZR2AJNAVCNFSM6AAAAABLRELZE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJUGQZDGOJVGQ . You are receiving this because you were mentioned.Message ID: @.***>
@rainwoodman
Hi Yu, Admins on TACC have raised an issue with our code using the
bigfile
python interface. Apparently, it invokes too many queries to the filesystem, similar to your comment on README file:Not quite sure if this https://github.com/rainwoodman/bigfile/pull/43 fixes this though, since my issue is with reading not writing files.
Any starter clues, like which source file I need to dive into to fix this?