search

MP-Gadget / bigfile

A reproducible massively parallel IO library for hierarchical data
BSD 2-Clause "Simplified" License
18 stars 6 forks source link

Too many file access using the Python interfacte #45

Open qezlou opened 3 months ago

qezlou commented 3 months ago

@rainwoodman

Hi Yu, Admins on TACC have raised an issue with our code using thebigfile python interface. Apparently, it invokes too many queries to the filesystem, similar to your comment on README file:

The Python binding under MPI invoked more meta-data queries to the file system than we would like to be, though for small scale applications (thousands of cores) it is usually adequate

Not quite sure if this https://github.com/rainwoodman/bigfile/pull/43 fixes this though, since my issue is with reading not writing files.

Any starter clues, like which source file I need to dive into to fix this?

qezlou commented 2 months ago

After a bit of digging in the code:

  1. 43 doesn't fix the issue we have here which is over-accessing file during file read

  2. The python wrapper for open() and Dataset() don't use the MPI routines implemented for C (crc/bigfile-mpi.c) instead invoke the serial version, i.e. src/bigfile.c. It is already mentioned in the poster [https://github.com/rainwoodman/bigfile/tree/documents] that

    Python binding is based on the Non-MPI API, with its own MPI interface;

However, in contrary to the C-MPI, python wrapper opens the file on all processes for read. After a dataset is indexed, the flow is :

Dataset() in bigfile.__init__.py ---> Dataset.read() in pyxbigfile.pyx ---> big_file_read_records() in src/bigfile.h and not the MPI version , big_file_mpi_read_records in src/bigile-mpi.h.

This might be why the I/O is overwhelmed with large number of processes. I am planning to try mpi4py.File.Open and Read() and check the I/O.

sbird commented 2 months ago

Yup, that would do it!

rainwoodman commented 2 months ago

One reason the python api did not use the c MPI api was that back then it was hard to create a python surrogate for the c MPI communicator object. Recent mpi4py offers a way ( I recall still not a public api but it is intended to be supported) of creating a python surrogate from a c pointer, so a rewrite based on that could be nice.

On Fri, Aug 16, 2024 at 4:02 PM Simeon Bird @.***> wrote:

Yup, that would do it!

— Reply to this email directly, view it on GitHub https://github.com/MP-Gadget/bigfile/issues/45#issuecomment-2294423954, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABBWTC6ISZWQBFVEWBCPMTZR2AJNAVCNFSM6AAAAABLRELZE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJUGQZDGOJVGQ . You are receiving this because you were mentioned.Message ID: @.***>