parallel find using pcircle

[adilger]

The IO-500 project (https://github.com/VI4IO/io-500-dev) is currently using https://github.com/hpc/libcircle to implement a basic parallel find (pfind) utility. However, the libcircle library is dependent on OpenMPI and not portable to all environments (e.g. https://github.com/hpc/libcircle/issues/27).

It would be useful for the IO-500, as well as (IMHO) the pcircle project to have a parallel find that at least implements the basic find(1) options. Since fprof has already got the traversal/stat operations implemented, the main logic needed would be filtering the returned files that do not match the specified find attributes before printing.

[fwang2]

As you have correctly identified, pcircle implemented similar algorithms of libcircle in Python, and shielded itself from any particular MPI implementation. More specifically, most of the libcirlce logic has been replaced by circle.py. Parallel find, will likely be built on top of circle.py itself, not fprof. There will be a little bit boilerplate code in both, but not much, I think.

On Tue, Jun 12, 2018 at 7:19 PM adilger notifications@github.com wrote:

The IO-500 project (https://github.com/VI4IO/io-500-dev) is currently using https://github.com/hpc/libcircle to implement a basic parallel find (pfind) utility. However, the libcircle library is dependent on OpenMPI and not portable to all environments (e.g. hpc/libcircle#27 https://github.com/hpc/libcircle/issues/27).

It would be useful for the IO-500, as well as (IMHO) the pcircle project to have a parallel find that at least implements the basic find(1) options. Since fprof has already got the traversal/stat operations implemented, the main logic needed would be filtering the returned files that do not match the specified find attributes before printing.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/olcf/pcircle/issues/55, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEF8uk41i7rzwZ_lvG6gtY4dFYo58jBks5t8EyGgaJpZM4UlTIO .

[JulianKunkel]

We had tried to use the python version a year ago, it was difficult to get that to work on the different HPC systems. I don't remember all the details, though, but various Python issues. Hence we relied on the C-Library.

I have prepared a patch for configure to handle MPICH: https://github.com/JulianKunkel/libcircle/commit/612ae5537d0c284af7a5d349f3a2428a3f11a99b At least on my system, that seems to detect MPICH correctly.

If it does for you, please apply the patch. Thanks.

[fwang2]

Julian - if you are willing to give pcircle another spin and let us know the problem you run into, I am here to help.

On Wed, Jun 13, 2018 at 7:39 AM Julian Kunkel notifications@github.com wrote:

We had tried to use the python version a year ago, it was difficult to get that to work on the different HPC systems. I don't remember all the details, though, but various Python issues. Hence we relied on the C-Library.

I have prepared a patch for configure to handle MPICH: JulianKunkel/libcircle@612ae55 https://github.com/JulianKunkel/libcircle/commit/612ae5537d0c284af7a5d349f3a2428a3f11a99b At least on my system, that seems to detect MPICH correctly.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/olcf/pcircle/issues/55#issuecomment-396907649, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEF8uWoA5CazhAV2wnpMQ5vScj0NO77ks5t8PoKgaJpZM4UlTIO .

[JulianKunkel]

Dear Feiyi, thank you!

Note that my interest is also a version in which I run multiple benchmarks together in a single MPI application, I actually have a test version that does that. With running Python together with a C-Application, I would not be that happy I presume. If possible, I would stick to the C-Version for now.

Regards, Julian

2018-06-13 14:50 GMT+01:00 Feiyi Wang notifications@github.com:

Julian - even if you are willing to give pcircle another spin and let us know the problem you run into, I am here to help.

On Wed, Jun 13, 2018 at 7:39 AM Julian Kunkel notifications@github.com wrote:

We had tried to use the python version a year ago, it was difficult to get that to work on the different HPC systems. I don't remember all the details, though, but various Python issues. Hence we relied on the C-Library.

I have prepared a patch for configure to handle MPICH: JulianKunkel/libcircle@612ae55 https://github.com/JulianKunkel/libcircle/commit/ 612ae5537d0c284af7a5d349f3a2428a3f11a99b At least on my system, that seems to detect MPICH correctly.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/olcf/pcircle/issues/55#issuecomment-396907649, or mute the thread https://github.com/notifications/unsubscribe-auth/ AAEF8uWoA5CazhAV2wnpMQ5vScj0NO77ks5t8PoKgaJpZM4UlTIO .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/olcf/pcircle/issues/55#issuecomment-396944164, or mute the thread https://github.com/notifications/unsubscribe-auth/AE1uyn9TK9mTPEQ5OQzTUcPxwlh0Umyxks5t8RiPgaJpZM4UlTIO .

-- Dr. Julian Kunkel Lecturer, Department of Computer Science +44 (0) 118 378 8218 http://www.cs.reading.ac.uk/ https://hps.vi4io.org/

[fwang2]

understood. thx