search

TheSkyentist / GELATO

Galaxy/AGN Emission Line Analysis TOol by Raphael Hviding
GNU General Public License v3.0
11 stars 1 forks source link

Multiprocessing #1

Open nisach02 opened 2 weeks ago

nisach02 commented 2 weeks ago

When running GELATO on multiple objects with multiprocessing option, the fitting time per object increases quite a lot. For example, if I run 1 object on a single CPU, it would take ~1 minute to fit the spectra. But if I run 4 objects with 4 CPUs on multiprocessing, it takes more than 4-5 minutes to fit each spectra. This behaviour was found when running GELATO on a cluster.

TheSkyentist commented 2 weeks ago

Thanks for bringing this to my attention! Were the single-core and multi-core tests performed on the same machine? Are you able to reproduce this on a laptop/desktop?

On Sat, Aug 31, 2024 at 00:54 nisach02 @.***> wrote:

When running GELATO on multiple objects with multiprocessing option, the fitting time per object increases quite a lot. For example, if I run 1 object on a single CPU, it would take ~1 minute to fit the spectra. But if I run 4 objects with 4 CPUs on multiprocessing, it takes more than 4-5 minutes to fit each spectra. This behaviour was found when running GELATO on a cluster.

— Reply to this email directly, view it on GitHub https://github.com/TheSkyentist/GELATO/issues/1, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEB6FNEMOSY3VXPENFXT2X3ZUDZ4BAVCNFSM6AAAAABNNIFARKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQ4TQMZUG42DIMY . You are receiving this because you are subscribed to this thread.Message ID: @.***>

nisach02 commented 1 week ago

Were the single-core and multi-core tests performed on the same machine? Are you able to reproduce this on a laptop/desktop?

They were performed on the same machine. I could not reproduce this issue in my personal computer though (MacbookPro i7 2020 version). i.e, no issues at all. The issues could be something to do with the EDR machines running the code in parallel.

TheSkyentist commented 1 week ago

Interesting. There are definitely more hurdles to getting multiprocessing with Python to work on cluster environments. Can you perform a more basic test with Python multiprocessing but not with GELATO but just with a basic example to see if you get a speed improvement or if the same issue persists?

On Mon, Sep 2, 2024 at 15:49 nisach02 @.***> wrote:

Were the single-core and multi-core tests performed on the same machine? Are you able to reproduce this on a laptop/desktop?

They were performed on the same machine. I could not reproduce this issue in my personal computer though (MacbookPro i7 2020 version). i.e, no issues at all. The issues could be something to do with the EDR machines running the code in parallel.

— Reply to this email directly, view it on GitHub https://github.com/TheSkyentist/GELATO/issues/1#issuecomment-2324806220, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEB6FNEGJY2WAMCFEEI6BSDZURUG7AVCNFSM6AAAAABNNIFARKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRUHAYDMMRSGA . You are receiving this because you commented.Message ID: @.***>