search

Weiming-Hu / AnalogsEnsemble

The C++ and R packages for parallel ensemble forecasts using Analog Ensemble
https://weiming-hu.github.io/AnalogsEnsemble/
MIT License
18 stars 5 forks source link

Improve the performance of MPI I/O of CAnEnIO #39

Closed Weiming-Hu closed 5 years ago

Weiming-Hu commented 5 years ago

This looks like a solution to the I/O bounded problem.

Weiming-Hu commented 5 years ago

PnetCDF document can be found here.

netCDF library needs to be linked to PnetCDF library during building. reference 1, reference 2

Weiming-Hu commented 5 years ago

Looks like the PnetCDF library is intended for parrallel access to classfic files.

PnetCDF is a high-performance parallel I/O library for accessing Unidata's NetCDF, files in classic formats, specifically the formats of CDF-1, 2, and 5.

Reference

This solution might not be fit to the problem here because I'm looking for a way to read data into a master process from a shared file in parallel.

Weiming-Hu commented 5 years ago

Check out this window storage implementation of MPI here.

Weiming-Hu commented 5 years ago

It turns out that using MPI to mimic a fork-join parallelization paradigm is going to be bound in performance by one thread at a certain stage of the program execution because after different processes finished reading data, they all need to send data back to the master process, which is a serial procedure.

Unless MPI provides the functionality to directly write into root's memory using the address, the performance will not get improved by using MPI in this way.

cervone commented 5 years ago

Can you ask Alessandro about this? I think he had a neat solution

On Mar 23, 2019, at 11:42 PM, Weiming notifications@github.com<mailto:notifications@github.com> wrote:

It turns out that using MPI to mimic a fork-join parallelization paradigm is going to be bound in performance by one thread at a certain stage of the program execution because after different processes finished reading data, they all need to send data back to the master process, which is a serial procedure.

Unless MPI provides the functionality to directly write into root's memory using the address, the performance will not get improved by using MPI in this way.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FWeiming-Hu%2FAnalogsEnsemble%2Fissues%2F39%23issuecomment-475925503&data=02%7C01%7Cguc18%40psu.edu%7Cf6f9ff176683401863a608d6b00ac881%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636889957702520918&sdata=vw0LLa%2FXn6TyCwOaIjIO6YYk3fzjO0A4x6IwS5%2B2xPk%3D&reserved=0, or mute the threadhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAKREglUGv9nyelZyoboxq8ywUTFuBfuAks5vZvQ1gaJpZM4b3lbP&data=02%7C01%7Cguc18%40psu.edu%7Cf6f9ff176683401863a608d6b00ac881%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636889957702530928&sdata=t0Y%2BtQOL0M1MRY6w6teXp9tTaYpOCxpl73BtqT%2BRHng%3D&reserved=0.

Weiming-Hu commented 5 years ago

I already did. He mentioned to me two mpi solution. One is pnetcdf. But this is actually a package for parallel access with classic format. We are dealing with netcdf4, which MPI is already supported.

The window based solution is the second one. But again it is bound by a single thread to aggregate data.

W

On Sun, Mar 24, 2019 at 2:51 AM Guido Cervone notifications@github.com wrote:

Can you ask Alessandro about this? I think he had a neat solution

On Mar 23, 2019, at 11:42 PM, Weiming <notifications@github.com<mailto: notifications@github.com>> wrote:

It turns out that using MPI to mimic a fork-join parallelization paradigm is going to be bound in performance by one thread at a certain stage of the program execution because after different processes finished reading data, they all need to send data back to the master process, which is a serial procedure.

Unless MPI provides the functionality to directly write into root's memory using the address, the performance will not get improved by using MPI in this way.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub< https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FWeiming-Hu%2FAnalogsEnsemble%2Fissues%2F39%23issuecomment-475925503&data=02%7C01%7Cguc18%40psu.edu%7Cf6f9ff176683401863a608d6b00ac881%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636889957702520918&sdata=vw0LLa%2FXn6TyCwOaIjIO6YYk3fzjO0A4x6IwS5%2B2xPk%3D&reserved=0>, or mute the thread< https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAKREglUGv9nyelZyoboxq8ywUTFuBfuAks5vZvQ1gaJpZM4b3lbP&data=02%7C01%7Cguc18%40psu.edu%7Cf6f9ff176683401863a608d6b00ac881%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636889957702530928&sdata=t0Y%2BtQOL0M1MRY6w6teXp9tTaYpOCxpl73BtqT%2BRHng%3D&reserved=0>.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FWeiming-Hu%2FAnalogsEnsemble%2Fissues%2F39%23issuecomment-475933378&data=02%7C01%7Cwuh20%40psu.edu%7Cca07be07862e45f7465008d6b0253275%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636890071180713028&sdata=dAkcx4%2Bk1rTHFCQBIAjKqkvP1TOXXYx1HFAqszGUkCA%3D&reserved=0, or mute the thread https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAPqR2sa-3JZDLjVInc2ve9B1a6vDZd2Cks5vZyCFgaJpZM4b3lbP&data=02%7C01%7Cwuh20%40psu.edu%7Cca07be07862e45f7465008d6b0253275%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636890071180713028&sdata=94Y7K6QLP%2F0YUUiv7mkHMoKvERTae90iJTV0blsmhBw%3D&reserved=0 .

--

Weiming Hu Geography Ph.D. Student Geoinformatics and Earth Observation Laboratory The Pennsylvania State University http://geoinf.psu.edu https://weiming.ddns.net Email: weiming@psu.edu

("-''-/").___..--''"-. `6 6 ) -. ( ).-.__.) We are..... (_Y_.)' ._ ). `. ``-..-' Penn State! ..`--'..-/ /--'_.' ,' Home of the (il),-'' (li),' ((!.-' Nittany Lions!

Weiming-Hu commented 5 years ago

But Guido, while I will be constantly looking for a solution, I think our situation is still positive. For the current implementation, I can still achieve good efficiency by reading fewer data and doing more computation. The following example profiling from shows the analog generation for 28 test days (1 month) and 2502 grid points using 3 years of data.

User Time for Analog Generator:

Total: 11659.6 seconds (100%)

Reading data: 143.946 seconds (1.23457%)

Computation: 11514.6 seconds (98.7563%)

-- SD: 167.301 seconds (1.43488%)

-- Mapping: 2.7459 seconds (0.0235506%)

-- Similarity: 10969.1 seconds (94.0774%)

-- Selection: 375.498 seconds (3.2205%)

Writing data: 1.06096 seconds (0.00909947%)

Real Time for Analog Generator:

Total: 480.943 seconds (100%)

Reading data: 85.2419 seconds (17.7239%)

Computation: 394.882 seconds (82.1057%)

-- SD: 8.238 seconds (1.71288%)

-- Mapping: 0.0789419 seconds (0.016414%)

-- Similarity: 372.416 seconds (77.4346%)

-- Selection: 14.1485 seconds (2.94183%)

Writing data: 0.819683 seconds (0.170432%)

36 cores are allocated for this job and in total 262792 / 2502 = 105 such jobs are submitted. User time measures how much CPU time I have actually used and real-time measures the wall time. The results are comforting. And I still think our need for MPI is not critical and urgent.

In this way, if we do the math, computing AnEn over the entire NAM takes about (480 / 3600) hours (262792 / 2502) 36 cores = 504 core*hours for using 3 years of serch data and the efficiency should be at least 80%.

So the problem comes back to why administrators observed low efficiency before. Maybe this can be easier if we talk face to face. But I have found out that when invoking analogGenerator on Cheyenne with EnTK, although I have requested for 36 cores and 36 threads are indeed created for each job, only one core is used. The program CPU usage is 100% rather than 3600%. I'm still working on this with Vivek. This looks like something new to them.

Weiming

On Sun, Mar 24, 2019 at 2:51 AM Guido Cervone notifications@github.com wrote:

Can you ask Alessandro about this? I think he had a neat solution

On Mar 23, 2019, at 11:42 PM, Weiming <notifications@github.com<mailto: notifications@github.com>> wrote:

It turns out that using MPI to mimic a fork-join parallelization paradigm is going to be bound in performance by one thread at a certain stage of the program execution because after different processes finished reading data, they all need to send data back to the master process, which is a serial procedure.

Unless MPI provides the functionality to directly write into root's memory using the address, the performance will not get improved by using MPI in this way.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub< https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FWeiming-Hu%2FAnalogsEnsemble%2Fissues%2F39%23issuecomment-475925503&data=02%7C01%7Cguc18%40psu.edu%7Cf6f9ff176683401863a608d6b00ac881%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636889957702520918&sdata=vw0LLa%2FXn6TyCwOaIjIO6YYk3fzjO0A4x6IwS5%2B2xPk%3D&reserved=0>, or mute the thread< https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAKREglUGv9nyelZyoboxq8ywUTFuBfuAks5vZvQ1gaJpZM4b3lbP&data=02%7C01%7Cguc18%40psu.edu%7Cf6f9ff176683401863a608d6b00ac881%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636889957702530928&sdata=t0Y%2BtQOL0M1MRY6w6teXp9tTaYpOCxpl73BtqT%2BRHng%3D&reserved=0>.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FWeiming-Hu%2FAnalogsEnsemble%2Fissues%2F39%23issuecomment-475933378&data=02%7C01%7Cwuh20%40psu.edu%7Cca07be07862e45f7465008d6b0253275%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636890071180713028&sdata=dAkcx4%2Bk1rTHFCQBIAjKqkvP1TOXXYx1HFAqszGUkCA%3D&reserved=0, or mute the thread https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAPqR2sa-3JZDLjVInc2ve9B1a6vDZd2Cks5vZyCFgaJpZM4b3lbP&data=02%7C01%7Cwuh20%40psu.edu%7Cca07be07862e45f7465008d6b0253275%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636890071180713028&sdata=94Y7K6QLP%2F0YUUiv7mkHMoKvERTae90iJTV0blsmhBw%3D&reserved=0 .