Enhancement Idea - Adding a section for compiling and running the example code

[brandonbiggs]

Hi!

Thank you for the great website. I've followed the project for a long time and I really appreciate all of the work to help make HPC programming concepts more accessible. I have been looking at ways to help contribute for a while and I had an idea that I wanted to propose.

One of the things I have struggled with in the past is compiling code. For the most part, single files are usually easy to compile if you've done it once or twice, but there are different compilers, and different parameters that could be included. Because of this, I wanted to propose including a basic compiling section and running the example code section. I forked the repo and created an example -

My proposed (but definitely open to better ways it could be implemented) solution for this was to modify two files: mpi/exercises/exercise_2/data.json and rk.js.

Changes to data.json:

{
    "Type":"Exercise",
    "Technology":"MPI",
    "Title":"Send a message",
    "DirectoryName":"exercise_2",
    "Introduction":"This test is for you to practice the fundamental feature of MPI: sending a message. The application you must develop is made of 2 MPI processes, the first one sends a message and the second one receives it. The message to send is just an integer with the value 12345. The receiver must print the value received. You are free to pick the tag value you want.",
    "Hints":"- MPI_Init\n- MPI_Comm_rank\n- MPI_Comm_size\n- MPI_COMM_WORLD\n- MPI_Send\n- MPI_Recv\n- MPI_INT or MPI_INTEGER\n- MPI_Status\n- MPI_STATUS_IGNORE\n- MPI_Finalize",
    "Difficulty":0,
    **"Compile-C": "gcc -Wall exercise_2.c –o exercise_2",
    "Compile-C++": "g++ ...",
    "Compile-F90": "gfortran ...",
    "Compile-F08": "gfortran ..."**
}

and rk.js starting on lines 1797:

// Compile Page Section
const CompilePage = document.createElement('article');
CompilePage.classList.add("PageArticle", "Version", RK.LANGUAGES[Language])
MainSectionDiv.appendChild(CompilePage);

const CompilePageHeader = document.createElement('div');
CompilePageHeader.classList.add("PageArticleHeader")
CompilePage.appendChild(CompilePageHeader);

const CompileHeaderTitle = document.createElement('h2');
CompileHeaderTitle.innerText = 'Compile';
CompilePageHeader.appendChild(CompileHeaderTitle );

const CompilePageBody = document.createElement('div');
CompilePageBody.classList.add("PageArticleBody");
CompilePage.appendChild(CompilePageBody);

const CompilePageConsole = document.createElement('div');
CompilePageConsole.classList.add("ConsoleOutput");
CompilePageBody.appendChild(CompilePageConsole);

const CompilePageBodyContent = document.createElement('p');
CompilePageBodyContent.innerHTML = RK.InterpretMarkdown(RK.InsertCrossReferencesFromTechnology(Entry['Compile-' + Language])) || "None: you are on your own on this one.";
CompilePageBody.appendChild(CompilePageBodyContent);

If this is something that could be useful to the project, I'd be happy to make the updates and put in a pull request.

Thanks again for all your work!

[rookiehpc]

Hi @brandonbiggs,

Thank you for the kind words and your will to contribute.

Showing the compilation commands (as well as the execution command mpirun or the standardised mpiexec) could be useful indeed, especially to HPC rookies as well as programmers discovering compiled languages in general.

Regarding their integration, the current JSON structure in examples could support it directly by using the "Hints" section. So far, it is used solely for coding hints, such as MPI routines, but it could also give hints about how to compile the code. Similarly to other hints, compilation hints could be removed in more advanced examples, so that programmers can practice from memory (and can always refer back to previous exercises if needed).

Note: when compiling MPI programs, however, one tends to use wrappers (mpicc, mpif90, mpif08 ...) which automatically handle the MPI set up behind the scenes (include paths, library paths etc...).

What do you think?

Kind regards, RookieHPC

[brandonbiggs]

Note: when compiling MPI programs, however, one tends to use wrappers (mpicc, mpif90, mpif08 ...) which automatically handle the MPI set up behind the scenes (include paths, library paths etc...).

Good point on this. I agree that the wrappers make it easier for compiling when using MPI and would be a good example to use instead of/in addition to the base gcc, g++, gfortran, etc with MPI.

Regarding their integration, the current JSON structure in examples could support it directly by using the "Hints" section. So far, it is used solely for coding hints, such as MPI routines, but it could also give hints about how to compile the code. Similarly to other hints, compilation hints could be removed in more advanced examples, so that programmers can practice from memory (and can always refer back to previous exercises if needed).

Is the "Hints" section language specific? It appears to just be a single section regardless of what language is being used? I thought maybe a dedicated compile section might be useful since there are multiple languages that can be picked from so the compile section could show the language specific compile command or no compile in advanced examples?

Thanks for your feedback! :)

[rookiehpc]

The hints section is not language-specific indeed. Given that compilation is not specific to an exercise, since the same compilation / execution command pair can be used throughout all exercises, maybe it could be the actual topic of an exercise dedicated to it, acting as a tutorial before other exercises. For example, it could be placed before the hello world exercise, and would consist in compiling and running a minimal MPI code provided. Since the focus of the exercise would be on compilation and execution, the exercise description could detail the full list of wrappers and compilers available, so that each reader can find their own.

The reason for the full list is that, a challenge with the mention of MPI wrappers, or compilers in general, is fairness. For instance, the GCC compiler is commonly assigned to the mpicc wrapper, whereas Intel typically uses mpiicc and Cray uses cc, to name a few. The same applies with FORTRAN codes across mpifort, mpiifort and ftn, respectively.

However, the RookieHPC project tries to not take position on prioritising a compiler suite, MPI wrapper or MPI implementation. Therefore, mentioning compilers, wrappers or implementations in a fair way will likely require mentioning them all.

What do you think about having an introductory exercise acting as an entire mini-tutorial on compiling and executing code? :)

[brandonbiggs]

The hints section is not language-specific indeed. Given that compilation is not specific to an exercise, since the same compilation / execution command pair can be used throughout all exercises, maybe it could be the actual topic of an exercise dedicated to it, acting as a tutorial before other exercises. For example, it could be placed before the hello world exercise, and would consist in compiling and running a minimal MPI code provided. Since the focus of the exercise would be on compilation and execution, the exercise description could detail the full list of wrappers and compilers available, so that each reader can find their own.

I think that's a really good idea.

The reason for the full list is that, a challenge with the mention of MPI wrappers, or compilers in general, is fairness. For instance, the GCC compiler is commonly assigned to the mpicc wrapper, whereas Intel typically uses mpiicc and Cray uses cc, to name a few. The same applies with FORTRAN codes across mpifort, mpiifort and ftn, respectively.

However, the RookieHPC project tries to not take position on prioritising a compiler suite, MPI wrapper or MPI implementation. Therefore, mentioning compilers, wrappers or implementations in a fair way will likely require mentioning them all.

Yup, I totally agree. While folks have their preferences, if someone doesn't have access to a specific compiler or doesn't know which to use, no reason to push our preference on them.

What do you think about having an introductory exercise acting as an entire mini-tutorial on compiling and executing code? :)

I think that's a great idea. Then the "hints" section, like you mentioned previously, could be used exclusively for compiling rather than compiling + other mpi related topics.

[rookiehpc]

Just to quickly double-check:

I think that's a great idea. Then the "hints" section, like you mentioned previously, could be used exclusively for compiling rather than compiling + other mpi related topics.

Did you phrase it the wrong way around? That is, did you mean "the hints section, <...>, then could be used exclusively for programming hints such as MPI routines or OpenMP directives and not such programming hints as well as compilation-related commands"?

[brandonbiggs]

Just to quickly double-check:

I think that's a great idea. Then the "hints" section, like you mentioned previously, could be used exclusively for compiling rather than compiling + other mpi related topics.

Did you phrase it the wrong way around? That is, did you mean "the hints section, <...>, then could be used exclusively for programming hints such as MPI routines or OpenMP directives and not such programming hints as well as compilation-related commands"?

Sorry for my poor wording. I meant that if there was an exercise based around compiling, the hints section can be used to focus on compilation commands rather than focusing on hints for the code since that isn't the point of the exercise. Does that line up with what you were thinking?

[rookiehpc]

Yes it does. Thank you for the clarification.

I misunderstood you the first time, assuming your comment applied to every exercise, which sounded contradictory with the idea of dedicating an exercise to compilation if compilation hints were going to appear in every exercise's hints section anyway. My apologies for the confusion.

So, to summarise:

• In the introductory exercise for compilation and execution: both aspects can be covered in detail. Since there is no actual documentation on the website about these, the Introduction section of the exercise could present the various wrappers and scripts options, and act as a mini-tutorial. If need be, the hints section can be used to remind the user of the main commands.
• Other exercises: compilation aspects are ignored and hints section keep the focus on programming hints such as MPI routines and OpenMP directives.

Does the above sound correct?

Regarding the source code provided in the introductory exercise itself, what about a program that just displays the number of processes (for MPI) or threads (for OpenMP) present in the application?

• Someone learning how to compile is likely at the phase of discovering the corresponding technology, whether it is MPI or OpenMP. Therefore, keeping the source code to a minimum could help minimise the number of lines containing concepts or routines new to the reader.
• It would make it easy for someone to verify they have correctly issued the execution command. For example, issuing ./MPIProgram instead of mpiexec -n 4 ./MPIProgram would display 1 instead of 4. Similarly with OpenMP and OMP_NUM_THREADS.