ipm_parse does not generate correct HTML file

[jscarretero]

Hi there! I need your help, please :)

I begin by installing Ubuntu 16.04 in a Virtual Machine (for testing purposes), with 1 processor and 2 cores

Then install basic stuff (gcc, g++, and fortran compilers):

sudo apt-get update
sudo apt-get install build-essential autoconf automake git
sudo apt-get install gcc g++ gfortran fort77

Then install OpenMPI:

sudo apt-get update
sudo apt-get install openmpi-bin openmpi-common openssh-client openssh-server libopenmpi-dev

Then install IPM:

cd  #go home
git clone https://github.com/nerscadmin/IPM.git
cd IPM

sudo -s #become root 
./bootstrap.sh
apt-get install libmxml1 libmxml-dev libunwind8

./configure --enable-coll-details --enable-parser

Produces: _################################################ IPM configuration: MPI profiling enabled : yes POSIX-I/O profiling enabled : no PAPI enabled : no CFLAGS : -g -O2 -DHAVE_DYNLOAD -DIPM_COLLECTIVE_DETAILS=1 -DOS_LINUX LDFLAGS : LIBS : -lmxml MPI_STATUS_COUNT : _ucount Fortran underscore : -funderscorepost Building IPM Parser : yes

make
make install
exit #to exit from root

cd ~/IPM/test
make

cd ~/IPM/utils
./make_mxml

And then, when testing:

export LD_PRELOAD=/usr/local/lib/libipm.so
export IPM_REPORT=full
export IPM_REPORT_MEM=yes
export IPM_LOG=full 
unset IPM_HPM
cd ~/IPM/test/test.allgather
mpirun -np 2 allgather

It generates a file jcarrete.1462829706.ipm.xml (around 7KB, file attached) Then, I install ploticus:

sudo apt-get install ploticus

And parse the xml file:

export IPM_KEYFILE=~/IPM/etc/ipm_key_mpi
ipm_parse -html jcarrete.1462829706.ipm.xml

The result is that it generates output to the stdout, but does not generate any directory. If I execute:

ipm_parse -html file jcarrete.1462829706.ipm.xml -o output.html

Then, it generates an html file with almost no information, no navigation options and fails to load these files: https://dl.dropboxusercontent.com/u/408013/protovis.js and http://dl.dropbox.com/u/408013/style.css (output.html attached)

Output files: outputs.zip

Am I doing something wrong? Please help me :)

[njwright]

Hi

The --enable-parser option is not working.

Please use the perl parser bin/ipm_parse

On Mon, May 9, 2016 at 3:09 PM, jscarretero notifications@github.com wrote:

Hi there!

I begin by installing Ubuntu 16.04 in a Virtual Machine (for testing purposes), with 1 processor and 2 cores

Then install basic stuff (gcc, g++, and fortran compilers):

sudo apt-get update sudo apt-get install build-essential autoconf automake git sudo apt-get install gcc g++ gfortran fort77

Then install OpenMPI:

sudo apt-get update sudo apt-get install openmpi-bin openmpi-common openssh-client openssh-server libopenmpi-dev

Then install IPM:

cd #go home git clone https://github.com/nerscadmin/IPM.git cd IPM

sudo -s #become root ./bootstrap.sh apt-get install libmxml1 libmxml-dev libunwind8

./configure --enable-coll-details --enable-parser

Produces:

_################################################ IPM configuration: MPI profiling enabled : yes POSIX-I/O profiling enabled : no PAPI enabled : no CFLAGS : -g -O2 -DHAVE_DYNLOAD -DIPM_COLLECTIVE_DETAILS=1 -DOS_LINUX LDFLAGS : LIBS : -lmxml MPI_STATUS_COUNT : _ucount Fortran underscore : -funderscorepost Building IPM Parser : yes

make make install exit #to exit from root

cd ~/IPM/test make

cd ~/IPM/utils ./make_mxml

And then, when testing:

export LD_PRELOAD=/usr/local/lib/libipm.so export IPM_REPORT=full export IPM_REPORT_MEM=yes export IPM_LOG=full unset IPM_HPM cd ~/IPM/test/test.allgather mpirun -np 2 allgather

It generates a file jcarrete.1462829706.ipm.xml (around 7KB, file attached) Then, I install ploticus:

sudo apt-get install ploticus

And parse the xml file:

export IPM_KEYFILE=~/IPM/etc/ipm_key_mpi ipm_parse -html jcarrete.1462829706.ipm.xml

The result is that it generates output to the stdout, but does not generate any directory. If I execute:

ipm_parse -html file jcarrete.1462829706.ipm.xml -o output.html outputs.zip https://github.com/nerscadmin/IPM/files/255996/outputs.zip

Then, it generates an html file with almost no information, no navigation options and fails to load this files: https://dl.dropboxusercontent.com/u/408013/protovis.js and http://dl.dropbox.com/u/408013/style.css (output.html attached)

Am I doing something wrong? Please help me :)

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/nerscadmin/IPM/issues/18

Nick.

[jscarretero]

Thanks njwright! You were right, and also the problem was with the way I installed IPM and its associated packages. Now I think that everything is working. yessss!! :)

I am attaching a description of the steps to install IPM on an Ubuntu 16.04 x86 operating system, just in case another newbie like me is facing the same problems:

I begin by installing Ubuntu 16.04 in a Virtual Machine (for testing purposes). Then install basic stuff (gcc, g++, and fortran compilers):

sudo apt-get update
sudo apt-get install build-essential autoconf automake git
sudo apt-get install gcc g++ gfortran fort77

Also install OpenMPI stuff:

sudo apt-get install openmpi-bin openmpi-common openssh-client openssh-server libopenmpi-dev Install other stuff required by IPM and XML parser:

sudo apt-get install libmxml1 libmxml-dev libunwind8
sudo apt-get install ploticus

IPM Installation:

cd
git clone https://github.com/nerscadmin/IPM.git
cd IPM
./bootstrap.sh
cd
mkdir -p installations/IPM
cd IPM
./configure --prefix=/home/jcarrete/installations/IPM --enable-coll-details

Will output something like this: ################################################ IPM configuration: MPI profiling enabled : yes POSIX-I/O profiling enabled : no PAPI enabled : no CFLAGS : -g -O2 -DHAVE_DYNLOAD -DIPM_COLLECTIVE_DETAILS=1 -DOS_LINUX LDFLAGS : LIBS : -lmxml MPI_STATUS_COUNT : _ucount Fortran underscore : -funderscore_post Building IPM Parser : yes

make
make install

cd ~/IPM/test
make

ln -s /usr/bin/ploticus ~/installations/IPM/bin/pl

export PATH=$PATH:/home/jcarrete/installations/IPM/bin
export IPM_KEYFILE=~/installations/IPM/etc/ipm_key_mpi

And add it to your .bashrc file with:

echo export PATH=\$PATH:/home/jcarrete/installations/IPM/bin >> ~/.bashrc
echo export IPM_KEYFILE=~/installations/IPM/etc/ipm_key_mpi >> ~/.bashrc

Testing the installation:

export LD_PRELOAD=/home/jcarrete/installations/IPM/lib/libipm.so
export IPM_REPORT=full
export IPM_REPORT_MEM=yes
export IPM_LOG=full 
unset IPM_HPM
cd ~/IPM/test/test.allgather
mpirun -np 8 allgather

It generates a file jcarrete.1462829706.ipm.xml (around 7KB, file attached). And parse the xml file:

export IPM_KEYFILE=~/installations/IPM/etc/ipm_key_mpi
ipm_parse -html jcarrete.1462829706.ipm.xml

It will generate a directory containing HTML files for better visualization purposes.

[hunsa]

Hello,

I've been curios to see whether the latest version of IPM (from github) works as you've described. I've run the MPI version of the NPB (CG, class A) using ipm_version="2.0.5".

As Nick suggested, I used the perl script to convert the xml file into the html output (I should say that the -full option seems to work find). However, using the -html does not work as expected. Ploticus reports some errors.

ipm_parse -html hunold.1464032694.ipm.xml 
# data_acquire    = 0 sec
# data_workup     = 0 sec
#  mpi_pie        = 0 sec
#  task_data      = 0 sec
#  load_bal       = 0 sec
#  time_stack     = 0 sec
#  mpi_stack      = 0 sec
#  mpi_buff       = 0 sec
#  switch+mem     = 0 sec
#  topo_tables    = 0 sec
#  topo_data      = 0 sec
#  topo_time      = 0 sec
# html_all        = 0 sec
# html_regions    = 0 sec
pl proc areadef: error 100: y range is invalid .. likely culprit is yautorange or yrange 
pl proc areadef: error 10: cannot set up plotting area .. likely culprits: bad xrange or yrange, or bad area rectangle 
# html_nonregion  = 0 sec

Do you know this issue or can you successfully convert the xml output? I also attach the xml file as zip. hunold.1464032694.ipm.zip Before I dive into the ipm_parse script, I would like to make sure that the ipm xml output is correct. Thank you. Sascha Hunold

[jscarretero]

Hi Sascha, I haven't seen this error, but the thing is that I have used IPM (v.2.0.5) just for one application (OpenFOAM) and I am actually an IPM newbie. Maybe you could try it with a different application? I guess that the HTML directory is not generated, right? Sorry, if I cannot help much :(

[hunsa]

Hi, The HTML directory was created, but not all images were correctly generated from the Ploticus input files. Could you please try to download the profile from my previous comment and see on your machine whether the HTML files and the images are complete in your case? Thank you -Sascha

[jscarretero]

Hey Sascha, Yep. Same problem happens when running ipm_parse with your xml data in my laptop:

# data_acquire    = 0 sec
# data_workup     = 0 sec
#  mpi_pie        = 0 sec
#  task_data      = 0 sec
#  load_bal       = 1 sec
#  time_stack     = 0 sec
#  mpi_stack      = 0 sec
#  mpi_buff       = 0 sec
#  switch+mem     = 0 sec
#  topo_tables    = 0 sec
#  topo_data      = 0 sec
#  topo_time      = 0 sec
# html_all        = 1 sec
# html_regions    = 0 sec
pl proc areadef: error 100: y range is invalid .. likely culprit is yautorange or yrange 
pl proc areadef: error 10: cannot set up plotting area .. likely culprits: bad xrange or yrange, or bad area rectangle 
# html_nonregion  = 0 sec

And only images for "Load balance by task: memory, flops, timings " and some for "Communication balance by task" are generated...

[jscarretero]

By the way, did you check this "issue": https://github.com/nerscadmin/IPM/issues/14 ?

[njwright]

The problem is your code is measured as running for zero time. So there is nothing to plot.

Either

1) you code is running for a really short time or 2) the timer is broken.

Try a longer running code 1st - that is easier to debug !

On Tue, May 24, 2016 at 9:35 AM, jscarretero notifications@github.com wrote:

Hey Sascha, Yep. Same problem in machine:

` data_acquire = 0 sec data_workup = 0 sec mpi_pie = 0 sec task_data = 0 sec load_bal = 1 sec time_stack = 0 sec mpi_stack = 0 sec mpi_buff = 0 sec switch+mem = 0 sec topo_tables = 0 sec topo_data = 0 sec topo_time = 0 sec html_all = 1 sec html_regions = 0 sec

pl proc areadef: error 100: y range is invalid .. likely culprit is yautorange or yrange pl proc areadef: error 10: cannot set up plotting area .. likely culprits: bad xrange or yrange, or bad area rectangle html_nonregion = 0 sec

`

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/nerscadmin/IPM/issues/18#issuecomment-221329759

Nick.

[hunsa]

Hi Nick, The XML output of IPM (I had attached) seems to be correct (in terms of timings). All measured times are larger than zero. I also tried longer runs (~30 secs) and the problem still exists. -Sascha

[RachelHnext]

Hi, Would appreciate help! (@jscarretero @njwright ) I deploy an xml file, a folder was created with html files containing the graphs and information, But for some reason, information appears only on mpi_wait and mpi_allreduce calls, Whereas mpi_send - 0 calls, although many Mpi_send calls were made that appear and are documented in the xml file. What could cause this? Many thanks in advance!