[Re] Ten years challenge: Velho and Legrand (2009) - Accuracy Study and Improvement of Network Simulation in the SimGrid Framework

alegrand commented 4 years ago

Original article: https://dl.acm.org/doi/10.4108/ICST.SIMUTOOLS2009.5592 or https://hal.inria.fr/inria-00361031/file/simutools09.pdf

@inproceedings{velho:inria-00361031,
  TITLE = {{Accuracy Study and Improvement of Network Simulation in the SimGrid Framework}},
  AUTHOR = {Velho, Pedro and Legrand, Arnaud},
  URL = {https://hal.inria.fr/inria-00361031},
  BOOKTITLE = {{SIMUTools'09, 2nd International Conference on Simulation Tools and Techniques}},
  ADDRESS = {Rome, Italy},
  YEAR = {2009},
  MONTH = Mar,
  PDF = {https://hal.inria.fr/inria-00361031/file/simutools09.pdf},
  HAL_ID = {inria-00361031},
  HAL_VERSION = {v1},
}

PDF URL: https://github.com/alegrand/reproducibility-challenge Metadata URL: https://hal.inria.fr/inria-00361031/ Code URL: https://github.com/alegrand/reproducibility-challenge

Scientific domain: Computer Science Programming language: C/C++ Suggested editor: @khinsen or @rougier

rougier commented 4 years ago

Thanks for your submission. I'm afraid I've conflict of interest (https://rr-france.github.io/bookrr/) and I cannot edit it.

@labarba Coudl you edit this submission for the Ten Years Reproducibility Challenge (only 1 reviewer needed) ?

rougier commented 4 years ago

@labarba @eroesch Could you edit this submission for the Ten Years Reproducibility Challenge (only 1 reviewer needed)

labarba commented 4 years ago

I don't have field expertise in computer networks, and I'm also slammed with multiple service roles, so I cannot take one more task.

rougier commented 4 years ago

@labarba Ok thanks for the quick answer. @benoit-girard Could you edit this submission ? I know it's not your domain but @alegrand may help in finding reviewers.

benoit-girard commented 4 years ago

OK, let's try that. @alegrand, any reviewer suggestion?

benoit-girard commented 4 years ago

And we have a reviewer: @rgrunbla

benoit-girard commented 4 years ago

Thanks a lot @rgrunbla !

rgrunbla commented 4 years ago

Hi, Thank you for the paper.

I have a few questions / comments:

The creation of the /root/simutools09/01-onelink directory is missing from the instructions (it could probably be put inside the original Dockerfile) in section 3.2, page 13
Formatting: In section 3.3, page 14, the second shell commands, it would be better to adopt a single norm regarding the copy of files into a directory (specify the destination filename or not), and not mix them. I'd go with « specifying the filename » as it underlines it goes into a directory.

Running the commands from page 14, the sweep-parse takes around 13 minutes to give me the shell back on my machine, at which point it crashes:

sh: line 1: 410408 Aborted                 (core dumped) ./gtnets /root/simutools09/01-onelink/tmp/plateforme-1-1.xml /root/simutools09/01-onelink/tmp/deployment-1-1.xml --cfg=workstation_model:compound --cfg=cpu_model:Cas01 --cfg=network_model:GTNets 2>&1 >&/root/simutools09/01-onelink/tmp/tempotrace-1-1.log

I get 10581 lines of output (according to wc -l) before it crashes (and it's always the same number of lines). Could you confirm whether (or not) this is happening from your side ?

While the argument of the separation of concerns (about the input files, perl script, and the simulation code) is a good one (section 3.2), i would also appreciate an overlay "Dockerfile" containing these things to avoid typing the commands (only available in the Pdf, it seems to me, so hardly copy/pastable) needed to reproduce the results of this paper. It would allow me to be sure the aformentioned segfault is a problem from my side, or something else :)
How is the "raw.data" file, mentioned in the section 3.4, generated ? Is that supposed to be generated by sweep-parse.pl ?

Thanks,

Rémy

alegrand commented 4 years ago

Hi Remy,

sorry for the late reply. Huge thanks for being more responsive than me. I've been quite busy over the last days so I haven't been able to peacefully look at what may be wrong. I should finally be able to do this in the following days. Thanks for your patience.

Best,

Arnaud

rougier commented 4 years ago

@benoit-girard @alegrand Gentle reminder

benoit-girard commented 4 years ago

@alegrand

Have you been able to address the questions raised by @rgrunbla? Let us know as soon as possible...

alegrand commented 4 years ago

Hi @rgrunbla, @benoit-girard and @rougier . I really apologize for this unacceptable delay. :(

I have finally been able to find some time to look into this this afternoon (on a completely fresh system on an old laptop as my regular one has recently crashed, 3 weeks after the end of the warranty :( ).

Good catch! There is indeed a problem even though it does not fail in the same way as you.

In my case the script runs to completion (no crash), which is why I had not noticed this. But indeed, when I monitor stderr, there is one of the configuration for which I get the same message as you:

sh: line 1: 363014 Aborted                 (core dumped) ./gtnets /root/simutools09/01-onelink/tmp/plateforme-1-1.xml /root/simutools09/01-onelink/tmp/deployment-1-1.xml --cfg=workstation_model:compound --cfg=cpu_model:Cas01 --cfg=network_model:GTNets 2>&1 >&/root/simutools09/01-onelink/tmp/tempotrace-1-1.log

I have to say I do not understand why it would crash and stop on your machine and run to completion on mine since we're both in a Docker image!

Anyway, I looked into the log to determine, which configuration fails. There is only one, which corresponds to this in the log file:

========> Bandwidth (B) : 1.000000e+05 B/s (Bytes per second)
========> Latency   (L) : 0.50000 s (seconds)
========> Size      (S) : 17000 B (Bytes) 
========> Model     (M) : GTNets
[0.000000] [simix_kernel/INFO] setting 'workstation_model' to 'compound'
[0.000000] [xbt_cfg/INFO] type in variable = 2
[0.000000] [simix_kernel/INFO] setting 'cpu_model' to 'Cas01'
[0.000000] [xbt_cfg/INFO] type in variable = 2
[0.000000] [simix_kernel/INFO] setting 'network_model' to 'GTNets'
[0.000000] [xbt_cfg/INFO] type in variable = 2
<<<<<================================>>>>>
Dumping GTNETS topollogy information
== LINKID: 0
  [SRC] ID: 0, router?: 0, hosts[]: [ 0]
  [DST] ID: 1, router?: 0, hosts[]: [ 1]
>>>>>================================<<<<<
[0.000000] [simix_kernel/INFO] Oops ! Deadlock or code not perfectly clean.
[0.000000] [simix_kernel/INFO] 2 processes are still running, waiting for something.
[0.000000] [simix_kernel/INFO] Legend of the following listing: "<process> on <host>: <status>."
[0.000000] [simix_kernel/INFO] master on S1:  Blocked on condition 0xdead; Waiting for the following actions: 'sleep'(0xdead) 'Task_0'(0xdead) 'sleep'(0xdead).
[0.000000] [simix_kernel/INFO] slave on C1:  Blocked on condition 0xdead; Waiting for the following actions: 'sleep'(0xdead) 'Task_0'(0xdead) 'sleep'(0xdead).
[0.000000] [simix_kernel/INFO] Return a Warning.
** SimGrid: UNCAUGHT EXCEPTION received on (0): category: unknown_err; value: 0
** Cannot cancel GTNetS flow
** Thrown by () in this process
[0.000000] xbt/ex.c:113: [xbt_ex/CRITICAL] Cannot cancel GTNetS flow

**   In action_cancel() at /root/simgrid-3.3/src/surf/network_gtnets.c:349
**   In action_cancel() at /root/simgrid-3.3/src/surf/workstation.c:126
**   In SIMIX_action_cancel() at /root/simgrid-3.3/src/simix/smx_action.c:170
**   In MSG_process_kill() at /root/simgrid-3.3/src/msg/m_process.c:196
**   In MSG_clean() at /root/simgrid-3.3/src/msg/global.c:237
=========================><=========================

This is extremely weird since when looking in the log files of the original article, I should get:

trace-file-1-1.log-114497->==================================================<
trace-file-1-1.log:114498:========> Bandwidth (B) : 1.000000e+05 B/s (Bytes per second)
trace-file-1-1.log-114499-========> Latency   (L) : 0.50000 s (seconds)
trace-file-1-1.log-114500-========> Size      (S) : 17000 B (Bytes) 
trace-file-1-1.log-114501-========> Model     (M) : GTNets
trace-file-1-1.log-114502-[0.000000] [simix_kernel/INFO] setting 'workstation_model' to 'compound'
trace-file-1-1.log-114503-[0.000000] [xbt_cfg/INFO] type in variable = 2
trace-file-1-1.log-114504-[0.000000] [simix_kernel/INFO] setting 'cpu_model' to 'Cas01'
trace-file-1-1.log-114505-[0.000000] [xbt_cfg/INFO] type in variable = 2
trace-file-1-1.log-114506-[0.000000] [simix_kernel/INFO] setting 'network_model' to 'GTNets'
trace-file-1-1.log-114507-[0.000000] [xbt_cfg/INFO] type in variable = 2
trace-file-1-1.log-114508-[S1:master:(1) 6.068680] [msg_test/INFO] Send completed (to C1). Transfer time: 6.068680   Agregate bandwidth: 2801.268151
trace-file-1-1.log-114509-[S1:master:(1) 6.068680] [msg_test/INFO] Completed peer: C1 time: 6.068680
trace-file-1-1.log-114510-[C1:slave:(2) 6.068680] [msg_test/INFO] ===> Estimated Bw of FLOW[1] : 2801.268151 ;  message from S1 to C1  with remaining : 0.000000
trace-file-1-1.log-114511-=========================><=========================

So out of the 7920 tested configurations, there is one that mysteriously fails (the simulation does not even start) and it does not appear to have any particular characteristic.

I'm going to investigate this during the week-end and I'll keep you posted.

benoit-girard commented 4 years ago

Thanks for the update!

alegrand commented 4 years ago

Hello,

This is really crazy. It seems to fail solely for this particular configuration (B=1E5, L=0.5, S=17000, M=GTNets) and I really cannot figure out why. It works like a charm for every value S in [16990,17010] but for 17000 (note that this parameter is the size of the message which is sent from a host to an other in the simulation so this is really weird) !!! I've activated SimGrid's debugging logs, and when you compare the execution for 17000 and 17001, if you ignore pointer address differences, the first difference is right before the deadlock message, because there is no next action end for GTNets whereas there should be one. So I've run gdb, and even attached to the forked child (because the GTNets simulation is forked to determine the next completion) and although a flow was created, it appears that there is no event to dequeue. I already took me quite some time and I do not know GTNets well enough (and I really do not want to debug it) to dig into what could be wrong.

The goal of this challenge was not to fix/improve the old code (especially as this one was a prototype whic has been deprecated when SimGrid moved to NS3 as GTNets was no longer developed). So I think I'll stop there the investigation and amend the article by explaining that I checked that all the (7920) "new" results match the (316800) "old" ones except for one that mysterious configuration where it stops.

While I'm there, I'll take @rgrunbla 's suggestion about the overlay into account and update the URLs with more stable ones (gforge is closing, docker recently announced that they would automatically remove images that had not been accessed in the last few months. :().

The other weird point is that the behavior of this old perl script seems different in my machine (simulation fails silently) and the one of @rgrunbla (stops). If we ever understand what was wrong, I'll also amend the article.

benoit-girard commented 4 years ago

@rgrunbla could you have a look at @alegrand 's answer and provide your reviewer's feedback?