search

researchart / fse16

info about artifacts from fse16
7 stars 3 forks source link

Yoga_PTRacer #16

Open adarshyoga opened 8 years ago

mvdbrand commented 8 years ago

The artefact description introduces the problem domain in a concise and clear way and is very relevant. The discussion on the insightfulness and usefulness are extensive and to the point. The description on usable addressed in a separate section. The installation steps are well described. The description is well structured, detailed and easy to understand. There are no tutorial notes available, only a README containing the same information as the Section 2 of the description. The dependencies are explicit described. The artefact does not provide a virtual machine or is online available. I was unfortunately not able to do the installation myself because of incompatible hardware.

I would classify this artefact as: ? maybe diamond

brusso123 commented 8 years ago

Dear authors,

The artefact is useful to researchers and practitioners that want to detect race condition in multithreaded programs. The tool is can be applied to specific programs written using the Intel Threading Building Blocks C++ library.

The artefact description is written in a bit fast fashion. For example, a concrete example that explains the advantage to use the authors' artefact against existing tools would have been good to read. The authors' FSE 16 paper that has introduced the tool is not cited in the reference list and googling for the name of the authors I was not able to find the paper Adarsh Yoga, Santosh Nagarakatte, and Aarti Gupta - Parallel Data Race Detection for Task Parallel Programs with Locks - FSE 2016. Is this the paper? The source in github it points to is not the one of the PTracer though.

Installation is a bit complex as there are few dependences. A single shell file with all the commands to install and configure all the dependencies would have been good to have here. The tool is openly available in github but the benchmarks used by the authors are contained in a large file stored in google drive. I would suggest to use github for large files (git-lfs) or split it into subfiles.

Barbara

timm commented 8 years ago

@mvdbrand says Maybe diamond? @brusso123 says maybe other? What will @emhill say?

timm commented 8 years ago

It resolving the different reviewer opinions, we were most moved by the way the artifact files are split across multiple sources.

timm commented 8 years ago

Note these labels are still "under discussion" and are still subject to change prior to the final notifications Friday.

adarshyoga commented 8 years ago

@timm Thank you for letting us know. @mvdbrand and @brusso123, thank you for the feedback. @brusso123 -- Yes, the paper associated with the artifact is "Parallel Data Race Detection for Task Parallel Programs with Locks". We have not made the paper available online since we are still working on the final camera-ready version of the paper. Attached with this post is the version of our paper that was submitted for review. Kindly let us know if that would suffice. PTRacer_FSE16.pdf

emhill commented 8 years ago

I attempted to install z3 on my mac, but it failed. Tried again on an ubuntu VM and was able to get it to run. However, the directions in the submitted PDF do not match the README on the web site: https://github.com/rutgers-apl/PTRacer -- is it necessary to download the separate benchmarks? I notice that PTRacer is already 2 GB -- is everything already in test_suite? I was running out of space in the VM I had created for this, and so wasn't able to run on the benchmarks, just the test suite.

This would have been much easier if the authors shared a VM image that was preloaded with everything needed, since the installation process is complex, and contradictory information is available from different web sources.

I feel this tool is useful for others, and since I was able to get it working on ubuntu, I think it should be included.

adarshyoga commented 8 years ago

@emhill , thank you for the feedback. The test_suite consists of unit tests that we use to demonstrate the effectiveness of PTRacer in detecting races in task parallel programs. We show that PTRacer detects all the races in the unit tests and does not report any false positives. We use the benchmarks for performance evaluation. The benchmarks show that PTRacer can be used on long running programs with the performance overhead similar to the state-of-the-art data race detector (SPD3) for task parallel programs.

obaysal commented 8 years ago

Seems like @emhill's review supports "gold", so I will keep it as is.