search

Sandia-OpenSHMEM / SOS

Sandia OpenSHMEM is an implementation of the OpenSHMEM specification over multiple Networking APIs, including Portals 4, the Open Fabric Interface (OFI), and UCX. Please click on the Wiki tab for help with building and using SOS.
Other
63 stars 53 forks source link

Run ORNL OpenSHMEM Benchmarks #661

Open jdinan opened 6 years ago

jdinan commented 6 years ago

Run OSB in testing, details below:

The ORNL OpenSHMEM Team is pleased to announce the public beta release of Oak Ridge openSHMEM Benchmarks (OSB). https://github.com/ornl-languages/osb/releases

The benchmark suite includes the following benchmarks.

  1. Graph 500 (Single and Multithreaded)
  2. SSCA1 (Single and Multithreaded)
  3. GUPs (Single, Multithreaded / Contexts)
  4. NAS
  5. SHOMS

The OpenSHMEM versions of Graph 500, NAS, and GUPs benchmarks were adapted from the MPI versions and we would like to thank and acknowledge the contribution of original authors.

wrrobin commented 6 years ago

@jdinan, is our plan to add OSB in Travis tests? In such case, should we run all the apps in all envs (e.g. single and multi-threaded)? Or a selected set?

jdinan commented 6 years ago

Depends on how long they run and whether they improve testing coverage over existing tests. Could you review the tests and make a recommendation? For example, we might want to treat these like other performance benchmarks and track performance over time.

wrrobin commented 6 years ago

Sure. I will check and do some test runs.

wrrobin commented 6 years ago

Here are my thoughts. All the tests that I have done are using np = 4, keeping similar to Travis.

  1. Graph 500: Useful application to check since we don't have anything similar yet in our tests. However, it takes too long to run even for smaller scales. So, adding it in Travis would be a disaster.
  2. SSCA1: Not sure whether the benchmark has a hanging issue or it is just too slow. Seems to be not an ideal option for Travis anyway because of the long run time.
  3. GUPs: We already have our own gups implementation in tests.
  4. NAS: It has only IS which we are already getting from ISx.
  5. SHOMS: Useful benchmarking for all the APIs (128 total) but takes a long time to run. We can configure the msg size to make it shorter for Travis. The number of iterations seems to be derived and cannot be configured.

In summary, none of these tests is suitable for Travis. But, we can definitely add them in Jenkins so that we check the tests from time to time. Specially, Graph500 and SHOMS. Please let me know your suggestions.

jdinan commented 6 years ago

Thanks for looking into this. Adding to Jenkins sounds good, but also sounds like we can put this at lower priority.

jdinan commented 5 years ago

Discussed running in Jenkins and determined:

  1. Graph 500 -- Ok, would be good to run in testing
  2. SSCA1 -- Skip
  3. GUPS -- Skip, already have our own version
  4. NAS -- Skip, already have ISX
  5. SHOMS -- ???, good to run if it works reliably