How to work with reports

GuutBoy commented 5 years ago

Maybe I have things set up wrong, but it looks like I am only getting a single report even if I have multi configurations in the my MATRIX config. Is this the intended behavior?

Also, its not very clear how to interpret the numbers of the report. I run an experiment with two configurations, each with ten internal iterations. Each iteration logs times for three different tasks. Yet I get only a single report with one number for each task. What are these numbers? An average over the combined 20 iterations of the two configurations?

I would expect to get a separate report for each configuration set up in the MATRIX config. That way I could use MATRIX to compare different parameters for my benchmark. To make it easy to compare, you could do it as separate sheets in the excel file. Also, it would be nice if you could include the raw timings for each configuration in separate sheets in the report file. That would make it really convenient to do further stastistics and comparison on the timings from the benchmark.

liorko87 commented 5 years ago

Can you please share your configurations and the report you received from the system?

GuutBoy commented 5 years ago

Sure config is as follows:

{
  "protocol":"FRESCO-SPDZ",
  "CloudProviders":
  {
    "aws":
    {
      "numOfParties":2,
      "instanceType": "c5.large",
      "regions":["eu-west-1a"],
      "git":
      {
        "gitBranch": ["master"],
        "gitAddress": ["https://github.com/aicis/fresco-matrix.git"]
      }
    }
  },
  "executableName": ["run.sh"],
  "configurations":
  [
    "matrix-distance.jar@-s@spdz@-Dspdz.preprocessingStrategy%DUMMY@-e@SEQUENTIAL_BATCHED",
    "matrix-distance.jar@-s@spdz@-Dspdz.preprocessingStrategy%DUMMY@-e@SEQUENTIAL"
  ],
  "preProcessTask": 8,
  "numOfRepetitions":1,
  "numOfInternalRepetitions":10,
  "IsPublished": "true",
  "isExternal": "true",
  "workingDirectory": ["~/Projects/test-matrix"],
  "resultsDirectory": "/Users/psn/Projects/MATRIX-EXP/MATRIX/results",
  "emails": ["peter.s.nordholt@alexandra.dk"],
  "institute":"The Alexandra Institute"
}

GuutBoy commented 5 years ago

I get the following report

Phases/Number of Parties	2
Evaluation	19.7125
Teardown	1.0625
Setup	11.9

cryptobiu / MATRIX

How to work with reports #43