search

scalameter / scalameter

Microbenchmarking and performance regression testing framework for the JVM platform.
http://scalameter.github.io
Other
508 stars 72 forks source link

Persister leaks personal information #112

Open iphydf opened 9 years ago

iphydf commented 9 years ago

I would like to store the .dat files created by the benchmarks in a public repository. This is currently not possible, because that file contains the classpath, hostname, osArch, and a number of other potentially sensitive pieces of information. It is not clear to me why those settings need to be persisted. In fact, I don't know why any of these settings should be persisted.

Please remove either all settings from the .dat file or limit the persisted settings to non-sensitive data.

axel22 commented 9 years ago

@kjanosz is currently implementing a binary-compatible JSON-based persistor (see this PR).

osArch is logged because it might be useful information about why a benchmark behaves in a certain way. Similarly, a host can provide insight about a regression that is valuable for debugging. For this reason, I would not eliminate them altogether.

We could add a setting which eliminates sensitive data from the benchmark output, but we would first need to decide on all such fields.

iphydf commented 9 years ago

If it's a JSON file, I can do a post-processing step to remove information I don't want to publish. That would solve my problem.