search

ThomasKaiser / sbc-bench

Simple benchmark for single board computers
BSD 3-Clause "New" or "Revised" License
655 stars 78 forks source link

README.md needs updating to show full range of supported OSes #97

Open alansill opened 2 months ago

alansill commented 2 months ago

Initially I was discouraged abut installing this as I don't use Debian or Ubuntu on most of the SBC systems in our student lab, so instead set about installing each of the components separately. After each of these installed fine on their own, I got inspired to go back and try to install the full project. It worked fine for me out of the box with Rocky Linux 9.4 and ran just fine to completion on the Apple Mac Intel i7 mini that I was using for the test.

I'd be happy to put in a pull request to update the README.md but don't know the full range of operating systems this tool will support. At minimum, the documentation should be updated to show more OS options if a full range is known.

ThomasKaiser commented 2 months ago

Well, the problem is not where this tool runs but which scores it produces there.

It should be generally known but is not: most benchmarks do overly depend on the version of OS, compiler, libs and so on. As such you can't compare scores that were made in different years (as in: different OS that uses different compiler version, different libs and so on). This is also true for sbc-bench's main metric: 7-zip.

That's the main reason I kept the warning when not running on Debian/Ubuntu since both distros follow the 'all software outdated as hell' principle (calling it 'stable') and they shipped for years with the very same and unmaintained p7zip version 16.02 (until recently).

Now I'm trying to detect the 7-zip version the distro is using and if it's not 16.02 try to build 7-zip 16.02 directly. But due to lack of resources I can not test this on all those distros where sbc-bench might execute flawlessly.

As such no idea how to proceed: advertise it for more distros (risking users generating bogus numbers) or stay conservative?

alansill commented 2 months ago

Hmm, I see your motivation for the approach. But other approaches might work also, such as labeling the OS and lib versions used. Fedora, the upstream of the RH-like distros, is generally as "up to date" as Ubuntu and Debian, and versions of each of these change over time, so labeling rather than exclusion seems to be the right approach to me. Also a genral "YMMV depending on these factors" seems to me to be a better approach than just trusting to one line of distros. Just suggestions.

I have a bunch of students working on similar topics in our lab, and weʻre likely to produce tests and can even do these with different distros. Building a structure that lets people sort results by distro, etc., might be an option. Let me knwo if you see a place where we could put some effort that would be welcome.