search

hilbert / hilbert-docker-images

Application with a dynamic choice of docker containers to run
Apache License 2.0
22 stars 8 forks source link

Docker prototype benchmarking #5

Closed porst17 closed 9 years ago

porst17 commented 9 years ago

Now that we have basic support for graphics, audio, etc., we have to test the various requirements of some apps in a real world setting.

The following things have to be tested and compared between host and docker container:

This should also be tested on a physical system, not only in a VM.

If there are significant differences between host and docker container, we have to find out, if this can be fixed and how, and how this influences or infrastructure.

Independent issues should be opened for issues that arise during benchmarking. This issue is for general discussion of the benchmark, posting of benchmark results and its consequences.

malex984 commented 9 years ago

It seems that fbsuite is mostly unusable since it contains ELF 32-bit dynamic executables: piozone, bonnie++, stream, cachebench, which cannot be run (i suspect: because of incompatible modern amd64 linux kernels or libc used by our images) :(

Fortunately after some searching i was able to uncover these tools:

Although they look ancient i can build and bundle them for us (64 bit modern libc/linux kernel) but would like to look for more recent alternatives... ok?

Besides:

In order to be able to compare benchmarks i would propose to measure them as follows:

  1. on a real Linux machine with good GPU and CPU with lots of memory
  2. inside virtual box VM (on the same Linux machine) using 100% CPU and enough memory, with VB Guest Additions and 3D Acceleration enabled
  3. testing apps inside docker container using the same original X11 (as in 1.)
  4. running X11 from x11 image with testing apps added to it
  5. like 4. but testing apps in a separate container

what do you think?

porst17 commented 9 years ago

Just one short comment: direct rendering: Yes just means that X11 is bypassed for the most part and most OpenGL calls are send directly to the OpenGL implementation (usually to the device driver via a certain library). Also the Mesa software renderer is such an implementation. Check if glxinfo | grep "renderer string" gives something with Gallium, which is the Mesa software renderer or Chromium, which is the OpenGL driver of VirtualBox, which in turn gives access to the 3D acceleration of the VM host.

Benchmarking Gallium is certainly not very meaningful.

malex984 commented 9 years ago

glxinfo | grep "renderer string gives Gallium 0.4 on llvmpipe (LLVM 3.4, 128 bits) independent on whether LIBGL_ALWAYS_INDIRECT was set or not.

setting LIBGL_ALWAYS_INDIRECT=1 indeed results in direct rendering: No (LIBGL_ALWAYS_INDIRECT set) and GpuTest would not even start anymore. :-(

malex984 commented 9 years ago

note that i currently have x11 image running Xorg (no Guest Additions) in a separate container and GpuTest in another container.

malex984 commented 9 years ago

my understanding was that we use VirtualBox for testing the proof-of-concept and we are not planning to run heavy 3D or GPU-demanding applications from under VirtualBox due to its limitations... or am i wrong?

malex984 commented 9 years ago

i am planning to run GpuTest (with and without LIBGL_ALWAYS_INDIRECT) using x11 with guest additions (e.g. using vboxvideo) under my testing VM... and will also check the glxinfo...

should i skip that?

porst17 commented 9 years ago

The point of accessing the VirtualBox dri module is that it provides hardware acceleration (though virtualized), while Mesa with Gallium 0.4 on llvmpipe (LLVM 3.4, 128 bits) is just software rendering. The Proof-Of-Concept must prove that it can actually run our target applications on docker, i.e. this is part of the concept to be proven. If we are able to use the VirtualBox driver properly (meaning: reliably, without using ugly hacks), we should be able to use the Intel, NVidia and AMD drivers as well when the platform is running a physical, not a virtual host.

Regarding your request about demanding 3D applications in VirtualBox: There are two kinds of demands: OpenGL feature set and OpenGL speed. The feature set is limited to OpenGL 2.x in VirtualBox, but the speed should be comparable to OpenGL 2.x speed on the VM host due to OpenGL call forwarding (+ some overhead which makes some simple applications like glxgears quite slow). Hence, we could even add some of the demanding OpenGL 2.x applications of IMAGINARY to the Proof-Of-Concept if we get VirtualBox 3D acceleration to work properly in a docker container.

malex984 commented 9 years ago

Hm, do you think we should try 3D benchmarking on Intel & ATI GPU's as well?

porst17 commented 9 years ago

No, I think the result will be similar. We now know what actually affects the performance of programs running in a container (esp. related to 3D) due to your extensive testing.