search

cvmfs / cvmfs

The CernVM File System
http://cernvm.cern.ch/portal/filesystem
BSD 3-Clause "New" or "Revised" License
292 stars 131 forks source link

Hi IO-load / CPU activity in CVMFS processes when running (multiple) ALICE MC payloads in parallel #3066

Open sawenzel opened 2 years ago

sawenzel commented 2 years ago
  1. Context

ALICE using a multi-process system for Monte Carlo GRID workflows. Processes are started from CVMFS, scheduled by some graph-scheduler.

  1. Problem description

When multiple such jobs land on the same server, a very high load is observed in CVFMS components (up to multiple cores). In principle this should not be the case since a limited set of executables is loaded and caching mechanism should be at work. Increasing our CVMFS cache size to 50GB has alleviated the problem slightly but not to a full satisfactory extend. Also, it seems that the cache size actually used is below 4GB.

  1. Link to further material

Slides presented at recent workshop: https://indico.cern.ch/event/1079490/contributions/4939522/attachments/2506129/4306981/ALICE-CernVM.pdf

  1. Expectation:

Starting processes repeatedly from CVMFS should be as fast (or close) as loading processes from local disc, when the cache size is sufficient.

  1. Reproducer:

We provide a self-sufficient version of a MC workflow, running in a CC7 container. The setup is provided here: https://cernbox.cern.ch/index.php/s/qCj54zBrB8m362L

Unpack the tar file and follow instructions in README. One job-slot needs 8cores, so best to run it on a server with many cores.

mharvey-jt commented 2 years ago

As a CVMFS user, here are things we've observed that adversely impact performance under high parallel load: 1) Using RHEL7-generation kernel rather than 8 or later. The older kernel just doesn't perform as well. Switching to libfuse version 3 may help too. 2) Heavy use of symlinks. CVMFS currently doesn't support kernel-caching of symlinks 3) Heavy use of statfs(), which can be an expensive operation

mharvey-jt commented 2 years ago

You might try gstacking or perfing the busy cvmfs processes to get a sense for where its time is going.