search

Open-CAS / standalone-linux-io-tracer

Standalone Linux IO Tracer (iotrace) is a tool for block device and filesystem I/O tracing
BSD 3-Clause "New" or "Revised" License
58 stars 22 forks source link

tool will fail to start after couple of runs #254

Closed gwnet closed 4 years ago

gwnet commented 4 years ago

===error when start run agian=== intel@intel:~/wayne/tmp$ sudo iotrace --start-tracing --devices /dev/cas1-1 --time 30 --size 1024 modprobe: FATAL: Module iotrace is in use. { "timestamp": "2020-06-02 12:18.064", "system": "iotrace", "severity": "Error", "trace": "Cannot close iotrace kernel module" } modprobe: FATAL: Module iotrace is in use.

===try use modprobe -r to work around but failed==== intel@intel:~/wayne/tmp$ sudo modprobe -r iotrace modprobe: FATAL: Module iotrace is in use.

gwnet commented 4 years ago

I found the pattern, if I run 4K IO tests on it, it will hang like this.

intel@intel:/mnt/data1$ sudo iotrace --start-tracing --devices /dev/nvme1n1 --time 45 --size 1024 { "timestamp": "2020-06-03 04:02.805", "system": "iotrace", "severity": "Information", "trace": "Trace collecting has started" } { "timestamp": "2020-06-03 04:03.894", "system": "iotrace", "severity": "Information", "trace": "Trace collecting has completed" }

^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C ^C^C^C ^C^C^C

mmkayPL commented 4 years ago

Could you post some additional info about the environment - Version of iotrace and octf (post results of iotrace -V) Kernel version (uname -r) OS (Fedora/Ubuntu/Centos version) Any kernel logs around the time of the tracing

gwnet commented 4 years ago

my OS is centOS7 kernel is 3.10 I used to check dmesg, no info related. I will get iotrace -V next time. I use the minio to run the 4K small objects. then I use iotrace, then it hang there. it can stable reproduce the issue. after setup minio, then run below benchmark test. intel@intel:~/wayne/s3-benchmark$ sudo ./s3-benchmark -a minioadmin -b jeff-s3-benchmark -s minioadmin -u http://127.0.0.1:9000 -t 24 -d 20 -z 4k S3 benchmark program v3.0 Parameters: url=http://127.0.0.1:9000, bucket=jeff-s3-benchmark, duration=20, threads=24, loops=1, size=4k Wed, 03 Jun 2020 10:08:09 GMT Loop 1: PUT time 20.0 secs, objects = 158201, speed = 30.9MB/sec, 7908.9 operations/sec. Wed, 03 Jun 2020 10:08:29 GMT Loop 1: GET time 20.0 secs, objects = 119104, speed = 23.3MB/sec, 5954.0 operations/sec. Wed, 03 Jun 2020 10:08:49 GMT Loop 1: DELETE time 19.8 secs, 7999.9 operations/sec. Benchmark completed.

mmkayPL commented 4 years ago

Thanks for the information, I am able to consistently reproduce it on my end on the current master so no extra information is necessary for now