search

axboe / fio

Flexible I/O Tester
GNU General Public License v2.0
5.23k stars 1.26k forks source link

when run fio Memory consumption increased , #1248

Closed guoweifeng216 closed 3 years ago

guoweifeng216 commented 3 years ago

Please acknowledge the following before creating a ticket

Description of the bug:

when run big size by fio config file ,the system memory increase and led to kill fio. 1、big_size.fio [global] filename=H: size=500g thread ioengine=windowsaio runtime=3600 continue_on_error=none exitall_on_error bs_unaligned random_generator=tausworthe64 rw=randwrite

[random write 1njobs] stonewall direct=0 bssplit=512/10:1k/10:1.5k/10:3k/10:9k/10:127k/10:128k/10:512k/10:1m/10:2m/10 iodepth=32 numjobs=1

2、result H:\fio_test\windows>fio big_size.fio random write 1njobs: (g=0): rw=randwrite, bs=(R) 1B-2048KiB, (W) 1B-2048KiB, (T) 1B-2048KiB, ioengine=windowsaio, iodepth=32 fio-3.27 Starting 1 thread random write 1njobs: Laying out IO file (1 file / 512000MiB) Jobs: 1 (f=0): [w(1)][0.3%][eta 59m:58s] fio: terminating on signal 2 ^Cbs: 1 (f=0): [w(1)][0.3%][eta 59m:58s] H:\fio_test\windows>

2 image

Environment: win pro

fio version: fio-3.27

Reproduction steps 1、run "fio big_size.fio" cmd window 2、open Windows Task Manager 3、find fio process and memory window

sitsofe commented 3 years ago

@guoweifeng216 As mentioned in https://github.com/axboe/fio/issues/1246#issuecomment-878289467 do you see anything wrong with how you are using the filename parameter? Does the amount of memory used scale with the size of your file? Does removing bssplit line impact memory usage? What about removing as many parameters as possible like bs_unaligned? Basically can you remove as many options as possible and simplify your jobfile as much as possible. Please answer ALL of the questions and try ALL of the suggestions. Thanks!

A wild guess is that the axmap to track 512 byte regions across 500 gigabytes is going to be huge: 500*1024**4/512 = 1073741824000. If each region needs 16 bytes to track that's 1073741824000*16/1024.0**4 ~ 15.625 GBytes. The answer might well be "you need more memory" or "you need to turn off fio's randommap".

As this is a "How do I/Why is?" question (look at home much back-and-for there has been) it would be better aimed at the fio mailing list (note: the list only accepts plain text emails).

sitsofe commented 3 years ago

Closing due to lack of reply from reporter. If this issue is still happening with the latest fio (see https://github.com/axboe/fio/releases to find out which version that is) please reopen. Thanks!