expose st_blksize in os.FileInfo to avoid hardcoding `bufferSize = 8192` - Githubissues

nim-lang / RFCs

A repository for your Nim proposals.

136 stars 26 forks source link

expose st_blksize in os.FileInfo to avoid hardcoding `bufferSize = 8192` #286

Closed timotheecour closed 3 years ago

timotheecour commented 3 years ago

proposal

in os.nim:

add FileInfo.blockSize
edit os.rawToFormalFileInfo to also expose blockSize computed from Stat.st_blksize
make sameFileContent default to that blockSize so users don't have to guess this

example

when true:
  import posix
  let path = currentSourcePath
  var rawInfo: Stat
  doAssert stat(path, rawInfo) >= 0
  echo (rawInfo.st_blksize,)

on my system (OSX), prints: (4096,) which shows 8192 may not be the optimal.

On other systems it could be larger than 8192, which would incur large overhead (see https://stackoverflow.com/a/237495/1426932) for applications if they used 8192.

(note, block size can also vary per file, but this might be rare though)

links

this came up while reviewing https://github.com/nim-lang/Nim/pull/16008#discussion_r525554699
see also https://stackoverflow.com/a/15540773/1426932

See what Microsoft has to say about IO size: http://technet.microsoft.com/en-us/library/cc938632.aspx. Basically, they say you should probably do IO in 64K blocks. On *NIX platforms, struct stat has a st_blksize member which says what should be a minimal IO block size.

timotheecour commented 3 years ago

=> https://github.com/nim-lang/Nim/pull/16023