Support for sparse files in LxFS (flow type checker fails)

[XDTZ]

• A brief description I downloaded 0.40.0 version from its repository and tried to run it.

• Expected results Flow checking my files.

• Actual results (with terminal output if applicable)

  Launching Flow server for /mnt/c/projects/jstraining
  Spawned flow server (pid=34)
  Logs will go to /tmp/flow/zSmntzSczSprojectszSjstraining.log
  Error: Failed to start server. exited prematurely with code 2.
  Could not start Flow server!

• Your Windows build number 15042

• Steps / All commands required to reproduce the error from a brand new installation

  1. download the flow binary from github repo
  2. chmod 755 flow
  3. touch .flowconfig
  4. ./flow

• Strace of the failing command strace flow.zip

• Required packages and commands to install Flow 0.40.0

Edit: attached strace files. Also found flow works fine on another machine which has way more disk space available. Considering reinstalling subsystem now.

[aseering]

Hi @XDTZ , thanks for posting! To generate an strace, do strace -ff -o strace.txt ./flow. This will generate one or more numbered strace.txt files in the current directory. I think an strace would be useful here.

[XDTZ]

@aseering I've updated my post with strace files

[therealkenc]

Looks like flow is asking for ~28GB of shared memory, which appears to be de rigueur these days. Code is here.

stat("/run/shm", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=512, ...}) = 0
statfs("/run/shm", {f_type="TMPFS_MAGIC", f_bsize=4096, f_blocks=13651967, f_bfree=3708102, f_bavail=3708102, f_files=999, f_ffree=1000000, f_fsid={1, 0}, f_namelen=255, f_frsize=4096, f_flags=1062}) = 0
gettimeofday({1488213391, 819285}, NULL) = 0
open("/run/shm/fb_heap-mLEsVg", O_RDWR|O_CREAT|O_EXCL, 0600) = 5
unlink("/run/shm/fb_heap-mLEsVg")       = 0
ftruncate(5, 27927781376)               = -1 ENOSPC (No space left on device)
stat("/tmp/flow", {st_mode=S_IFDIR|0777, st_size=512, ...}) = 0
statfs("/tmp/flow", {f_type=0x53464846, f_bsize=4096, f_blocks=13651967, f_bfree=3708102, f_bavail=3708102, f_files=999, f_ffree=1000000, f_fsid={1, 0}, f_namelen=255, f_frsize=4096, f_flags=1056}) = 0
ftruncate(5, 27927781376)               = -1 ENOSPC (No space left on device)
time(NULL)                              = 1488213391
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=127, ...}) = 0
write(2, "[2017-02-28 00:36:31] We've run "..., 76) = 76
write(2, "SharedMem.Out_of_shared_memory\n", 31) = 31
``

[XDTZ]

Looks like flow is asking for ~28GB of shared memory, which appears to be de rigueur these days. Code is here.

I don't even have that much disk space left (14.7GB available.) Would this be the cause? Maybe I should have posted the issue under Flow's repo?

[benhillis]

@XDTZ - Yes, it might checking with them to see why they're creating a 28 gig file. Our driver is attempting to resize the file but it can't because you're out of disk space.

[therealkenc]

/run/shm is where (by convention) programs stick their backing store for shared memory. So (without looking at the code), typically the next thing that happens, if it got that far, is the file would be mmap()ed. The file is unlinked, but you can pass the descriptor to a child process.

@benhillis - On Real Linux ext2/3/4 you should be able to ftruncate() that file even if you don't have physical space for it, because it's just one big hole. You can ftruncate() a 28 gigabyte file onto a floppy disk. Not sure whether WSL follows this behavior or not.

[therealkenc]

Looks like not. Real Linux:

$ truncate -s 27927781376 foo
$ ls -l foo
-rw-r--r-- 1 ken ken 27927781376 Feb 27 21:00 foo
$ du --block-size=1 foo
0   foo

WSL 15042:

$ truncate -s 27927781376 foo
$ ls -l foo
-rw-r--r-- 1 ken ken 27927781376 Feb 27 21:00 foo
$ du --block-size=1 foo
27927781376     foo

[benhillis]

Good call. Our sparse file support has some holes (pun intended). Improving our support is on our backlog.

[therealkenc]

Thanks. Technically I suppose this is a dup of #622.

[sunilmut]

Thanks @therealkenc. Marking it as such.

[kumarharsh]

If this is dupe of #622 and that issue is closed, is this resolved?

[therealkenc]

is this resolved?

Sadly no. The #622 close actually happened after this one was duped down. I think possibly it got closed in a by-design cleanup sweep (note the other tag).

Pedantically, this would be a feature request for UserVoice. Something like: "Support sparse files in the LxFS". If you are feeling motivated you could make a submission and try to drum up votes. But I'll de-dupe this for you so sparse files has a landing zone here in github (there's a lot worse). I considered asking you to open a new entry, but the sparse 28GB flow use-case in the OP is actually a pretty good one. Or good as any.

[rhengles]

Here is the feature request:

https://wpdev.uservoice.com/forums/266908-command-prompt-console-windows-subsystem-for-l/suggestions/35413555-support-for-sparse-files-in-lxfs-flow-type-checke