too many files open

[seguler]

seguler@seguler-sdk-test:~/test/linux_amd64$ ./blobporter -f /mnt/20x1GB/* -c bigdata -n blobporter/ BlobPorter Copyright (c) Microsoft Corporation. Version: 0.5.25

2018/02/06 22:59:12 container name not specified seguler@seguler-sdk-test:~/test/linux_amd64$ ./blobporter -f "/mnt/20x1GB/*" -c bigdata -n blobporter/ BlobPorter Copyright (c) Microsoft Corporation. Version: 0.5.25

Files to Transfer (file-blockblob) : Source: /mnt/20x1GB/xam Size:1073741824 Source: /mnt/20x1GB/xuma Size:1073741824 Source: /mnt/20x1GB/yuop Size:1073741824 Source: /mnt/20x1GB/rat Size:1073741824 Source: /mnt/20x1GB/btk Size:1073741824 Source: /mnt/20x1GB/hat Size:1073741824 Source: /mnt/20x1GB/luf Size:1073741824 Source: /mnt/20x1GB/pat Size:1073741824 Source: /mnt/20x1GB/sup Size:1073741824 Source: /mnt/20x1GB/tuv Size:1073741824 Source: /mnt/20x1GB/abc Size:1073741824 Source: /mnt/20x1GB/jap Size:1073741824 Source: /mnt/20x1GB/oss Size:1073741824 Source: /mnt/20x1GB/ffa Size:1073741824 Source: /mnt/20x1GB/kat Size:1073741824 Source: /mnt/20x1GB/nap Size:1073741824 Source: /mnt/20x1GB/upmr Size:1073741824 Source: /mnt/20x1GB/ups Size:1073741824 Source: /mnt/20x1GB/ztv Size:1073741824 Source: /mnt/20x1GB/cuv Size:1073741824 --> 43 % [||||||||||...............] Committed Count: 0 Buffer Level: 100%Error while opening the file open /mnt/20x1GB/luf: too many open files 2018/02/06 23:00:05 open /mnt/20x1GB/luf: too many open files

[giventocode]

Thanks for reporting this, more details about the environment will helpful - including the number of cores.

[seguler]

20 cores; Ubuntu 16.04.

Here is what happens to the open file descriptors during the run: seguler@seguler-sdk-test:~$ lsof | grep REG | wc -l lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/113/gvfs Output information may be incomplete. 62 seguler@seguler-sdk-test:~$ lsof | grep REG | wc -l lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/113/gvfs Output information may be incomplete. 25626 seguler@seguler-sdk-test:~$ lsof | grep REG | wc -l lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/113/gvfs Output information may be incomplete. 30415 seguler@seguler-sdk-test:~$ lsof | grep REG | wc -l lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/113/gvfs Output information may be incomplete. 33771 seguler@seguler-sdk-test:~$ lsof | grep REG | wc -l lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/113/gvfs Output information may be incomplete. 62 seguler@seguler-sdk-test:~$

[seguler]

35K-40K open file handles.. Obviously it leaks somewhere... The whole list is full of open handles to the files I am trying to upload

blobporte 38254 38303 seguler 789r REG 8,17 1073741824 4194313 /mnt/20x1GB/luf

[giventocode]

Fixed in #87