search

cfe-lab / Kive

Archival and automation of bioinformatic pipelines and data
https://cfe-lab.github.io/Kive
BSD 3-Clause "New" or "Revised" License
8 stars 1 forks source link

uploading large dataset files can crash a machine #669

Open wrpscott opened 7 years ago

wrpscott commented 7 years ago

On 2017-10-09 a large file (ca 30 Gb) was uploaded into kive. This caused the load of the machine to become so high that the machine was unresponsive and had to be rebooted. The effect can be reproduced -- uploading a large file causes both httpd and postgres daemons to work excessively.

donkirkby commented 7 years ago

Suggestion: only allow big datasets as external datasets.

jamesnakagawa commented 7 years ago

What file system is being used on the Kive server? I doubt we're on fat32, but ext4 can have a max file size of as little as 16 GiB depending on its allocation block size.

rhliang commented 7 years ago

All the local filesystems are ext4.

jamesnakagawa commented 7 years ago

From the ext4 wiki:

ext4 allocates storage space in units of "blocks". A block is a group of sectors between 1KiB and 64KiB, and the number of sectors must be an integral power of 2. Blocks are in turn grouped into larger units called block groups. Block size is specified at mkfs time and typically is 4KiB. You may experience mounting problems if block size is greater than page size (i.e. 64KiB blocks on a i386 which only has 4KiB memory pages). By default a filesystem can contain 2^32 blocks; if the '64bit' feature is enabled, then a filesystem can have 2^64 blocks.

So probably not this, unless we for some reason specified 1 KiB at mkfs time.