High memory usage with big files.

[Messiah93]

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all. Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD

Version of s3fs being used (s3fs --version)

s3fs --version

Amazon Simple Storage Service File System V1.91 (commit:unknown) with OpenSSL Copyright (C) 2010 Randy Rizun rrizun@gmail.com License GPL2: GNU GPL version 2 https://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

pacman -Qi fuse

Name : fuse2 Version : 2.9.9-4 Description : A library that makes it possible to implement a filesystem in a userspace program. Architecture : x86_64 URL : https://github.com/libfuse/libfuse Licenses : GPL2 Groups : None Provides : fuse=2.9.9 Depends On : glibc fuse-common Optional Deps : None Required By : s3fs-fuse Optional For : grub Conflicts With : fuse Replaces : fuse Installed Size : 458.21 KiB Packager : Felix Yan felixonmars@archlinux.org Build Date : Sat 16 May 2020 12:21:51 PM CEST Install Date : Sun 13 Nov 2022 10:46:04 AM CET Install Reason : Installed as a dependency for another package Install Script : No Validated By : Signature

Kernel information (uname -r)

uname -r

5.15.78-1-lts

GNU/Linux Distribution, if applicable (cat /etc/os-release)

cat /etc/os-release

NAME="Arch Linux" PRETTY_NAME="Arch Linux" ID=arch BUILD_ID=rolling ANSI_COLOR="38;2;23;147;209" HOME_URL="https://archlinux.org/" DOCUMENTATION_URL="https://wiki.archlinux.org/" SUPPORT_URL="https://bbs.archlinux.org/" BUG_REPORT_URL="https://bugs.archlinux.org/" PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/" LOGO=archlinux-logo

s3fs command line used, if applicable

s3fs backup-extended-us1 /media/backup-extended-us1 -o allow_other -o use_path_request_style -o passwd_file=/etc/passwd-s3fs -o url=https://usc1.contabostorage.com/

/etc/fstab entry, if applicable

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages

Details about issue

VPS: 4vCore, 4Go RAM, 500Go SSD, 200mbits bandwidth @ Archlinux. My s3fs bucket is empty at this point. The file "m93_cpre2022-11-13-_00h00.tgz" size is ~7go.

Before mounting bucket with s3fs, RAM usage is about ~20%. After mounting bucket with s3fs, RAM usage is about ~22%.

cp "/media/backup/m93_cpre2022-11-12-_00h00.tgz" "/media/backup-extended-us1/m93_cpre2022-11-12-_00h00.tgz"

After 1 minute, RAM usage is about 99%. After 2 minutes, VPS take ~1 minute to connect by SSH with new connection. After 4-5 minutes, VPS is no longer responding for 24 hours+, I need to hard restart it.

CPU usage is very low, I/O too, bandwidth same... I don't know if this high memory usage is normal, or if I do something wrong, i'm new to s3fs since 2 days.

Thank's.

[ggtakec]

@Messiah93 I'm sorry for my late reply. Are you downloading or uploading a 7GB file? This is my guess but the tmpfs is used, and there is a possibility that a 7GB file is created as a temporary(or cache file) in the tmpfs volume. In this case, using tmpfs reduces memory usage, so it may match your symptoms. If so, you can specify the use_cache option to place the cache file on a volume other than tmpfs, or limit disk usage with the ensure_diskfree option. Please try to use these option. Thanks in advance for your assistance.

[Messiah93]

Hi,

No i'm not downloading or uploading a 7gb file... I'm using S3FS for my Plex Media Server... I just try to use the "use_cache" option like you said, and when I use it and start a library scan on Plex, S3FS is litterally downloading all my bucket in local to scan each file, and don't delete these until the scan of all files is not finished in plex process... So, without "use_cache" option, it's fully using my ram and swap... I don't know why it happens, I see a lot of peoples using S3FS with plex without problem...

Thank's for help, now I know where come from the problem, now i have to found why.

[ggtakec]

@Messiah93 If you use the use_cache option, create a cache file under the specified directory. If this option is not specified, temporary work files are created and they are created in /tmp(default). If you want to change this /tmp, you can use tmpdir option.

In either case, Plex Media Server(?) scans for files and seems to use cache or temporary files, so I think you may want to put those files in a non-tmpfs directory.