Potential Memory Leak

[pauldeng]

Problem

Thank you for the great project.

I see that memory usage growing gradually and never released until I restart the application. This could be memory leak but need your help to guide me on how to debug.

My application is data collection from thousands of embedded devices. Embedded devices HTTP Post file to DUFS.

Memory usage increase rate is like 0.5MB per minute. Last time, it reached 2.1GB of usage out of 4GB total and DUFS CPU usage skyrocket. Restart DUFS fix the problem temporarily.   Configuration

serve-path: '/home/ubuntu/dufs/storage/'
bind:
  - 0.0.0.0
port: A_PORT_NUMBER
auth:
  - USERNAME:PASSWORD@/:rw
allow-upload: true
allow-delete: false
allow-search: true
allow-symlink: false
allow-archive: true
log-format: '$remote_addr "$request" $status $http_user_agent'

Log

Not sure if this is the root cause of the problem:

...
2024-05-31T09:34:20+10:00 ERROR - 44.216.145.121 "PUT /1754769701/28617664.bin" 500 UBLOX-HttpClient V2.0 error reading a body from connection
...

Environment:

• Dufs version: 0.41.0
• OS info: Ubuntu 20.04
• Proxy server: no

[sigoden]

@pauldeng

The OS can take a while to release memory or may not release it until it needs too. There is nothing specific here and its likely just what your os is setup to do. Its quite common to see memory increases like this. You can try to use valgrind to see if there are actually any leaks.

The following test can prove whether dufs has a memory leak:

1. Run dufs with docker and limit the memory to 20m

   docker run -d --name dufs -p 5000:5000 -m 20m sigoden/dufs

2. Generate loads

   while true; do ab -n 20000 -c 30 http://localhost:5000/; done

If dufs has a memory leak, the docker container will be killed due to OOM. But from my own actual test, this is not the case.

So dufs does not have a memory leak.