Memory not released?

[railsmechanic]

First, many thanks for that library, I'm using it in a backup application. Works great!

But I'm not sure whether the following behavior is a bug: I've developed a backup application which runs as a daemon and compresses files on a daily schedule. While the application is in an idle state, it consumes ~ 6MB RAM. When the daemon executes some tasks (other than xz compression), the memory consumption grows as long as the task is executed. When such a task has completed, the memory usage normalizes back to ~6MB.

But now, when the task starts which compresses files, the daemon consumes ~ 110MB memory which is absolutely ok, but this amount of memory is not released after the task has completed. So, after execution of the compression task the daemon process utilizes still that amount of memory. This memory usage remains the same even after multiple executions of the 'compression' task. Is this a normal behavior or am I doing something wrong?

Here is the code snippet which is responsible for the compression:

    destinationFile, err := os.OpenFile(destinationFilePath, os.O_CREATE|os.O_WRONLY, 0644)
    if err != nil {
        return err
    }
    defer destinationFile.Close()

    xzWriter, err := xz.NewWriter(destinationFile)
    if err != nil {
        return err
    }
    defer xzWriter.Close()

    if _, err := io.Copy(xzWriter, sourceFile); err != nil {
        return err
    }

    return destinationFile.Sync()

What do you think?

[ulikunitz]

Hi, how do you measure memory consumption? Please note that Go waits several minutes (ca. 5 min) before memory is released back to the operating system. You might want to create a memory profile for your program or using GODEBUG='gctrace=1' documented for package runtime.

[railsmechanic]

Ok, many thanks. I'll,try this... 👍