[Question] Is it possible to delete files/directories in parallel ?

[stulluk]

Title describes all.

Simply: I have huge sized directories with thousands of files in them ( Such as AOSP codebase) . When I try to delete the directory, almost all disk analyzer tools handle this task in single thread. Is it possible to remove a directory with hundreds of thousands of files in them in multiple threads?

What I am looking for is something like this:

[dundee]

We are using os.RemoveAll from standard Go library which really does the job in a single thread. I guess we might be able to dive into first level of the being-deleted directory and run deletion on each item in parallel 🤔

[dundee]

Will deletion on background solve the situation for you? https://github.com/dundee/gdu/issues/293

[stulluk]

Will deletion on background solve the situation for you? #293

Well, to be honest, no.

If my understanding is correct, even if we delete in the background, it will be still single thread, is my understanding correct?

[stulluk]

BTW, Chatgpt offered me goroutines, such as:

package main

import (
    "fmt"
    "os"
    "path/filepath"
    "sync"
)

func deleteFile(path string, wg *sync.WaitGroup, mutex *sync.Mutex) {
    defer wg.Done()

    err := os.Remove(path)

    // Handle errors or log them as needed
    if err != nil {
        mutex.Lock()
        fmt.Printf("Error deleting file %s: %s\n", path, err)
        mutex.Unlock()
    }
}

func deleteFilesInDirectory(directory string, numWorkers int) error {
    var wg sync.WaitGroup
    var mutex sync.Mutex

    err := filepath.Walk(directory, func(path string, info os.FileInfo, err error) error {
        if err != nil {
            return err
        }

        if !info.IsDir() {
            wg.Add(1)
            go deleteFile(path, &wg, &mutex)
        }

        return nil
    })

    if err != nil {
        return err
    }

    // Wait for all goroutines to finish
    wg.Wait()

    return nil
}

func main() {
    directory := "/path/to/your/directory"
    numWorkers := 10 // Adjust the number of workers based on your system's capabilities

    err := deleteFilesInDirectory(directory, numWorkers)
    if err != nil {
        fmt.Printf("Error: %s\n", err)
    }
}

Do you think that it is feasible to implement such thing ?

[dundee]

Yes, we can try somethink like that. I will first implement the deletion in background and then focus on this one.

[stulluk]

@dundee thank you very much, I tried this just now:

How did I built gdu in my ubuntu 22.04 host:

#add ubuntu 22.04 ppa golang 1.22: https://go.dev/wiki/Ubuntu#using-ppa
git clone https://github.com/dundee/gdu.git
make build-static

copy dist/gdu to /usr/bin and check version:

stulluk ~ $  gdu --version
Version:     v5.27.0-33-gbece451
Built time:  Mon Apr 15 02:16:28 AM +03 2024
Built user:  stulluk
stulluk ~ $ 

Add parallel delete to yaml:

stulluk ~ $  cat .gdu.yaml 
delete-in-parallel: true
stulluk ~ $

My Hardware: Ryzen 5700G ( 8 core 16 threads ) 64G RAM Samsung 980Pro NVME

and here is my result:

As you see, it only uses 4 cores, it is because number of mutex locked directories (hardware limitation) or am I missing something here?

[dundee]

It was using 4 physical threads which doesn't need to mean it was running only 4 internal Go threads (goroutines). Go starts only as many physical threads as it could utilise and it runs the internal threads in the physical ones as possible (if some of them is able to run).

Gdu uses maximum of 3 * GOMAXPROCS internal threads for deletion. Default value of GOMAXPROCS is the number of cores.

[stulluk]

So if I understand correctly, we couldn't get much benefit from this change ?

As you see above, basically CPU was almost doing nothing while disk write was at 84.4MBps ? ( Which is weird imho, given that Samsung 980Pro is able to handle 500+ MBps easily ? )

But thanks for implementing this anyway.

[dundee]

Interesting. I would guess we should get more disk write speed here. Maybe it's slowed down by some other constraint. I will give it more testing to see if the implementation is not missing something.

[stulluk]

Thank you @dundee . I am ready to support you for testing on my hardware. I suppose we should just create a simple script that reads /dev/urandom with 16Mbyte sized chunks, and create thousands of files inside thousands directories , with a total size about 100GByte ( with fdatasync probably)

Then, try to use new GDU with paralell delete, and see if we can gain something or not.

Just FYI, I deleted an AOSP build directory, and it took more than 1 minute again. I would suppose we couldn't utilize all our HW.

[dundee]

That would be a great help, thanks!

I will try use tracing to see where the time is really spent.

[dundee]

I was tracing deletion of directory with 16 items and it really spawned 16 internal threads. But it created threads not only for directories but files as well, which is not wanted, I will fix it.

I will also perform more benchmarks to see what benefit we are getting from this.

[dundee]

From the initial benchmark (https://github.com/dundee/gdu/pull/344/files) it unfortunately seems that the gain is very small or even negative. I am not an expert on Linux kernel, but I guess there is some locking happening that blocks the operations from running faster in parallel.

Trace for deletion in single thread:

Deletion in multiple threads:

There were two big files in separate directories, which could have been deleted faster in parallel theoretically, but the hypothesis was most likely wrong.

[stulluk]

@dundee thanks for looking at it. But what I see from your unit test code was that you are creating two 1Gbyte huge file, and tracing it how fast it deletes.

My use-case is different: I have hundreds of thousands of a few megabyte sized files in tousands of directories.

Should I create a test script to measure this use-case, or do you think it is better to do it with your unit test way ?

Pls let me know, I am ready to support.

[dundee]

@stulluk I think you can use my branch to try deleting your files both in parallel and sequential way (go run ./cmd/deleter/main.go --parallel ...) and measure the speed.

[stulluk]

@dundee Here is the performance of my system:

Here is a comparison of running your test code with "--parallel" argument ( 10x speed improvement in 100Gbyte directory with 100.000 files with 1megabyte size with random content)

Thank you so much for your kindness & support. I think we are done now.

[stulluk]

BTW, here was my simple & hacky script to create "testdir" (if anyone is interested):

#!/usr/bin/env bash

set -euo pipefail

NUMDIRS=100
NUMFILES=1000
EACHFILESIZE=1M

mkdir testdir
pushd testdir

#Create 100 directories and put 1000 files inside each

for dirno in $(seq 1 1 ${NUMDIRS})
do
    printf "this is directory number %d\n" "${dirno}"
    mkdir "dir-${dirno}"
    pushd "dir-${dirno}"
    curdir=$(pwd)
    for fileno in $(seq 1 1 ${NUMFILES})
    do
        printf "creating file number %d\n" "${fileno}"
        #touch "file-${fileno}"
        dd if=/dev/urandom of="${curdir}/file-${fileno}" bs="${EACHFILESIZE}" count=1 conv=fdatasync status=none &
    done
    wait
    popd

done
popd
sync