[DRAFT] Ubuntu 20.04 allows executable rewrite unexpectedly

zhouhaibing089 commented 1 month ago

Launch a ubuntu 20.04 environment - https://releases.ubuntu.com/focal/ (Tested with Virtualbox)
Install Docker - apt install docker.io
Prepare a Dockerfile like below:

FROM busybox
COPY --from=gcr.io/kaniko-project/executor:v1.23.2 /kaniko /kaniko

Build it via kaniko

$ docker run -it --entrypoint /bin/sh --rm -v $(pwd):/workspace gcr.io/kaniko-project/executor:v1.23.2-debug
/workspace # executor --no-push

(which works)

/workspace # executor --no-push

...
error building image: error building stage: failed to execute command: copying dir: creating file: open /kaniko/executor: text file busy

executor is a binary from /kaniko directory, and when copying /kaniko directory from another image, it actually copies to the current container /kaniko, so an error like text file busy is expected, the problem is why the first run is permitted without text file busy error.

Notes

ubuntu 20.04 uses kernel 5.4
overlayfs

zhouhaibing089 commented 1 month ago

strace

First run:

$ strace -p <pid> -f -e trace="/(open|exec)" -o syscall.txt
$ cat syscall.txt
2705  openat(AT_FDCWD, "/root/.ash_history", O_WRONLY|O_CREAT|O_APPEND, 0600) = 3
2895  execve("/app", ["/app"], 0x559425259660 /* 6 vars */) = 0
2895  openat(AT_FDCWD, "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", O_RDONLY) = 3
2895  openat(AT_FDCWD, "/etc/localtime", O_RDONLY) = 3
2895  openat(AT_FDCWD, "/app", O_RDONLY|O_CLOEXEC) = 3
2895  openat(AT_FDCWD, "/app", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 3

Second run:

2900  execve("/app", ["/app"], 0x559425259660 /* 6 vars */) = 0
2900  openat(AT_FDCWD, "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", O_RDONLY) = 3
2900  openat(AT_FDCWD, "/etc/localtime", O_RDONLY) = 3
2900  openat(AT_FDCWD, "/app", O_RDONLY|O_CLOEXEC) = 3
2902  openat(AT_FDCWD, "/app", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = -1 ETXTBSY (Text file busy)

This is to give us an idea on how many openat system calls were being made.

ftrace

Enable function-fork:

$ cat /sys/kernel/tracing/trace_options | grep function-fork
function-fork

If not, run echo function-fork >> /sys/kernel/tracing/trace_options

Now clear the existing trace if there are any:

$ echo nop > /sys/kernel/tracing/current_tracer

Now enable function_graph trace:

$ echo <pid> > /sys/kernel/tracing/set_ftrace_pid
$ echo __x64_sys_openat > /sys/kernel/tracing/set_graph_function
$ echo function_graph > /sys/kernel/tracing/current_tracer
$ cat /sys/kernel/tracing/trace > output

Disable the tracing after we are done:

$ echo nop > /sys/kernel/tracing/current_tracer

Check that we do get 9 __sys_openat:

$ cat output | grep __x64_sys_openat
 31)               |  __x64_sys_openat() {
 34)               |  __x64_sys_openat() {
 34)               |  __x64_sys_openat() {
 34)               |  __x64_sys_openat() {
 34)               |  __x64_sys_openat() {
 32)               |  __x64_sys_openat() {
 32)               |  __x64_sys_openat() {
 32)               |  __x64_sys_openat() {
 34)               |  __x64_sys_openat() {

We care about the call at 4th, 5th (the first run) and 8th, 9th (the second run):

zhouhaibing089 commented 1 month ago

Testing file:

package main

import (
    "bytes"
    "fmt"
    "io"
    "log"
    "os"
    "syscall"
)

func main() {
    if len(os.Args) > 1 && os.Args[1] == "help" {
        fmt.Println("This program is used to verify whether a process can rewrite itself.")
        os.Exit(0)
    }

    // print pid
    log.Printf("pid: %d", os.Getpid())

    path, err := os.Executable()
    if err != nil {
        log.Fatal("failed to find executable")
    }

    showStat(path)

    file, err := os.Open(path)
    if err != nil {
        log.Fatalf("failed to open %s: %s", path, err)
    }
    data, err := io.ReadAll(file)
    if err != nil {
        log.Fatalf("failed to read file: %s", err)
    }
    file.Close()

    // trying to rewrite it
    newfile, err := os.Create(path)
    if err != nil {
        log.Fatalf("failed to create %s: %s", path, err)
    }
    defer newfile.Close()
    _, err = io.Copy(newfile, bytes.NewReader(data))
    if err != nil {
        log.Fatalf("failed to copy file: %s", err)
    }

    showStat(path)
}

func showStat(path string) {
    fi, err := os.Stat(path)
    if err != nil {
        log.Fatalf("failed to stat %s: %s", path, err)
    }

    stat, ok := fi.Sys().(*syscall.Stat_t)
    if !ok {
        log.Fatalf("failed to get underlying stat for %s", path)
    }

    log.Printf("path: %s, dev: %d, inode %d", path, stat.Dev, stat.Ino)
}

Dockerfile

FROM golang:1.22
WORKDIR /go/src/app
COPY . .
RUN CGO_ENABLED=0 go build -o /go/bin/app

FROM busybox
COPY --from=0 /go/bin/app /app

zhouhaibing089 commented 1 month ago

On 5.15:

strace

2474  openat(AT_FDCWD, "/root/.ash_history", O_WRONLY|O_CREAT|O_APPEND, 0600) = 3
2613  execve("/app", ["/app"], 0x559240ec4660 /* 6 vars */) = 0
2613  openat(AT_FDCWD, "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", O_RDONLY) = 3
2613  openat(AT_FDCWD, "/etc/localtime", O_RDONLY) = 3
2613  openat(AT_FDCWD, "/app", O_RDONLY|O_CLOEXEC) = 3
2613  openat(AT_FDCWD, "/app", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = -1 ETXTBSY (Text file busy)

zhouhaibing089 / blog