golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
123.87k stars 17.65k forks source link

cmd/go: no compilation error for invalid filenames in subdirectories with embed.FS #54003

Open dwrz opened 2 years ago

dwrz commented 2 years ago

What version of Go are you using (go version)?

$ go version

go version go1.18.4 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env

GO111MODULE="auto"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/dwrz/.cache/go-build"
GOENV="/home/dwrz/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/dwrz/.go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/dwrz/.go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18.4"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3921777299=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I am trying to use embed.FS to store photographs for a website. There are photographs with a colon in the filename (an RFC3339 formatted timestamp) which were not appearing on the site. The logs reported the following error:

failed to open: open static/media/1920/image_2020-10-07T12:36:46Z.jpg: file does not exist

On investigation, it appears that files with a colon in the filename are not included in an embed.FS. It appears that an error is generated at compilation if a filename with a colon exists in the embedded root directory, but not if the problematic filename is in a subdirectory. In the latter case, the file is simply not included in the embedded FS.

This came as a surprise to me, since (1) the limitation with colons is not listed in the embed package documentation, (2) fs.ValidPath is documented as follows:

// Paths containing other characters such as backslash and colon
// are accepted as valid, but those characters must never be
// interpreted by an FS implementation as path element separators.

To replicate, I set up a directory like the following:

.
├── go.mod
├── snippet.go
└── static
    ├── 2022-07-22T15:02:45Z.txt
    ├── foo.txt
    └── subdir
        ├── 2022-07-22T15:02:45Z.txt
        └── bar.txt

snippet.go contains the following:

package main

import (
    "embed"
    "fmt"
)

//go:embed static/*
var static embed.FS

func main() {
    dir, err := static.ReadDir("static/subdir")
    if err != nil {
        fmt.Println(err)
        return
    }

    for _, f := range dir {
        fmt.Println(f.Name())
    }
}

When compiled, this returns the following compilation error:

snippet.go:8:12: pattern static/*: cannot embed file static/2022-07-22T15:02:45Z.txt: invalid name 2022-07-22T15:02:45Z.txt

However, if the filename with colons is removed from the top level, the program compiles. It then outputs the following:

bar.txt

subdir/2022-07-22T15:02:45Z.txt is excluded, and no compilation error is reported.

What did you expect to see?

What did you see instead?

seankhliao commented 2 years ago

Technically the restriction is implied by

Patterns must not match files outside the package's module

and if you try to download a module with a file containing colons it says

create zip: hello:world: malformed file path "hello:world": invalid char ':'

but it could be clearer.

cc @bcmills @matloob