extrafiles: make executable

[oliverpool]

The goal is to get grafana running on a raspberry pi 1. I struggled a lot to get it statically cross-compiled (more than for the linux kernel ;), but it now works: https://github.com/oliverpool/grafana-armv6/

I include the server binary using the special _gokrazy folder:

github.com/oliverpool/grafana-armv6/_gokrazy/extrafiles/grafana

It gets embedded in my build, but I can't run it, because it seems that the go package management removes the executable bit (the bit is set in the git repo, but missing in go/pkg/mod/github.com/oliverpool/grafana-armv6@v1.0.0-v8.4.4/_gokrazy/extrafiles/grafana).

So I currently get the error fork/exec /grafana/server: permission denied.

---

1. Changing the permission before running is not possible, since the filesystem is read-only.

2. I tried https://0xcf9.org/2021/06/22/embed-and-execute-from-memory-with-golang/ to execute it from memory, but it fails with memfd_create: function not implemented

3. Adjusting the bit before packing is not really feasible (I don't want to mess with the go/pkg/mod folder

4. The alternative would be to copy the executable on start to /perm/ and set the right bits, before running

This last solution should work, but does not seem clean. Do you see any better way to solve this issue?

Thank you!

[oliverpool]

Actually the 4th solution is not that bad: https://github.com/oliverpool/grafana-armv6/blob/main/main.go (currently the copy only happens if the file is missing or has a different size)

[oliverpool]

So feel free to close this issue, if this is the "right" solution

[stapelberg]

AFAICT, Go module zip files intentionally leave out any file modes (per https://go.dev/ref/mod#zip-files).

So, in the long run, we probably need some mechanism for configuring file attributes (other file system based attributes that come to mind are Linux capabilities).

As an immediate workaround, I think you can use the replace directive in your go.mod file so that gokr-packer won’t work with the read-only zip files in go/pkg/mod/…, but with a directory in your file system, where the executable bit is still present.

We could also consider automatically marking as executable extrafiles that are included into a certain location in the root file system, e.g. usr/local/bin. I think that’d be the best solution for now.

[stapelberg]

I struggled a lot to get it statically cross-compiled

By the way, have you seen https://gokrazy.org/prototyping/? It has grafana as an example.

[oliverpool]

So, in the long run, we probably need some mechanism for configuring file attributes (other file system based attributes that come to mind are Linux capabilities).

Sounds good! Since this mechanism is probably for edge cases, it could be kept user-side only, so that a package can't easily mess with the users' system (so the package would have to document the user to add a attributes/path/to/package/file_describing_attributes_and_perms, to specifically make a package-file executable).

In my case attributes/grafana/server should indicate the executable bit.

As an immediate workaround...

That's how I prototyped, but then I lose the version management in go.mod.

automatically marking as executable extrafiles that are included into a certain location

I don't know how many packages are concerned with this issue. If it is very fee, then maybe you shouldn't give the "packager" too much power ;)

---

have you seen https://gokrazy.org/prototyping/?

Yes, and it was quite helpful! However as mentioned in https://github.com/Pro/raspi-toolchain:

By default, newer GCC versions do not create correct binaries for ARMv6. Even though you pass the correct -mcpu= flag to gcc, it will create startup code for the newer ARMv7 architecture. Running them on your RasPI Zero will cause an "Illegal Instruction" exception.

So by default I was getting an armv7 executable (even when asking for GOARM=6), which would not run on my raspberry pi.

Then I found out that the build script of grafana was broken: https://github.com/grafana/grafana/pull/46989 (it produced an amd64 binary...).

And at the end, it appeared that some frontend files where missing (which then were (git)ignored by the auto-commit)...

The story looks short, but all in all, it took about 36 hours...

[stapelberg]

Since this mechanism is probably for edge cases, it could be kept user-side only, so that a package can't easily mess with the users' system

If by “user-side only” you mean that the package shouldn’t be able to declare a _gokrazy/fileattributes/ directory, I disagree — I don’t really see the harm that a package could do. On the contrary, if a package needs certain attributes to function, why not configure that in the package itself and make things a little easier for the user? :)

I don't know how many packages are concerned with this issue. If it is very fee, then maybe you shouldn't give the "packager" too much power ;)

I assume that an executable binary is the most common case. Not necessarily the most common extra file (I suppose config files could be more popular), but definitely the most common file attribute.

Hence, for now, I think I’d prefer a bin directory special case over a general file attribute handling.

---

Thanks for the explanation about the ARM version issue. Good job on getting to the bottom of it!

[stapelberg]

For https://github.com/gokrazy/gokrazy/issues/124 I need not only executable files, but also files that differ only in case (libxt_tos.so vs. libxt_TOS.so) and executable files that are located outside a bin/ directory.

Perhaps we should make the packer optionally take extrafiles packaged in a .tar archive. That way, we can preserve all the file attributes we care about without having to model them using separate configuration.

[oliverpool]

Using tar is a very good idea!

I still have an issue, because grafana has a very very large public folder (109MB - I reported a bug uptream, but I don't think it will be handled soon...) and github limits single objects to 100 MB...

Do you mind if I add the possibility to use extrafiles.tar.gz ? (grafana is only 50 MB gzipped) https://github.com/gokrazy/tools/blob/ec0f87b2604a13cdad850a94df399497397ce220/cmd/gokr-packer/packer.go#L392-L393

    r := io.Reader(f)
    if strings.HasSuffix(path, ".gz") {
        r, err = gzip.NewReader(r)
        if err != nil {
            return err, time.Time{}
        }
    }
    rd := tar.NewReader(r)

And below ae.extractArchive(dir + ".tar.gz")

[stapelberg]

Ah, I saw the warning at 50 MB, but didn’t realize GitHub will error at 100 MB.

Yeah, adding support for .gz is fine, but that’s only a workaround of course. Sooner or later there will likely be tarballs that exceed 100 MB even when gzipped.

I think GitHub recommends Git LFS, but I’m not sure if there’s any way to use Go modules with Git LFS…?

[oliverpool]

Another solution in the case of grafana would be to include both extrafiles.tar and extrafiles/.

I would tar the server, to keep the executable bit (about 70 MB) and let the front files as-is (6000 files ~ 100 MB).

---

that’s only a workaround of course

Actually, distributing binary files is already a workaround :laughing:

---

Yes, github recommend Git LFS:

You may want to try Git Large File Storage - https://git-lfs.github.com./

But I have now idea, if go modules supports it.