build failures when converting to syscall struct types

[jm33-m0]

Run with

go env -w "GOPRIVATE=*"
garble.exe -literals -tiny build -trimpath .\path\to\cmd\main

I get a bunch of errors:

garble.exe : # golang.org/x/crypto/ssh
At C:\Users\Administrator\go\src\github.com\xxx\xxx\build.ps1:2 char:1
+ garble.exe -literals -tiny build -trimpath .\cmd\xxx > err.log ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (# golang.org/x/crypto/ssh:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x20 pc=0xf7c8f2]

goroutine 1 [running]:
golang.org/x/tools/go/ast/astutil.Apply.func1(0xc000d2aec0, 0xc0019abb38)
        C:/Users/Administrator/go/pkg/mod/golang.org/x/tools@v0.1.1/go/ast/astutil/rewrite.go:47 +0xad
panic(0xfac980, 0x1187940)
        C:/Program Files/Go/src/runtime/panic.go:965 +0x1c7
go/types.(*Package).Scope(...)
        C:/Program Files/Go/src/go/types/package.go:42
main.(*transformer).transformGo.func1(0xc0003139f0, 0xc0003139f0)
        C:/Users/Administrator/go/pkg/mod/mvdan.cc/garble@v0.2.0/main.go:1121 +0x6b2
golang.org/x/tools/go/ast/astutil.(*application).apply(0xc0003139e0, 0x1049db8, 0xc0004272d8, 0xff2bbe, 0x3, 0x0, 
0x1049b88, 0xc00043ada0)
        C:/Users/Administrator/go/pkg/mod/golang.org/x/tools@v0.1.1/go/ast/astutil/rewrite.go:198 +0x35bb
golang.org/x/tools/go/ast/astutil.(*application).apply(0xc0003139e0, 0x1049a20, 0xc000437b40, 0xff2e0b, 0x4, 0x0, 
0x1049db8, 0xc0004272d8)
        C:/Users/Administrator/go/pkg/mod/golang.org/x/tools@v0.1.1/go/ast/astutil/rewrite.go:248 +0xe85
golang.org/x/tools/go/ast/astutil.(*application).apply(0xc0003139e0, 0x1049a48, 0xc0004384b0, 0xff2dc3, 0x4, 
0xc000313a28, 0x1049a20, 0xc000437b40)
        C:/Users/Administrator/go/pkg/mod/golang.org/x/tools@v0.1.1/go/ast/astutil/rewrite.go:221 +0x204b
golang.org/x/tools/go/ast/astutil.(*application).applyList(0xc0003139e0, 0x1049a48, 0xc0004384b0, 0xff2dc3, 0x4)
        C:/Users/Administrator/go/pkg/mod/golang.org/x/tools@v0.1.1/go/ast/astutil/rewrite.go:473 +0xac
golang.org/x/tools/go/ast/astutil.(*application).apply(0xc0003139e0, 0x1049e58, 0xc0004272f0, 0xff3375, 0x6, 0x0, 
0x1049a48, 0xc0004384b0)
        C:/Users/Administrator/go/pkg/mod/golang.org/x/tools@v0.1.1/go/ast/astutil/rewrite.go:226 +0x2831
golang.org/x/tools/go/ast/astutil.(*application).apply(0xc0003139e0, 0x1049ed0, 0xc000438480, 0xff2e0b, 0x4, 0x0, 
0x1049e58, 0xc0004272f0)
        C:/Users/Administrator/go/pkg/mod/golang.org/x/tools@v0.1.1/go/ast/astutil/rewrite.go:288 +0x12ef
golang.org/x/tools/go/ast/astutil.(*application).apply(0xc0003139e0, 0x1049b38, 0xc000437b80, 0xff310d, 0x5, 
0xc000313a28, 0x1049ed0, 0xc000438480)
        C:/Users/Administrator/go/pkg/mod/golang.org/x/tools@v0.1.1/go/ast/astutil/rewrite.go:403 +0x166b
golang.org/x/tools/go/ast/astutil.(*application).applyList(0xc0003139e0, 0x1049b38, 0xc000437b80, 0xff310d, 0x5)
        C:/Users/Administrator/go/pkg/mod/golang.org/x/tools@v0.1.1/go/ast/astutil/rewrite.go:473 +0xac
golang.org/x/tools/go/ast/astutil.(*application).apply(0xc0003139e0, 0x1049a70, 0xc0003ff400, 0xff308b, 0x5, 
0xc000313a28, 0x1049b38, 0xc000437b80)
        C:/Users/Administrator/go/pkg/mod/golang.org/x/tools@v0.1.1/go/ast/astutil/rewrite.go:411 +0x314f
golang.org/x/tools/go/ast/astutil.(*application).applyList(0xc0003139e0, 0x1049a70, 0xc0003ff400, 0xff308b, 0x5)
        C:/Users/Administrator/go/pkg/mod/golang.org/x/tools@v0.1.1/go/ast/astutil/rewrite.go:473 +0xac
golang.org/x/tools/go/ast/astutil.(*application).apply(0xc0003139e0, 0x104a2b8, 0xc000d2aec0, 0xff2de3, 0x4, 0x0, 
0x1049a70, 0xc0003ff400)
        C:/Users/Administrator/go/pkg/mod/golang.org/x/tools@v0.1.1/go/ast/astutil/rewrite.go:424 +0x1145
golang.org/x/tools/go/ast/astutil.Apply(0x1049a70, 0xc0003ff400, 0xc000d2aeb0, 0x1005e30, 0x0, 0x0)
        C:/Users/Administrator/go/pkg/mod/golang.org/x/tools@v0.1.1/go/ast/astutil/rewrite.go:52 +0x131
main.(*transformer).transformGo(0xc0001822d0, 0xc0003ff400, 0x5)
        C:/Users/Administrator/go/pkg/mod/mvdan.cc/garble@v0.2.0/main.go:1189 +0x9e
main.transformCompile(0xc000039020, 0x26, 0x3e, 0x7, 0xc000021f70, 0xc000039010, 0x12, 0x12)
        C:/Users/Administrator/go/pkg/mod/mvdan.cc/garble@v0.2.0/main.go:654 +0xeda
main.mainErr(0xc000039010, 0x27, 0x3f, 0x3f, 0x0)
        C:/Users/Administrator/go/pkg/mod/mvdan.cc/garble@v0.2.0/main.go:330 +0x6d5
main.main1(0xc00001c118)
        C:/Users/Administrator/go/pkg/mod/mvdan.cc/garble@v0.2.0/main.go:153 +0xe7
main.main()
        C:/Users/Administrator/go/pkg/mod/mvdan.cc/garble@v0.2.0/main.go:88 +0x29
note: module requires Go 1.17
exit status 2

If I set GOPRIVATE=example.com/user/* it gives me public package xxx can't depend on obfuscated package yyy

[jm33-m0]

If I set GOPRIVATE=example.com/user/* it gives me public package xxx can't depend on obfuscated package yyy

And when I move my project to example.com, I got new error saying relocation target runtime.something_random not defined

[jm33-m0]

My go version is 1.16.3 by the way.

[mvdan]

Please try your original command with master and share the results.

[jm33-m0]

Please try your original command with master and share the results.

I installed garble with go get, I think it pulls directly from the master branch

[mvdan]

You're using modules, so you installed 0.2.0. Use go install mvdan.cc/garble@master.

[jm33-m0]

@mvdan I have tried with latest code from master branch, with go env -w "GOPRIVATE=*", CGO is disabled, build target is 386, error becomes:

# vendor/golang.org/x/crypto/chacha20
typecheck error: C:\Program Files\Go\src\vendor\golang.org\x\crypto\chacha20\chacha_generic.go:10:2: could not import crypto/cipher (can't find import: "crypto/cipher")
# vendor/golang.org/x/crypto/cryptobyte
typecheck error: C:\Program Files\Go\src\vendor\golang.org\x\crypto\cryptobyte\asn1.go:8:16: could not import encoding/asn1 (can't find import: "encoding/asn1")
# vendor/golang.org/x/text/unicode/norm
typecheck error: C:\Program Files\Go\src\vendor\golang.org\x\text\unicode\norm\composition.go:7:8: could not import unicode/utf8 (can't find import: "unicode/utf8")
# vendor/golang.org/x/text/secure/bidirule
typecheck error: C:\Program Files\Go\src\vendor\golang.org\x\text\secure\bidirule\bidirule.go:12:2: could not import errors (can't find import: "errors")
exit status 2

If I leave out the standard libraries with GOPRIVATE=github.com/*, it builds fine, and the resulting binary works as expected

[mvdan]

Interesting. Can you reproduce on Linux, e.g. inside a Docker container? I haven't seen errors like that on Linux, and I don't have a Windows machine to test on.

[mvdan]

Also worth noting that we already test on Windows, and we have a test that does a cross-build for both GOOS and GOARCH. So everything should work on Windows.

[mvdan]

Timed out. Happy to reopen if you can provide a way to reproduce this, or if you can help us debug what's going on.

[jm33-m0]

Sorry for the late reply, I just tested my code on Linux with GOPRIVATE=*, it fails with following error:

# github.com/vishvananda/netlink/nl
:1: cannot convert *a (type h5iKSdK5.L3J_TlQ2) to type syscall.RtAttr
:41: undefined: iota
exit status 2
exit status 2

[jm33-m0]

Similar errors can be triggered at runtime as well.

For example, a binary is built without errors, but when trying to run it, some of its functions might fail due to obfuscated names that it cannot handle.

[mvdan]

The runtime errors due to obfuscated names is probably due to reflection; see https://github.com/burrowers/garble#caveats. Garble should get better at automatically doing the right thing in most cases, but reflection by definition is just arbitrary logic, so it's impossible to tell what names can or cannot be obfuscated.

That RtAttr error looks like a bug. Can you reproduce with garble master? If so, could you provide a standalone way for us to reproduce the error?

[jm33-m0]

That RtAttr error looks like a bug. Can you reproduce with garble master? If so, could you provide a standalone way for us to reproduce the error?

I think simply importing the netlink package will do

[mvdan]

Thanks, I can reproduce via cd core && GOPRIVATE=* garble build github.com/vishvananda/netlink/nl in your repo.

[mvdan]

Fully obfuscating the syscall package to fix this problem at its root will take some time, I think, because that also requires obfuscating the runtime.

For now, you can manually mark your RtAttr type to not be obfuscated via reflect.TypeOf, as shown in https://github.com/burrowers/garble#caveats.

[jm33-m0]

Recently I found another bug, if a pointer of structure is used as function return variable, it fails to parse as it's obfuscated, even with reflect.TypeOf, because upper level name is also obfuscated.

My solution is to create the same variable in function body, and use its address as return value

If you can reproduce it I think we should open another issue for this bug.

[mvdan]

Can you file that as a separate issue with details on how I can reproduce it? Ideally with a standalone example program.