cmd/go: remove -buildmode=shared (not c-shared)

[rsc]

-buildmode=shared has been broken since modules, and it is apparently unused. See https://github.com/golang/go/issues/47257#issuecomment-897243256. Let's remove it.

[rsc]

This proposal has been added to the active column of the proposals project and will now be reviewed at the weekly proposal review meetings. — rsc for the proposal review group

[electricface]

I really want to use this buildmode=shared mode.

[scott-cotton]

@rsc how has it been broken since modules?

[ainar-g]

Just to clarify, this is not related to --buildmode=plugin, right?

[ianlancetaylor]

@electricface Why do you want to use it? Are you using it today?

@scott-cotton You can't use -buildmode=shared with a module and then use -linkshared to link against it from other modules. The whole idea of -buildmode=shared doesn't fit very well with modules. Each separate program can be linked against different versions of dependent modules, so if it did work people would in practice wind up with many different versions of the shared libraries of the same modules.

@ainar-g Correct, this is not about -buildmode=plugin, although plugins have their own, different, problems.

[electricface]

@ianlancetaylor I am a debian-baseed Linux Desktop Environment developer. Our team have developed lot of daemon programs with golang. But these programs have significantly larger binary file size and use more ram than C or C++ programs. We currently working on these two problems. An idea is making some modules pluggable, which means only load modules that we need.

As other C or C++ programs are using shared-library, we are researching on some shared-library-like mechanism of golang. Inspired by buildmode=plugin and goloader, we found a way that bases on buildmode=shared to make modules pluggable recently, you can found a demo here. This solution has not been using in production environment, because of some problems hasn't been solved yet. I found out that programs built with -linkeshared can not be debugged using gdb and dlv.

After this commit, our "unoffical plugin" solution can work fine with go 1.14.9~1.16.7. Programs built in this way can run stably. But this won't work on go 1.17. Program will exit with a "fatal error: unreachable method called. linker bug?"

Plan of remove buildmode=shared make me feel bad.

[ianlancetaylor]

@electricface It sounds like you want plugins, which are supported (though not very well) by -buildmode=plugin. We aren't going to support -buildmode=shared just to support plugins, since we already have -buildmode=plugin.

[electricface]

@ianlancetaylor I know that using static linking can eliminate the problems caused by the mismatched version of the dependent dynamic library. But in the linux desktop system ecosystem, C or C++ languages are mostly used. These languages generally use dynamic libraries to save memory. The main reason for not using -buildmode=plugin is that static linking cannot save memory. For example, if the modules are divided into smaller ones and a lot of modules are added, each module occupies a relatively large amount of memory, and the overall occupancy will eventually become unacceptably large. Compared with C or C++ languages, they only use less memory to implement these functions, while the golang implementation uses several times the memory. There is even an idea in our team that we must change the development language if we can't reduce the memory. I personally like to use golang for development, so I plan to study this part of the technology.

[rsc]

"I want to use it" is different from "I am using it today". We believe that it is not possible to use it today, and furthermore it is difficult to provide. We shouldn't advertise something that doesn't actually work.

[rsc]

Based on the discussion above, this proposal seems like a likely accept. — rsc for the proposal review group

[mewmew]

Just to confirm, this proposal suggests to remove --buildmode=shared while keeping --buildmode=c-shared? I am using --buildmode=c-shared for a number of projects and would be very sad to see it go.

[ianlancetaylor]

This proposal is only about -buildmode=shared.

[rsc]

No change in consensus, so accepted. 🎉 This issue now tracks the work of implementing the proposal. — rsc for the proposal review group

[rasky]

@rsc not sure if this is a good place to give feedback about the proposal process, but this is the first time (since the proposal process exists) that I missed an issue. I was mildly interested in this (not enough now to vehemently protest), but even if I'm subscribed to the proposal process issue and I normally read minutes every week, I missed this. I took 10 days off for vacation, and those 10 days covered the only 2 emails where this issue was mentioned, before today when it was approved. Again, I don't mind about this specific issue at this point but I wanted to raise a general point.

Maybe a proposal should stay a minimum time in the proposal process before being closed, so that even people that don't work on Go full time have time to notice and react even with personal life happening in-between.

[zhouguangyuan0718]

To limit binary file size, I use command go install -buildmode=shared std to install libstd.so. And I build my application by go build -linkshared. I have been using this in production environment for a long time. I hope I can still use it in the furture. Can we keep -buildmode=shared std, it's useful.

[Jason7602]

Our company has been using the shared mode to reduce the binary size for a long time. The plugin mode is not useful in some scenarios cause the entire runtime package needs to be compiled and packed. So we strongly hope that shared mode can be reserved.

[mlaventure]

We also use buildmode=shared to reduce binary size / allow pages to be shared across several binaries using a common set of packages when running in on devices with (relatively) low RAM. Hopefully a solution can be found to make this viable with modules, but I'd prefer for the option to not disappear. Without this, we would have to resort to a busybox like binary which is not ideal as each binary is maintained by a different set of people and released independently.

[gopherbot]

Change https://golang.org/cl/359096 mentions this issue: cmd/go: remove support for -buildmode=shared

[grolfcry]

It's very strange decision to remove shared completely. I think shared or plugin mode was not well designed and now, we can't use golang with something like java jars.

[ianlancetaylor]

@grolfcry Note that go build -buildmode=c-shared still works. I don't see how -buildmode=shared is related to Java jars.

[grolfcry]

How I can build go dependencies as external libs now if I cant build shared libs and link shared? Java jars relate go shared libs and java fat jar like default go build (one executable).

[ianlancetaylor]

So, just to be clear, you are mentioning Java jars as an analogy? You're not literally trying to use Go with Java jars?

Yes, this is functionality that will no longer be available. But it already didn't work for anything other than the standard library. Perhaps we can introduce some similar functionality in a way that can actually work, but it won't be go build -buildmode=shared.

[grolfcry]

Of course, java jars it's analogy. I can use any go module (any lib) as external dependency with shared mode and link shared, not only for standard modules in golang earlier 1.16 - https://github.com/golang/go/issues/47455#issuecomment-942648877. Now it's impossible, and it's very strange way of language evolution - If we can't do it right, we won't do it at all.

[ianlancetaylor]

Yes: it used to work, but it wasn't possible to make it work with modules.

My apologies for the loss of functionality.

[gopherbot]

Change https://golang.org/cl/359575 mentions this issue: Revert "cmd/go: remove support for -buildmode=shared"

[rsc]

@ianlancetaylor, @bcmills, and I talked a bit more about this yesterday. I had not seen the comments from Sept 2 onward until the last couple days. (I can't keep up with all the GitHub mail I get, and once a proposal is accepted I tend not to watch it.)

We've decided to leave -buildmode=shared alone (not delete it) for Go 1.18, but it's not going to work any better than it does today. In particular, we're not fixing any bugs in it anymore, and we still intend to remove it in a future release. However, I would like to better understand how to serve existing use cases, if that's possible.

The most fundamental bug is that -buildmode=shared stores state in $GOROOT/pkg (#22196), making that directory not a cache that can be deleted and recreated at any time. This made a little bit of sense when you had to go install individual packages in order to use them in future builds, but not really, and especially not now that go install is completely unnecessary for packages.

A second important bug is that nothing about the shared objects is cached properly (#24034). Consider this pair of commands:

go install -buildmode=shared std
go build -linkshared helloworld.go

The go build repeats all the work of the first go install, rebuilding the entire standard library in shared mode and creating a new libstd.so, installing it over the (identical) existing one. If you run the go build again, it does all the work yet again. Every time.

This makes builds using -linkshared incredibly slow, and it also means that they don't work at all without write access to $GOROOT.

A third important bug is that the toolchain fundamentally assumes that packages are being compiled against the exact versions of the packages they import (#19389, #21510). For example, suppose draw.Point is struct { X, Y int } and you build the standard library into libstd.so and then you build a graphics program against it. And suppose then the standard library changes to add Z int to Point. Any stack allocation of a Point in the graphics program is now failing to reserve enough space, which will probably lead to memory corruption when invoking any function that takes or returns a Point, or invalid address arithmetic accessing points[i] for var points []draw.Point. The problem is that the compiled client code assumes something about draw.Point, namely that it is two words long, so that can't change in libstd.so without recompiling the clients. Escape analysis, inlining, and other optimizations make the same kinds of assumptions: if draw.Point.Add has been inlined into the client code, then changing the copy in libstd.so will have no effect (or, worse, inconsistent effects depending on inlining decisions). And if a function parameter that didn't escape starts escaping, optimizations applied in the client code are no longer valid.

These may seem like obvious examples, especially to people familiar with shared library gotchas. We could of course add even more special logic to the toolchain, so that it knows where the shared library boundaries are, and then we could disable any use of escape analysis results or inlining of function bodies across those boundaries. But we are never going to disable things like doing a simple multiplication to access points[i]. The result would be too horribly inefficient. Instead we change the ABI hash when almost any detail of the original source changes, which means that you can basically never drop in a new libstd.so and use it with programs compiled against an older libstd.so (#23405).

On top of those fundamental problems, there are many corner cases that don't work, most of which @seankhliao helpfully closed (thanks!):

• cmd/go: go install buildmode=shared doesn't work without first building the standard library #12136
• test: nilptr.go fails on ppc64le when externally linked #13436
• cmd/link: -buildmode=shared includes dead symbols #14578
• cmd/link: panic: runtime error: invalid memory address or nil pointer dereference when using -buildmode=shared -linkshared on ppc64le #15770
• build: run more buildmode=shared tests #16602
• cmd/link: using -linkshared dies if compile not passed -p option #16632
• cmd/go: -buildmode=plugin -linkshared -> build failure #18671
• cmd/internal/dwarf: incorrect or missing dwarf information in libstd.so on ppc64le, amd64 #20328
• cmd/link: seg fault in shared buildmode from main package with exported variable #22566
• cmd/go, cmd/link: support buildmode=shared on darwin #23126
• runtime: segfault in sync.Pool when using a shared runtime #24640
• cmd/link: run tests failed with lots of cases in buildmode=shared #26400
• cmd/link: run test in -linkshared mode failed #26582
• cmd/go: cannot use packages ... from different roots #33037
• cmd/go: Building a shared library in modules mode produces empty library #37675
• cmd/link: buildmode=shared with many packages produces "file name too long" error #38330
• cmd/link: dynamic linking cannot be stepped through using gdb #38378
• cmd/go: executable created using linkshared option is not linking .so created by buildmode=shared with go modules enabled #38499
• cmd/go: Mac Catalina 10.15.6 -buildmode=shared not supported on darwin/amd64 #40532
• cmd/go: crashes when -buildmode=shared is set and modules enabled #42189
• cmd/link: can't build shared time/tzdata #44073
• cmd/link: building a package that uses cgo errors out with "cannot implicitly include runtime/cgo in a shared library" #47183
• cmd/link: panic: runtime error: makeslice: cap out of range #47455

We can leave those closed, by the way.

There are two main reasons that people reach for shared libraries. First, the argument goes, shared libraries make it possible to handle a security fix for a library by updating the one copy of the library and to avoid updating all programs using it. Second, shared libraries result in smaller binaries.

The security fix argument basically doesn't hold up at all. Even in C, it only works if you are incredibly careful in the first place and don't need to change any details of the library headers, such as struct layouts, #defined macros, and so on. In Go, as noted above, it works approximately never. Never has. Even if we solved the escape analysis and inlining problems, it's not uncommon to need to add a new field to a struct in order to fix a security problem. If you have been using Go's -buildmode=shared to make security fixes easier to deploy, you probably have unpatched or unstable programs and should reconsider that decision immediately.

That leaves smaller binaries. The last three comments above all mention binary size as the reason for using -buildmode=shared. And I appreciate that hello world linked against libstd.so is only 1.6% the size of its statically linked cousin! (A more realistic example: the go command is 31% the size when std is dynamically linked.)

Assuming that smaller binaries is the only use case we need to support, I think it would work to simplify down to supporting only a single .so, which you can build with -buildmode=shared but then have to provide explicitly to future builds as “this is the .so I want to share code with.” Those future builds read what they need from the .so (bypassing $GOROOT entirely), and they fail if the .so is stale.

The current idiom

go install -buildmode=shared std
go build -linkshared helloworld.go

would be replaced with:

go build -buildmode=shared -o libmyshared.so std
go build -linkshared=libmyshared.so helloworld.go

Of course, it wouldn't have to be just std: it could be std plus other packages. But you only get one .so, and you have to manage installing it and providing it to future go build commands. All the .shlibname code in cmd/go gets deleted.

Distributions might choose to do

go build -buildmode=shared -o libgo1.17.2.so std golang.org/x/...
go build -linkshared=libgo1.17.2.so cmd1
go build -linkshared=libgo1.17.2.so cmd2
go build -linkshared=libgo1.17.2.so cmd3

And then if they need to support multiple Go versions, it's obvious what to do.

Glancing through the bug titles above, it looks like this design would eliminate the vast majority of the issues we've had. It definitely addresses the most serious problems that I mentioned above. The only problem I'd still be concerned about is the handling of debug information, but that should be doable.

We are already planning to keep -buildmode=plugin and -buildmode=c-shared, so I am hoping that the amount of extra code for this more limited -buildmode=shared will not be a significant burden.

To everyone, but especially to @rasky, @zhouguangyuan0718, @Jason7602, and @mlaventure: would this alternate UX for -buildmode=shared work for your current use cases?

Thanks.

[rsc]

Bumped this back into "Likely Decline" so that we can include the status update in the next proposal minutes.

[mlaventure]

To everyone, but especially to @rasky, @zhouguangyuan0718, @Jason7602, and @mlaventure: would this alternate UX for -buildmode=shared work for your current use cases?

Thanks for reopening and considering this use case @rsc .

Your proposal would work for me indeed. Not as flexible as having every module as a separate library, but given the added complexity, I find the one library to be a nice compromise. :+1:

[rasky]

Thanks for reopening. It would work for me. My use case is embedded system with limited flash where you want to deploy multiple binaries written in Go. The current workaround is to link all unrelated applications in a single binary and then use symlinks and check argv[0] to internally dispatch to the right entrypoint (ala busybox).

Putting common code (std + other packages) in a single .so file would surely solve my problem.

[thanm]

Russ wrote:

The only problem I'd still be concerned about is the handling of debug information, but that should be doable

One thing we can do that would make the problem more tractable in general would be to move DWARF type DIE generation out of the linker and into the compiler (this is an idea that we've toyed with the in the past but haven't gotten to). Doing that would definitely help for the "link shared against libstd.so" use case.

[rsc]

No change in consensus, so declined. — rsc for the proposal review group

[tandr]

So... I got excited and intrigued (I want smaller executables too...), and tried

go install -i -buildmode=shared std
go build -linkshared helloworld.go

but getting

$ go version
go version go1.17.2 linux/amd64

$ go run -linkshared helloworld.go
fatal error: unreachable method called. linker bug?

goroutine 1 [running]:
runtime.throw({0x7f6f2569d855, 0x55948fd23ae8})
    /usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc00035eec8 sp=0xc00035ee98 pc=0x7f6f24f6f351
runtime.unreachableMethod()
    /usr/local/go/src/runtime/iface.go:561 +0x25 fp=0xc00035eee8 sp=0xc00035eec8 pc=0x7f6f24f39fa5
fmt.Fprint({0x55948fd23b18, 0xc000420008}, {0xc00035ef60, 0x1, 0x1})
    /usr/local/go/src/fmt/print.go:233 +0x75 fp=0xc00035ef38 sp=0xc00035eee8 pc=0x7f6f25083115
fmt.Print(...)
    /usr/local/go/src/fmt/print.go:242
()
    /home/atch/ww/at/go/test/helloworld.go:6 +0x55 fp=0xc00035ef80 sp=0xc00035ef38 pc=0x55948fd211d5
runtime.main()
    /usr/local/go/src/runtime/proc.go:255 +0x282 fp=0xc00035efe0 sp=0xc00035ef80 pc=0x7f6f24f72a02
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00035efe8 sp=0xc00035efe0 pc=0x7f6f24fab4e1
exit status 2

So, question to people who saw it working - when was it? (my understanding is that because this feature is on a borrowed time right now, there is no point to open a bug report)

[kiap]

Hello, It's quite funny that Mr. Cox with all due respects, propose removing shared library support, and get it accepted and closed, on the basis nobody uses it,

WHY DO YOU THINK I HAVE BEEN reporting such bugs? Do you have any legal review board? Obviously not,

What shall we do with legal license requirements that shared library is ok but compiling into static is not?

I think a sane programming language should have shared library support

It seems we have to move to rust as Go is getting crazier everyday

[ianlancetaylor]

@kiap I don't know if you realize that this proposal was declined and was not implemented.

There is a new suggestion in https://github.com/golang/go/issues/47788#issuecomment-954890659. It would be helpful to know whether that would or would not work for your case.

[DeedleFake]

I, for one, was unaware of that, despite the fact that I follow the issue tracker fairly closely a lot of the time. I stopped following this one, however, once the accepted label got applied. Could the labels be updated to reflect the actual status?

[zhouguangyuan0718]

To everyone, but especially to @rasky, @zhouguangyuan0718, @Jason7602, and @mlaventure: would this alternate UX for -buildmode=shared work for your current use cases?

I'm so sorry for my so late reply, @rsc .The new proposal seems more perfect and graceful for smaller binaries. But I think there is some details to think over and over.

go build -buildmode=shared -o libgo1.17.2.so std golang.org/x/...
go build -linkshared=libgo1.17.2.so cmd1
go build -linkshared=libgo1.17.2.so cmd2
go build -linkshared=libgo1.17.2.so cmd3

If I don't misunderstand, in this proposal, cmdx is a main package, all the dependencies of cmdx should be contained in libgo1.17.2.so. It is a new constraint. Maybe we can support some more packages build in the build command or other way.

And more details I can realize is the dependency chain of dynamiclink. Will it support a commands looks like go build -buildmode=shared -linkshared=libgo1.17.2.so -o libcmd.so cmd? I think this is useful, And comparing with the new proposal, it will not be a significant burden.

The last thing I'm care about is that if the new proposal is implemented, can it support the plugin with linkshared? Looks like go build -buildmode=plugin -linkshared=libgo1.17.2.so -o plugincmd.so cmd. I had implemented limited "plugin with linkshared" and I'm planing to use it in our producting enviroment. I think if the new proposal is accepted, it will more easy and less situation to resolve for "plugin with linkshared" .

I'm always focus on shared library or dynamiclink in go. And I'am glad to contribute any code about shared library or dynamiclink in the future.

Thanks.

[zhouguangyuan0718]

So, question to people who saw it working - when was it? (my understanding is that because this feature is on a borrowed time right now, there is no point to open a bug report)

@tandr It is a bug in go 1.17~1.17.2. I had fixed it in go 1.17.3 and master. You can try 1.17.3 or gotip.

[kiap]

Let me explain: I am quoting from @rsc

There are two main reasons that people reach for shared libraries. No there are three at least: (or even more)

First: Security fix, one shared library for all of its clients/users code

The argument he gave is it is broken unless you are careful; Sir, you need to stick to SemVer and security fix should not break that compatibility, Go is loosey-goosey, breaking everything of course, (My complaint)

Second: Code size While this argument is true, I never grasped the idea of one exec file do it all you make, while all bosses and clients and techs used to C style library + executables + other stuff, Deploying a single large exec makes your commercial product weird to them, there are other obvious technical merits, such as space constraint, Which I do not think Go fits in, When we picked up Go we wanted to see efficiency but got bloated memory usage, did not let us use it for much system-level management of internal Cloud Virtual Machines (VPSes) The idea seems for Google is memory is cheap so who cares right! To Me Go is thinking of itself as a replacement for Java or Python, we gravitated to it from C, for system-level management and distributed systems management.

Third: Legal issues, Let's say LGPL license, so many commercial codes or licensed code based on patent and any commercial code that permits only shared linking, don't ask my inlining cross-code optimization and thus break that, tell the lawyers who wrote them, those who want to break free from liability need to avoid derivative work, and thus shared linking is required.

There could be other reasons I cannot think of now, I cannot tell even it is three use cases.

@rsc wrote:

We've decided to leave -buildmode=shared alone (not delete it) for Go 1.18, but it's not going to work any better than it does today. In particular, we're not fixing any bugs in it anymore, and we still intend to remove it in a future release.

This is exactly what I experienced since 2016, every major release, something breaks the shared library and I had to report it to get it fixed. the ultimate one is modules.

@ianlancetaylor no the proposed solution in the comments does not work,

for exactly the same reasons of (1) (2) and (3)

We create a library directory that holds shared libraries in /usr/local/libs/

if that library exists go uses shared otherwise just pie see below: (You already have broken this in every major release and I reported and got it fixed)

even we do not create libstd.so we create disaggregate libs

This is how we do it: (for every version) below is for 1.17

If you see multiple libraries in ONE LINE OF the list because of Go gotchas, this is the only way you can compile disaggregate (beyond a single comment)

`vi lib_list_1.17.txt

runtime sync runtime/race internal/unsafeheader unicode unicode/utf8 internal/itoa math/bits container/list container/ring crypto/internal/subtle crypto/subtle unicode/utf16 vendor/golang.org/x/crypto/cryptobyte/asn1 internal/nettrace vendor/golang.org/x/crypto/internal/subtle math encoding internal/goversion hash/maphash image/color internal/cfg reflect/internal/example1 reflect/internal/example2 image/color/palette math/cmplx runtime/metrics internal/reflectlite internal/testlog internal/sysinfo internal/singleflight math/rand errors sort io internal/oserror path crypto/elliptic/internal/fiat vendor/golang.org/x/net/dns/dnsmessage strconv syscall container/heap strings bytes hash crypto/internal/randutil hash/crc32 hash/adler32 crypto/hmac text/tabwriter vendor/golang.org/x/crypto/hkdf hash/crc64 vendor/golang.org/x/text/transform reflect crypto bufio crypto/rc4 encoding/ascii85 encoding/base32 net/http/internal/ascii go/build/constraint regexp/syntax hash/fnv html net/http/internal/testcert compress/bzip2 image internal/syscall/unix internal/syscall/execenv time time/tzdata plugin regexp image/internal/imageutil image/draw image/jpeg context io/fs internal/poll embed os internal/fmtsort encoding/binary crypto/ed25519/internal/edwards25519/field crypto/sha512 crypto/cipher crypto/md5 crypto/sha1 crypto/sha256 encoding/base64 vendor/golang.org/x/crypto/poly1305 fmt encoding/pem crypto/ed25519/internal/edwards25519 net path/filepath io/ioutil internal/lazyregexp crypto/aes crypto/des vendor/golang.org/x/crypto/chacha20 vendor/golang.org/x/sys/cpu index/suffixarray os/exec internal/obscuretestdata runtime/debug os/signal vendor/golang.org/x/crypto/chacha20poly1305 compress/lzw encoding/hex os/user net/url compress/flate math/big vendor/golang.org/x/crypto/curve25519 database/sql/driver debug/dwarf debug/gosym debug/plan9obj database/sql encoding/csv archive/zip compress/gzip compress/zlib encoding/gob encoding/json debug/elf debug/macho debug/pe archive/tar encoding/xml crypto/dsa crypto/elliptic encoding/asn1 crypto/rand log crypto/ed25519 crypto/rsa vendor/golang.org/x/text/unicode/bidi vendor/golang.org/x/text/unicode/norm vendor/golang.org/x/crypto/cryptobyte crypto/x509/pkix vendor/golang.org/x/net/http2/hpack mime mime/quotedprintable net/http/internal flag vendor/golang.org/x/text/secure/bidirule go/token text/template/parse crypto/ecdsa internal/buildcfg internal/execabs internal/execabs internal/goroot go/build go/internal/gccgoimporter go/internal/srcimporter vendor/golang.org/x/net/idna go/scanner go/constant internal/xcoff text/scanner image/gif go/ast image/png internal/profile runtime/trace internal/trace crypto/x509 net/textproto vendor/golang.org/x/net/http/httpproxy text/template testing vendor/golang.org/x/net/http/httpguts mime/multipart log/syslog runtime/pprof go/internal/typeparams net/internal/socktest net/mail go/parser go/printer os/signal/internal/pty testing/iotest internal/testenv crypto/tls testing/fstest testing/quick go/doc html/template internal/lazytemplate testing/internal/testdeps vendor/golang.org/x/net/nettest go/types go/format net/http/httptrace net/smtp net/http expvar net/http/cgi net/http/cookiejar net/http/httptest net/http/httputil net/http/pprof net/rpc net/http/fcgi net/rpc/jsonrpc go/internal/gcimporter go/importer `

Another bash file (This is an example)

`#! /bin/bash GO_DIR=/usr/src/dev LIB_DIR=/usr/local/lib/project WORK_DIR=/usr/src/project STD_LIB_LIST=lib_list_1.17.txt

            IFS_saved=$IFS
            IFS=$'\n'
            for i in `cat ${WORK_DIR}/${STD_LIB_LIST}`
            do
                    IFS=$IFS_saved
                    go install -v -ldflags '-s -w' -gcflags "-trimpath ${WORK_DIR}" -buildmode=shared -linkshared -pkgdir "${LIB_DIR}" $i
            done

            IFS=$IFS_saved

`

then to build:

    # -linkshared here  falls back to static linking if the -pkdir is empty or the libraries could not be found
    # we use this auto-detection of shared libraries, to compile shared or static library just by 
    # creating the library directory in Makefile and compiling them using lib option above
    # so if no shared library is found static linking is invoked with pie linkshared  otherwise the available libraries
    # are compiled in as dynamic linking

$2 is your project

go install -v -ldflags '-s -w' -gcflags "-trimpath ${WORK_DIR}" -buildmode=pie -linkshared -pkgdir "${LIB_DIR}" "$2"

All just to satisfy the legal issue! The build scripts are more complex I just provided a simplified piece

Even our libraries are placed like that

Unless you assume every single piece of code you got is licensed as BSD o MIT or apache or you wrote it then feel free for using static linking.

If you cannot satisfy dynamic shared libraries requirement, Go is at the level of Perl, PHP, in our usage, as a medium-size business we need to satisfy the legal department, I cannot comment on Small Businesses who do not care about legalities.

[kiap]

Just to comment on my own comment that The idea of a shared library is to create once and used it as many times

With Go you have to recompile the entire shared library every time you compile (Unless you keep cache) which is more complicated

Go is not C, I see, but cannot change legalities of licenses too.

So it would be very much appreciated if a shared library acted like exactly as C, an OS vendor then shipped one library of a version in disaggregate.

[ianlancetaylor]

@DeedleFake Sorry, you're right, the history of this proposal has been confusing.

I removed the "Accepted" label.

[ianlancetaylor]

@kiap I would be very interested in hearing of any Go package anywhere that is distributed under the LGPL. Given Go's current technology such a package would effectively be under the GPL, so I don't understand why anybody would choose the LGPL.

If there are no Go packages under the LGPL, then I don't understand the legal concerns.

Thanks.

[kiap]

@ianlancetaylor No it is not, GPL

Technically even shared libraries in C : part of the code is derived from the original by the compiler, however:

    "A program that contains no derivative of any portion of the Library, but is designed to work with the Library by being
    compiled or linked with it, is called a "work that uses the Library". Such a work, in isolation, is not a derivative work of 
     the Library, and therefore falls outside the scope of this License. ",  the LGPL version 2.1

the exception for shared library allows the part the compiler or linker mixes in

Sample write a wrapper around libraries in C that are LGPL

There are other works that people write and put under MIT, BSD licenses, if you cannot be sure of the origin then you can be held legally liable regardless of the original author mistake or ill-fated decision,

If there are no Go packages under the LGPL, then I don't understand the legal concerns.

To me, you made your decision Sir, and are simply asking to avoid any other licenses that are not MIT, BSD, Apache, and the like. That's a solution of course why did they clutter the OS with shared libraries, right? perhaps at 46, I am old.

[ianlancetaylor]

Writing a wrapper around LGPL C libraries is a "work that uses the library." The C libraries can continue to be shared libraries. There is no need for the Go code itself to be in a shared library. That works today and will continue to work.

To me, you made your decision Sir, and are simply asking to avoid any other licenses that are not MIT, BSD, Apache, and the like. That's a solution of course why did they clutter the OS with shared libraries, right? perhaps at 46, I am old.

That is not our intent. It is fine for Go programs to use GPL libraries. It is fine for Go programs to use LGPL libraries, as shared libraries, if those libraries are written in other languages. It's true that it's hard for Go code to be licensed under the LGPL; it is in effect the same as being licensed uder the GPL. But that doesn't seem so bad as even the FSF recommends against using the LGPL (https://www.gnu.org/licenses/why-not-lgpl.html).

The difficulty with shared libraries in Go is not because we don't like shared libraries. It's because they are technically difficult to support, and nobody has figured out how to do it well.

And for what it's worth I'm a lot more than 46 years of age.

[kiap]

@ianlancetaylor thanks for addressing our concern but, A wrapper around the LGPLed C library for easier use in Go, is still a library (If it functions as a library it is a library)

A solution would be directly using that C-shared library with cgo and avoid the wrapper The difference is the programmers should now know the inner workings of the C library.

the ultimate solution is just to write the entire LGPL/GPL code in Go Too much work I suppose

The difficulty with shared libraries in Go is not because we don't like shared libraries. It's because they are technically difficult to support, and nobody has figured out how to do it well.

very well understood, Thank you

[ianlancetaylor]

A wrapper around the LGPLed C library for easier use in Go, is still a library (If it functions as a library it is a library)

Yes, but the Go wrapper is not a derivative of the C code, and as such the Go wrapper can be under a different license. The C code, under the LGPL, will be in a shared library, and the Go wrapper, not under the LGPL, will not.

[kiap]

@ianlancetaylor

but the Go wrapper is not a derivative of the C code, and as such the Go wrapper can be under a different license

No sir, that's your technical opinion as an engineer, not a legal opinion

LGPL defines these terms on its own legal universe, Basically, it means you cannot circumvent LGPL by actually writing another library and putting another license to it, doing the same thing, otherwise, this is possible with C, or other languages, if making a library in Go of a library of LGPL, it must be LGPL and be a library, the usage of LGPL directly does not fall under this (as I wrote),

It is well understood legally an LGPL library must be dynamically linked or the application violates this,

This is based on LGPL 2.1 otherwise 3.0 is even more difficult.

A "library" means a collection of software functions and/or data prepared so as to be conveniently linked with application 
programs (which use some of those functions and data) to form executables.

...

 2. You may modify your copy or copies of the Library or any portion of it, thus forming a work based on the Library, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions:

a) The modified work must itself be a software library.

b) You must cause the files modified to carry prominent notices stating that you changed the files and the date of any change.

c) You must cause the whole of the work to be licensed at no charge to all third parties under the terms of this License.

d) If a facility in the modified Library refers to a function or a table of data to be supplied by an application program that uses the facility, other than as an argument passed when the facility is invoked, then you must make a good faith effort to ensure that, in the event an application does not supply such function or table, the facility still operates, and performs whatever part of its purpose remains meaningful.

If Go does not support shared library for Go libraries, it is legally safe to say, not to write wrappers around LGPL C code and statically link it, it is very easy to show the user is violating LGPL

[ianlancetaylor]

No sir, that's your technical opinion as an engineer, not a legal opinion

That is correct. However, it is also my informed opinion due to having worked on free software licensing issues for decades. I was one of many people who contributed to the design of the LGPL in the first place.

LGPL defines these terms on its own legal universe, Basically, it means you cannot circumvent LGPL by actually writing another library and putting another license to it, doing the same thing, otherwise, this is possible with C, or other languages, if making a library in Go of a library of LGPL, it must be LGPL and be a library, the usage of LGPL directly does not fall under this (as I wrote),

It is well understood legally an LGPL library must be dynamically linked or the application violates this,

Yes. In the scenario I described, the LGPL code continues to be dynamically linked. Absolutely nothing changes with respect to the LGPL code. No part of the LGPL code is modified in any way. It continues to be under the LGPL, it continues to be distributed as a dynamically linked shared library. It continues to be possible for the end user to substitute in a different version of the shared library when running the executable.

[kiap]

@ianlancetaylor

Thanks

My problem: Lawyers at our company: Neither GPL nor LGPL has been explicitly tested in a court of law, the linking issue subject to MySQL vs. NuSphere case was settled outside of the court.

So we take the most conservative way; defending in court is too expensive. exceeding the boundaries of compliance to a license is not.

[kiap]

@ianlancetaylor

Some folks contacted me from ITF USA Saying the GPL / LGPL problem with Go can easily be fixed in the most conservative way:

Sending messages from process A (EULA or any license) to process B (GPL licensed), embracing 100% separation

I guess that's doable,

Thanks for @ianlancetaylor 's help and other folks for their input.

[pjanx]

@rsc Is there any issue or discussion to track about your -o proposal here?

My so far hypothetical use case is a multiprocess architecture, with a bunch of small helper programs that possibly depend on C libraries and can be installed separately. Again, it's about size.

I get a feeling that Linux distributions could make use of a "go-rt" package for "std" and link their stuff against that. It would save a bit of bandwidth for utilities.