search

openSUSE / golang-packaging

golang packaging tool for openSUSE
GNU General Public License v3.0
6 stars 13 forks source link

goinstall macro installs useless files #4

Open flavio opened 8 years ago

flavio commented 8 years ago

The %goinstall macro installs all the static library files of the project under %{_libdir}/go/contrib. These files are not useful and we should not include them inside of the final package.

marguerite commented 8 years ago

I'd say it's useful. many cases in devel:languages:go. see golang-x-org stuff.

tboerger commented 8 years ago

I'd say it's useful. many cases in devel:languages:go. see golang-x-org stuff.

No it's not useful. We should disable that entirely, for libs and also for binaries. I will create something for reference.

Library

This is an example how to package only the .go source files since we don't need anything else packaged here. It's using the currently available macros, even if the naming doesn't match that good in this case.

%{go_nostrip}

Name:           golang-github-shurcool-sanitized_anchor_name
Version:        0.0.0+git20151027.10ef21a
Release:        0
Summary:        Go function to create sanitized anchor names
License:        MIT
Group:          Development/Languages/Golang
Url:            https://github.com/shurcooL/sanitized_anchor_name
Source:         sanitized_anchor_name-%{version}.tar.xz
BuildRequires:  golang-packaging
BuildRequires:  xz
BuildRoot:      %{_tmppath}/%{name}-%{version}-build
%{go_provides}

%description
Package sanitized_anchor_name provides a func to create sanitized anchor names.
Its logic can be reused by multiple packages to create interoperable anchor names
and links to those anchors. At this time, it does not try to ensure that
generated anchor names are unique, that responsibility falls on the caller.

%prep
%setup -q -n sanitized_anchor_name-%{version}

%build
%{goprep} github.com/shurcooL/sanitized_anchor_name

%install
%{gosrc}
%{go_filelist} --source

%check
%{gotest} github.com/shurcooL/sanitized_anchor_name

%files -f source.lst
%defattr(-,root,root,-)
%doc README.md LICENSE

%changelog

Binary

This is an example for packaging a tool that includes a binary that can be called. This is always only the case if somewhere in the code we've got a main function. Here we need a package for the binary itself and another package to provide the source files since there are various Go projects out there which provide subpackages that can be imported into totally different projects.

%{go_nostrip}

Name:           golang-github-cpuguy83-go-md2man
Version:        1.0.5+git20160104.2724a9c
Release:        0
Summary:        Convert markdown into man pages
License:        MIT
Group:          Development/Languages/Golang
Url:            https://github.com/cpuguy83/go-md2man
Source0:        go-md2man-%{version}.tar.xz
BuildRequires:  golang-packaging
BuildRequires:  xz
BuildRoot:      %{_tmppath}/%{name}-%{version}-build
Provides:       go-go-md2man = %{version}
Obsoletes:      go-go-md2man < %{version}
%{go_provides}

%description
Tool to converts markdown into man pages

%gosrc_package

%prep
%setup -q -n go-md2man-%{version}

%build
%{goprep} github.com/cpuguy83/go-md2man
%{gobuild} ...

%install
%{goinstall}
%gosrc

%go_filelist
%go_filelist --source

%check
%{gotest} github.com/cpuguy83/go-md2man

%files -f file.lst
%defattr(-,root,root,-)
%doc README.md LICENSE.md

%files source -f source.lst
%defattr(-,root,root,-)

%changelog
marguerite commented 8 years ago

@tboerger

Hi,

I may have some misunderstandings of Go since my knowledge of it entirely comes from packaging experience and through Google Search. So, any correction will be greatly appreciated.

.a is not supported as a first-class citizen in Go

I strongly doubt it. the standard libraries of Go are compiled into .a by default.

it reduces reusability (you can't update the library without updating the Go toolchain)

I don't quite understand this. can you explain in detail?

it requires a lot of rebuilds for no good reason (each Go bump means that obs will spend the next 3 days rebuilding everything)

Each C bump will cause the whole OBS rebuilding for days. That's exactly what we did in the past. What's the problem with that?

In theory, I think you may be right that if a binary have got every library, every go toolchain stuff compiled statically into itself, it won't need to be rebuilt when Go itself is updated.

But golang-packaging requires go for sure. and almost everything inside devel:languages:go requires golang-packaging. No matter what you do, they will get rebuilt. That is RPM dependencies which can't be resolved from Go side.

Below are my research on this topic:

At first, openSUSE discourage the distribution of plain source codes, unless the language itself is an interpreted language like Python. If a language is able to be both compiled and interpreted, the best effort should be to ship prebuilt ones. Even Python has precompiled .pyc and .pyo files. Currently Go packaging looks like it. We provide ".pyc"s through the main package and ".py"s with "-source" package.

Let's change a way of thinking, since C also has interpreters, would you like to distribute plain ".c" and ".cpp" files of all the libraries to prevent from rebuilding and to increase reusability? apparently not. Go has shared library support since 1.5. It is very much like C now.

I know the proposal coming from Fedora way. But to me, it looks like we abandoned all those amazing features of compilation of Go. It is the main reason that I'm not interested in distribution of plain source codes solely.

Secondly, .a and .go share the same purpose. eg:

if you 'go get github.com/cpuguy83/go-md2man', you will get: src/github.com/shurcooL/sanitized_anchor_name src/github.com/russross/blackfriday src/github.com/cpuguy83/go-md2man pkg/linux_amd64/github.com/shurcooL/sanitized_anchor_name.a pkg/linux_amd64/github.com/russross/blackfriday.a pkg/linux_amd64/github.com/cpuguy83/go-md2man/md2man.a bin/go-md2man

if you delete source codes of sanitized_anchor_name and blackfriday, the layout is: src/github.com/cpuguy83/go-md2man pkg/linux_amd64/github.com/shurcooL/sanitized_anchor_name.a pkg/linux_amd64/github.com/russross/blackfriday.a bin/

'go install github.com/cpuguy83/go-md2man' still build because .a is present. and of course:

src/github.com/shurcooL/sanitized_anchor_name src/github.com/russross/blackfriday src/github.com/cpuguy83/go-md2man pkg/linux_amd64/ bin/

will build too because all the source codes are all there.

That is, if we distribute .a. You can start from middle of the build, which is faster. OR, .a serves the same purpose as those source code files.

Third, I think you were trying to say "we can ship 3rd-party libaries as plain source code, and compile binaries who have a "main" function only". Actually we have little such "binaries". Most of the packages in devel:languages:go are libraries. Maybe we can find a special way for docker, eg like: BuildRequires: golang-github-cpuguy83-go-md2man-source directly

And you mentioned we don't need to build all the .a files. Many projects import like this:

import golang.org/x/text/unicode

which can't be fulfilled by shipping only "golang.org/x/text.a" (of course it can be fulfilled by shipping source codes, but it can also be fulfilled by shipping golang.org/x/text/unicode.a)

PS: the specfile of your binary demo will certainly not build. Apparently go-md2man needs blackfriday and sanitized_anchor_name at build time.

Marguerite

cornelius commented 8 years ago

@marguerite I would like to chime in here.

My Go knowledge comes from using Go as a developer. I haven't seen any cases where shared libraries would have been used. Everybody seems to compile binaries statically from all the library sources. There might be use cases where providing .a files is useful, e.g. when extending C programs by functions written in Go, but as far as I can tell these are the rare exception in the Go development world.

So I think having packages of a library can be useful to provide a specific version of the sources and to provide build dependencies of go executables in the build service in a well-defined way. But having the .a files there as well limits their usefulness and adds a lot of overhead. So I would rather not do it.

I think we can get much cleaner, more elegant, and easier to maintain packaging if we follow what upstream does and what developers of Go applications need, and that is statically compiling from sources.

tboerger commented 8 years ago

Thanks @cornelius, can't say it better

marguerite commented 8 years ago

@cornelius

Glad to see you here.

it is true that shared libraries were rarely used. there might be lots of reasons, eg: golang codes are not written with shared support considered (currently we can't use 1.6.2 to build shared libraries, I tried last night with an unresolved/identified bug of go itself), or golang just started to support it in 1.5 and it takes time for these usage to grow blabla...

my point is...as a packaging tool, I have to support it since upstream support builds that way. eg "-buildmode=shared -linked" or "-buildmode=c-shared" ....which may have impact on me because the current codebase to build .a is easy to be extended to build .so too. So I will not face the situation that after a long time an issue is reported saying "how can I build shared with golang-packaging" and I have to rewrite it from zero to support that, which happened in my another project that is nodejs-packaging.

And. as a developer, you just want the "binary" most of the times right?

Then at build time, .a and .go can get the same result. who cares? but the .a files have the good things I mentioned in a single build (@tboerger may be right considering massive build).

And, you can always install -source package. Also, if there's any unwanted .a files created in a project that you just want the binary. it is easy to delete them all before running %go_filelist

I want to remind that currently the go stuff we provide in openSUSE are almost all libraries which are used in development, not in final products. That is, they're used for build, not for end use. eg: it is totally useless for any end user to install a golang-googlecode-log4go.

"So I think having packages of a library can be useful to provide a specific version of the sources and to provide build dependencies of go executables in the build service in a well-defined way"

how to provide a specific version of the sources? that .go can but .a can't?

please define "well-defined" way of yours...because my definition might be different.

"I think we can get much cleaner, more elegant, and easier to maintain packaging if we follow what upstream does and what developers of Go applications need, and that is statically compiling from sources."

The thing is:

I follow what upstream does too. upstream will fetch source code of dependencies, build into .a, and use those .a to build the final application. in my understanding, if I want to build a C program on openSUSE, I don't need to download all the source codes of C libraries, compile and install them. then use them to build the program. I just need to install "-devel" files and build the program. So it sounds like to me that providing a golang dependency is to provide its build result which is the .a files.

And keep in mind, we have few Go applications. I know it's growing all the time. but in devel:languages:go I didn't see many. docker is one. But I have to see the whole picture. I can't focus everything on just one application. or golang-packaging should be renamed into docker-go-packager or something...but the problem is I don't have the chance to see the whole picture right now.

I just want to get a clear view, that what is the exact difference between:

a: compiling everything from source codes b: compiling dependencies first, and use the compiled dependencies to compile the application.

I see no difference. So I provide them both (through the main package and the "-source" package) and choose one to use. Then you guys insisted on making another the default. but the problem is I didn't see any difference that urgently pull in the need.

Any idea appreciated.

Marguerite

tboerger commented 8 years ago

You can't compare go with c or another language we currently support.

flavio commented 8 years ago

First of all this is the complete picture:

I totally agree with what has been said by @cornelius and @tboerger and on the opensuse-go mailing list about static libraries. I don't think they are useful right now. Handling them in the proper way is just causing these macros to grow in size and complexity. This can lead only to more troubles from the debugging and maintenance point of view. I would be willing to go down that path if the benefits outweigh the costs. But so far this does not seem to be the case. Rather it seems we are just doing that to save a small amount of build time and because it's possible.

As for shared libraries, I think it's too early to address them. They are still being worked out by upstream and, most important of all, they are not even used by other Go programs. However I do recognize they might be interesting in the future because they could ease the maintenance of Go packages.

Finally to answer your question:

I just want to get a clear view, that what is the exact difference between: a: compiling everything from source codes b: compiling dependencies first, and use the compiled dependencies to compile the application.

The end result should be the same, but the price to pay is too high:

Note well, none of the major Linux distributions (debian, ubuntu, fedora, rhel) are shipping these static libraries. All of them are just focusing on delivering the source code because this is what matters both for packages shipping Go programs and for developers building their Go code by hand.

I think golang-packaging should focus on the main use case of Go and then, according to real demands, be extended to cover new areas.

SchoolGuy commented 2 years ago

So this issue is still open. I am trying to get into go packaging deeper and deeper. As far as I have read this, it is discouraged to use the %go_install macro?