fix for SLE11SP3 - Githubissues

marguerite commented 8 years ago

Sorry, I don't know how to send parts of my work as pull request...any one help?

marguerite commented 8 years ago

@tboerger @flavio @jordimassaguerpla

Hi, guys,

I just update the code.

now golang-packaging distributes plain source code and have no runtime requires for static builds.

a working demo is here: https://build.opensuse.org/project/show/home:MargueriteSu:branches:devel:languages:go

see:

golang-github-shurcool-sanitized_anchor_name
golang-github-russross-blackfriday
golang-org-x-text
golang-github-cpuguy83-go-md2man

these are 4 dependencies of docker.

I hope in this way I will not stand in front of your way for packaging golang programs like docker/etcd and etc.

marguerite

marguerite commented 8 years ago

I still keep the build step for all go packages for two reasons:

Firstly, if we just package plain source codes without even building them, we will not find any build error that brings the need to patch source codes (or run go fix command). that'll be packages all over the devel:languages:go project that will "build" successfully and pass all OBS tests until someone actually use it in real life, that is, we'll distribute something that may not work. of course, you'll see the errors in the buildlog from your side if you are a "go program" packager. but actually it's not your responsibility to fix them. we should expose the errors to the one who actually packages/maintains the dependency. and there're even a few packages without any downstream use that locates in the project.

Meanwhile, I still want to experiment the shared build possibility. Go 1.5 supports it although no actual program is actually using it. but as a toolchain developer and a person who has interest in that, I think I should keep that extendability. of course, I shouldn't stand in front of anyone's way.

marguerite

sysrich commented 8 years ago

Meanwhile, I still want to experiment the shared build possibility. Go 1.5 supports it although no actual program is actually using it. but as a toolchain developer and a person who has interest in that, I think I should keep that extendability. of course, I shouldn't stand in front of anyone's way.

Can you please consider experimenting with this in a seperate project from devel-lanaguages-Go? It sounds like an invasive experiment that would potentially block other standard use of Go, like Docker very much requires.

marguerite commented 8 years ago

@sysrich

Hi,

golang-packaging is a tool to universally package golang stuff
shared build is an official feature supported by golang 1.5.

so the word "invasive" is wrong. shared build is a "must" feature of golang-packaging. and I have separated this shared build case from the existing working static one. there's little chance to break static build from the possibility to be able to shared building. a requirement to remove one condition to not break another condition is just wrong. I have seen some attempts to "clean" golang-packaging which remove lots of fundamental stuff. that indicates a full rewrite in the future to add shared build back. it'll be more painful.

the standard use of Go is fulfilled by Go itself, not by golang-packaging. you can "standard use" any time with and without golang-packaging. golang-packaging is just a helper that aims to make packaging Go easy, it tries to support as many programs as it can. it is a combination of "standard use" so there's no way for it to block other standard use of Go. packager can still use the standard upstream procedures.
Docker is just one of the Go programs. we can't apply a special need universally. especially when its upstream provides a build script and it doesn't use golang-packaging at all now. all BuildRequires of it is go-md2man, which contains a binary to convert manpages for it. both the old way and the modified new way in my current commit will produce the same binary.

so literally there's no way for golang-packaging to block Docker. even before the discussion and my commits, golang-packaging didn't block Docker. because Docker is currently out of golang-packaging's scope.

based on that, it should be Docker packagers to submit feature requests to support it, not stand in front of my way to block me from developing new features of my program. actually I don't think it is right to build Docker in another way even with such feature request submitted since its upstream has made a build script.

the idea to distribute go codes in compiled format (".a") was not orginally coming from me, but from previous repository maintainers of devel:languages:go since go 1.x, which has a longer history than the way currently use by Fedora. they didn't choose our way, but it doesn't mean our way is wrong. there're many ways to Roman.
as I said, the build of codes is a must. so there's no need to forclly ship ".go" when there's already ".a" since they're identical. I switched to package in plain source code format just because I want to make you guys happy since what I really care is the build process not the result (the results are identical to me, both of them can be used further to build working programs).

but you guys just gave two reasons: Fedora uses it and cost of resources. the first is not even a reason, it is a fact. for the second, the resources are needed and not costs. they are used to find out errors and keep OBS review process working.

we use the main package eg go-md2man to provide "-devel" packages since there's no header file in golang. there's no any "sanitized_anchor_name" line in go-md2man's codebase. that import comes from "blackfriday". but when linking go-md2man binary, it needs sanitized_anchor_name, no matter whether the codes from sanitized_anchor_name are compiled statically into "blackfriday" or not (it needs sanitized_anchor_name.a not blackfriday.a, it's golang upstream who decided that way). so literally "sanitized_anchor_name" is a runtime dependency of "blackfriday-devel" which is provided by blackfriday itself. so there's no problem of runtime requires at all. actually even fedora applies "Requires" on golang-github-russross-blackfriday-devel which contains the ".go" source codes, which is idential to our "golang-github-russross-blackfriday" now.

I removed the runtime dependencies just to make you guys happy. because anyway you can "BuildRequires" sanitized_anchor_name for go-md2man to workaround it (which means lots of specfile modication of course)

so I think you were misleaded. I strongly suggest you guys get your hands dirty, start to package a few golang packages and compare them to the existing ones in Fedora. Then you''ll know why I chose to develop golang-packaging that way.

no offense but I'm really tired of reoccuring the errors I met to demostrate why my way is right. I think you guys don't catch up, having an "user view" that tries to apply "go principles" on things that upstream didn't say at all.

Marguerite

sysrich commented 8 years ago

shared build is an official feature supported by golang 1.5.

If by 'shared build' you mean the 'vendoring external dependencies' feature mentioned here: https://golang.org/doc/go1.5

The detailed documentation makes it very clear this is highly experimental https://docs.google.com/document/d/1Bz5-UB7g2uPBdOx-rw5t9MxJwkfpx90cqG9AFL0JAYo

If you mean the -dynlink feature in the 1.5 compiler, then that is supported by the amd64 architecture only which I think is not suitable to be used that heavily when we need to support Tumbleweed on at least i386, amd64, and aspiratioanlly on aarch64, ppc64 and s390x in addition.

I know you're passionate about this marguerite but can you please at least consider a compromise in the contentious areas raised in this PR? This is all I am asking..

marguerite commented 8 years ago

@sysrich

yes, we're talking about the same thing. but here it is:

static build is one feature and shared build is another. they're isolated in my codebase.

there's an option to use shared build. it means shared build will not be applied to all golang packages. I didn't even think about it. everything will just keep as it is. it will be selectively applied only when the packager decide to turn the switch on, on a "single package" basis. so actually it doesn't and will not affect anything existing in devel:languages:go.

even if we drop shared build in this case, the codebase will just remove one condition. it doesn't affect the other condition, that is, static build.

here's what golang-packaging will do currently:

prepare the build environment and buildroot
build golang codes statically
install golang source codes that are able to build to buildroot.
pack the rpm

if I understand correct, the only thing I didn't do but requested by the SUSE team is to skip the build process, to "save Build Service resources". but if I do that, things will turn to be:

prepare the buildroot
install golang source codes to buildroot.
pack the rpm

but a "no build at all" way will introduce many further problems we both know. So I think I am right to keep the build process. the SUSE team will get all they requested, for codes that are able to build, in a package.

that is, I have made my compromise in my last commit...if you see the demos I provide in my branched project, you will know that.

flavio commented 8 years ago

Firstly, if we just package plain source codes without even building them, we will not find any build error that brings the need to patch source codes (or run go fix command). that'll be packages all over the devel:languages:go project that will "build" successfully and pass all OBS tests until someone actually use it in real life, that is, we'll distribute something that may not work. of course, you'll see the errors in the buildlog from your side if you are a "go program" packager. but actually it's not your responsibility to fix them. we should expose the errors to the one who actually packages/maintains the dependency. and there're even a few packages without any downstream use that locates in the project.

devel:languages:go is a place where package dependencies are located. I expect most of the times it will be the maintainer of program X to package also dependencies Y and Z.

As for the packages without any downstream users, I can only assume someone created them because he's using them for some private project that might not even be located on OBS. In that case it's his job (as package maintainer) to make sure they keep working.

Finally, building a library is not enough to make sure it actually works. The best solution would be to have a %check section triggering the unit tests of the project.

I'm against shipping .a files, not building them. But building them should be done for proper (aka unit) testing.

flavio commented 8 years ago

I want to address the remaining issues with a single message

The mission of golang-packaging

This is how we see golang-packaging

A set of macros that have to be used to build golang packages.
Packages should be built in a safe and consistent way.
It has to work both with libraries and binaries.
All the golang binaries should be able to reuse the macros relevant to them. For example Docker does not build itself using the traditional go build approach, but can definitely use some of the best practices enforced by golang-packaging.
Something easy to understand and maintain.

golang-packaging should not be a set of macros that allow users to play with whatever experimental, unknown feature of the go compiler. This is a "tool for the masses". It covers the most common usages.

Right now we have a 300 LOC script with lot of corner cases to cover. This is the opposite of what I described above.

Getting our hands dirty

We are getting our hands dirty, @tboerger is already working on a proposal for golang-packaging.

You can find his work here and inside of this branch.

There are still some small issues to be solved, but we like how the project is turning out. We think it reflects all the goals I wrote above. As usual you are welcome to provide feedback about it.

Not reinventing the wheel

We didn't start coding out of the blue our proposed solution. Before doing that we did some research and evaluations. This is why we are convinced about not shipping static and shared libraries.

What we decided is not written in stone. The go community is evolving and we want to make sure golang-packaging reflects the true needs of a modern Linux distribution.

Please take a look at this extract from the Fedora go packaging guidelines:

At this time, Go libraries packaged in Fedora are primarily for the purpose of being BuildRequires for building Fedora binary RPMs, and not meant to be developed against otherwise -- for that, we encourage the upstream "go get" idiom and a per-user $GOPATH.

We do not provide or recommend a system-wide GOPATH that users should inherit and there is no notion of a "site" or "vendor" path for system libriaries.

GOROOT is reserved for golang standard library only. This way developers do not need to be concerned with library path conflicts, but instead can choose to include the system path in their per-user path (e.g. export GOPATH=$HOME/go:/usr/share/gocode). This way a call to go get ... would land new source in the $HOME/go directory

The standard golang compiler only produces static libraries. There is little value in shipping these prebuilt, especially since these libraries are very specifically tied to the exact minor release of the golang compiler. Instead, each library package should consist of a -devel subpackage which installs .go source code to /usr/share/gocode/src, under the appropriate import path.

The following extract is from debian go packaging guidelines:

Libraries written in Go are packaged for Debian with the only purpose of building other Go programs for Debian. They are specifically not available for users in their regular development workflow. For that, users should use go get. The rationale behind this decision is:

By using go get you are forced to set up the environment variable $GOPATH which you need for other go commands (e.g. go build, go test, etc.).

Debian packages install files to system locations, which users cannot modify. That means that users would not be able to upgrade packages with the canonical go get -u . Even worse, when using sudo to forcibly modify the system files, it still would not work since no VCS information is contained in the Debian packages.

As you can see we are not the only one thinking .a files are useless, even dangerous in certain cases and, most important of all, useless to people developing Go programs.

Go developers will use what upstream is recommending: vendoring. Packagers can rely on the contents of devel:languages:go to build stable and consistent packages by packaging the libraries and the go binaries using the same set of macros.

Conclusion

We really respect your opinions and we value your contributions. However I'm personally getting tired of going over and over these arguments.

This is how we (the SUSE guys who care about Go, not @tboerger) and other major Linux distributions think Go packages should be built at this right point in time. Things might change in the future and if that happens we will be glad to change it.

In the meantime this is how we want to proceed with Go packaging. We are ready to take responsibility of this choice by maintaining go, golang-packaging, the Go packaging guidelines for openSUSE, assisting openSUSE users on the go-dedicated mailing list and the packages we care about inside of devel:languages:go.

As you can see we ready and willing to "get our hands dirty".

marguerite commented 8 years ago

@flavio

Hi,

devel:languages:go is not only a place for package dependencies. it's a place for anything of go in openSUSE (see grafana, influxdb, ctags there). and it is used as a development project for openSUSE Factory.

I expect the program maintainer to be the dependencies maintain too. but that's not always possible. eg, most of the programs rely on golang.org/x/{text,net} stuff, but that package is actually maintained by me. I expect the help from downstream packagers. but here's how I maintained it: I didn't package any actual Go program, I maintain the go itself and packaged some of its standard libraries to make the development environment possible for developers. see? common dependencies come at first then programs follow.

there're plenty of reasons for an orphan package inside devel:languages:go. The 1st generation of maintainers created some of the common libraries that are popular at their time. along with time, most of them are not used by any newer programs. for a long time, people just fix builds for Factory and no one care to send a drop request since most of times they're not inside your scope. Some of the go programs that not build are dropped (like the "liteide") but the dependencies are still there because they actually still builds. you'll agree if you maintained some projects: it's not always possible you can find a maintainer, and it's not always possible that you can drop a package.

based on this, I treat the actual build Essential, because we can not always rely on downstream packagers to find the build problem of the dependencies.

And I agree a pass build doens't gurantee a working library. I agree we should implement the unit testing. I will do this this weekend.

flavio commented 8 years ago

based on this, I treat the actual build Essential, because we can not always rely on downstream packagers to find the build problem of the dependencies.

Even if a library is successfully building there's no final guarantee it will work inside of a program using it. Unfortunately most Go projects are not tagging their releases and not using semantic versioning. It's not the first time a library changes its API between random git commits causing some programs to fail during their build time.

I don't see a real value in building everything indiscriminately.

And I agree a pass build doens't gurantee a working library. I agree we should implement the unit testing. I will do this this weekend.

Testing is a different thing. This could be a valuable step to add to the packages. However not all programs and libraries have unit tests. It should be up to the packager to add the %check section.

Above all:

where are you going to add this code? It wouldn't make sense to add it to the current ruby implementation given it's going to be replaced by the code I pointed in the previous comment.
what is going to be the purpose of this code, how should it looks like, why should a user use this code instead of calling go test manually? These are all questions we ask ourselves before writing any new feature (new code == more maintaining efforts == more chances of bugs).

I think it would be better to create a new issue to track this feature and provide the answers to the questions I wrote before. Just to be clear, you don't have to write any actual code. This is just a design and discussion phase that can prevent useless work and/or point your efforts in the right direction.

Also, I'm going to close this pull request given we are not going to have this ruby code around anymore.

marguerite commented 8 years ago

@flavio

wtf? we are not going to have this ruby code? who decide that and why?

flavio commented 8 years ago

@marguerite I explained to you what is going to happen inside of this comment. You replied ~5 hours ago to the first comment and didn't provide any feedback on the second. I assumed everything was clear.

Moreover I know @sysrich, in his role as chair of the openSUSE board, informed you we are going to become the maintainers of devel:languages:go. This communication should have been delivered to you at the beginning of the last week. Yet again there was no answer from you. @sysrich please correct me if I'm wrong.

We will always be grateful for your contributions and we hope you will keep providing us your opinion and expertise. I know we have a different point of view on this topic, however we hope you will still be interested in contributing to Go at openSUSE.

marguerite commented 8 years ago

@flavio

Hi,

I didn't reply the second comment because I'm still thinking of it. and I don't even know it is "going to happen". in my view, it is an argument from your team. it's not a final notifice at all. because you have no right to make such decision.

I'm the intial author of golang-packaging and the owner & maintainer of golang-packaging package. I didn't ever libterally give up my rights. this fork and the replacement of upstream URL in specfile are not authorized/reviewed by me. I see you guys as help hands from SUSE and patch submitters although you guys didn't submit any new features to my project (but help fixed something) and kept arguing with me/dragging my legs back for the features I have implemented. that is exactly why I kept nodejs-packaging under my own control.

during the argument, there were conflicts in opinions. you guys brought openSUSE board in, who tried to set up a meeting but with no success. I replied that I had time on Friday and Weedend, but then you still asked me for time on the work days. so nothing actually happened.

A board member asked me to conpromise on this. so I implemented the things we both agreed in my project. I thought this might be a new start of collaboration.

I didn't receive anything that you're going to become maintainers of devel:languages:go. even if you are the maintainers now, I am still the maintainer of devel:languages:go and have longer history of maintaining that repository. then we still have to negotiate. because we wear the same heats and I have something superior because I am the first author of this project and the package on OBS.

then suddenly I received a notice from you that my codes are going to be replaced, even the codes to replace are not written/not working in production environment at all now. and it sounds like I'm going to be kicked out from owning my own project and package.

Please keep in mind that openSUSE is a community. You guys seem to have a tone that suppose your in-house discussion known by everyone. Since the beginning, in my protest on the opensuse-go mailing list, you guys kept telling me that "something has been discussed and decided" that I never have the chance to know.

and please keep in mind that I am the author of this project and it is you guys who cooperate with me to get things done instead of the opposite. If you can't work together with the community in the community member's project, please reimplement things in and just in the SLE.

Now I have to treat this "going to be" action as a rude take-over, sadly. I didn't ever receive any invitation to discuss the replacement at all! I don't think this is collaboration and this is what co-maintainer should do. to maintain a project, you have to know the project. you guys don't even know with the latest commits now my project can do the things you wanted at all. you just want the namespace, then replace the current stuff, and send a "warm welcome" to the original author who is still active and didn't ever give up her project at all.

If it is true, all I can do is to use my rights and switch golang-packaging back to my own branch. you're still welcome to contribute to my project, though.