search

twpayne / chezmoi

Manage your dotfiles across multiple diverse machines, securely.
https://www.chezmoi.io/
MIT License
12.69k stars 475 forks source link

build for AIX #2787

Closed arozehnal closed 1 year ago

arozehnal commented 1 year ago

I really like chezmoi, thanks for it. I tried to create my own build for AIX, but the compilation end with errors. Is there any way to fix it?"

Describe the bug

build error

To reproduce

make install-from-git-working-copy
go install -ldflags "-X main.version=v2.31.0 \
        -X main.commit=461b175d1bb70657c249194bc95452ca82fc16d9 \
        -X main.date=1676974502 \
        -X main.builtBy=source"
# github.com/atotto/clipboard
/go/pkg/mod/github.com/atotto/clipboard@v0.1.4/clipboard.go:10:9: undefined: readAll
/go/pkg/mod/github.com/atotto/clipboard@v0.1.4/clipboard.go:15:9: undefined: writeAll
# github.com/containerd/console
/go/pkg/mod/github.com/containerd/console@v1.0.3/console.go:83:12: undefined: checkConsole
/go/pkg/mod/github.com/containerd/console@v1.0.3/console.go:86:9: undefined: newMaster
# github.com/go-git/go-billy/v5/osfs
/go/pkg/mod/github.com/go-git/go-billy/v5@v5.4.1/osfs/os_posix.go:15:14: undefined: unix.Flock
/go/pkg/mod/github.com/go-git/go-billy/v5@v5.4.1/osfs/os_posix.go:22:14: undefined: unix.Flock

# github.com/twpayne/chezmoi/v2/pkg/shell
pkg/shell/shellcgo.go:33:94: cannot use _Ctype_size_t(buflen) (value of type _Ctype_ulong) as type _Ctype_int in variable declaration
make: *** [Makefile:40: install-from-git-working-copy] Error 2

Expected behavior

binary for AIX

Additional context

OS AIX v.7.2 go version go1.18.10 aix/ppc64

halostatue commented 1 year ago

In addition to the build issue in shell/shellcgo.go (which doesn't look pretty to solve), there are dependencies that require update.

  1. atotto/clipboard#65 prevents compiling for AIX.
  2. containerd/console is also missing aix tags (see below). It seems that simply adding those will be sufficient, but…
  3. go-git/go-billy is not missing aix tags (the posix file uses +build !plan9,!windows,!js), but AIX does not have a flock system call (see below) and it will need an os_aix.go file created that uses what is provided by Go in flock_aix.go.

containerd/console/console_unix.go: https://github.com/containerd/console/blob/65eb8c0396d0cac15c888bcf4d47049c21317b18/console_unix.go#L1-L2

containerd/console/tc_unix.go: https://github.com/containerd/console/blob/65eb8c0396d0cac15c888bcf4d47049c21317b18/tc_unix.go#L1-L2

golang/go/src/syscall/flock_aix.go: https://github.com/golang/go/blob/e7cfcda67d430a6e3cf4e8dfef27416870a2b007/src/syscall/flock_aix.go#L9-L18

I haven’t had access to an AIX system in more than a decade, so I can’t contribute suggested fixes to the upstream projects.

@twpayne I think that the fix for the unsafe getpwnam_r in AIX would be to have an AIX-specific build file that uses the differently-sized pointers as appropriate, but I don’t know of any AIX CI hosts where this could be tested.

halostatue commented 1 year ago

This should probably also be labelled in dependency, but since part of the issue is in Chezmoi, not yet.

arozehnal commented 1 year ago

Thank you for your prompt response. I understand that AIX is not widely used, and that is why there is an issue with its support. I am not a programmer, but I think I am capable of trying any necessary fixes directly on AIX. However, if problems with code compatibility prove to be difficult to solve, it may be better don!t fix this issue and close this report by stating that it is not possible to create chezmoi for AIX.

twpayne commented 1 year ago

As well as requiring updates to dependencies to support AIX, this also requires a recent version of Go for AIX. chezmoi currently supports the same versions of Go as the Go team, i.e. currently 1.19.x and 1.20.x. Older versions of Go don't get security updates.

So, I'll close this as "not planned" for now as it requires updates to multiple dependencies.

twpayne commented 1 year ago

Sorry, I misread your message. If you're willing to update the dependencies, then let's keep this open.

arozehnal commented 1 year ago

I will wait for a newer build of Go for AIX as currently the maximum version available is v1.18.1 see also https://www.ibm.com/support/pages/node/883796

jwbowen commented 1 year ago

I have access to AIX machines and would be willing to help out where I could.

halostatue commented 1 year ago

That would be fabulous, @jwbowen.

As I noted in an earlier comment, there are three upstream libraries that need PRs created and/or merged that fix missing AIX compatibility flags and/or functionality.

It would also be ideal to have someone with access to AIX implement a replacement for the unsafe use of getpwnam_r that uses appropriately-sized pointers or implements something else entirely. There may be some examples on how to do this from the Go source code, as it looks like it calls getpwnam_r for several things (but not the piece required by chezmoi).

arozehnal commented 1 year ago

I have just performed an upgrade of Go https://www.ibm.com/support/pages/node/883796

go version go1.20.2 aix/ppc64 AIX 7.2

===================

make install-from-git-working-copy go install -ldflags "-X main.version=v2.33.6 \ -X main.commit=17359e2dec64c4709d9fdc345c63ad7d1ccd1b2e \ -X main.date=1684278184 \ -X main.builtBy=source" github.com/atotto/clipboard ../go/pkg/mod/github.com/atotto/clipboard@v0.1.4/clipboard.go:10:9: undefined: readAll ../go/pkg/mod/github.com/atotto/clipboard@v0.1.4/clipboard.go:15:9: undefined: writeAll github.com/containerd/console ../go/pkg/mod/github.com/containerd/console@v1.0.3/console.go:83:12: undefined: checkConsole ../go/pkg/mod/github.com/containerd/console@v1.0.3/console.go:86:9: undefined: newMaster github.com/mattn/go-runewidth open ../go/pkg/mod/github.com/mattn/go-runewidth@v0.0.14/runewidth.go: no such file or directory github.com/go-git/go-billy/v5/osfs ../go/pkg/mod/github.com/go-git/go-billy/v5@v5.4.1/osfs/os_posix.go:15:14: undefined: unix.Flock ../go/pkg/mod/github.com/go-git/go-billy/v5@v5.4.1/osfs/os_posix.go:22:14: undefined: unix.Flock github.com/twpayne/chezmoi/v2/pkg/shell pkg/shell/shellcgo.go:31:94: cannot use _Ctype_size_t(buflen) (value of type _Ctype_ulong) as _Ctype_int value in variable declaration gmake: *** [Makefile:46: install-from-git-working-copy] Error 1

twpayne commented 1 year ago

Thanks for bumping the Go version on AIX. The various errors here are from dependencies that do not yet support AIX. You will need to fix each dependency.

Do you know if it is possible to get AIX for zero cost and run it in a VM? If so, I would be able to help with some of the packages.

halostatue commented 1 year ago

So, none of the issues identified in the earlier comment have been addressed. The only one that we can address directly is the one in v2/pkg/shell, but as I don’t use AIX, there’s nothing that I could do in order to resolve that.

It also appears that there may be a new issue with mattn/go-runewidth, but the error there seems really odd.

I’ve marked this issue as help wanted, because it will require working with upstream package providers to resolve their issues in addition to resolving the issue in the chezmoi code itself.

twpayne commented 1 year ago

So, none of the issues identified in the earlier comment have been addressed.

Go has been updated from 1.18 (unsupported) to 1.20 (supported). chezmoi uses some standard library functions that are only available in Go 1.19 and later.

halostatue commented 1 year ago

So, none of the issues identified in the earlier comment have been addressed.

Go has been updated from 1.18 (unsupported) to 1.20 (supported). chezmoi uses some standard library functions that are only available in Go 1.19 and later.

The errors reported in the build, barring the new one for mattn/go-runewidth, are the same as were identified in the original build and are the outstanding errors.

I think that the mattn/go-runewidth error is a red herring as there’s nothing in there that should prevent building on AIX.

All of the code issues are fixable; there’s nothing hard about any of the changes, but a lack of access to AIX and the social aspect of using other projects code and convincing them to support AIX (without easy access) is going to be the main limiting factor.

arozehnal commented 1 year ago

@twpayne AIX for zero cost doesn't exists, AIX in my opinion can not be run under VM. But if I get the fixed code, I will be happy to test it on AIX.

I have solved issue mattn/go-runewidth, it was a collision of different versions. For my own personal use, I am able to extend build tag. I think I don't even need the clipboard module, because I don't use clipboard on AIX. The worst are the problems with v2/pkg/shell/shellcgo.go and flock system call on AIX.

twpayne commented 1 year ago

I think I don't even need the clipboard module, because I don't use clipboard on AIX.

github.com/atotto/clipboard is a dependency of github.com/charmbracelet/bubbles/textinput:

$ go mod why github.com/atotto/clipboard
# github.com/atotto/clipboard
github.com/twpayne/chezmoi/v2/pkg/chezmoibubbles
github.com/charmbracelet/bubbles/textinput
github.com/atotto/clipboard

It is probably sufficient to add aix to this line.

The worst are the problems with v2/pkg/shell/shellcgo.go and flock system call on AIX.

github.com/twpayne/chezmoi/v2/pkg/shell
pkg/shell/shellcgo.go:31:94: cannot use _Ctype_size_t(buflen) (value of type _Ctype_ulong) as _Ctype_int value in variable declaration

This error is confusing because the AIX documentation for getpwnam_r states that the third argument to getpwnam_r is a size_t, and chezmoi already uses a size_t. Do you know if the AIX documentation is wrong? Can you find out what the type of the third argument to getpwnam_r is on AIX?

For flock, I have no idea how to AIX handles file locking and do not know where to start on this one.

Since we can't even build for AIX without a copy of AIX (it requires cgo and no cross-compiler is available AFAIK), you'll have to try and test these changes locally. In case you're not aware, the replace directive in go.mod allows you to use a locally-modified version of a Go module when building locally, which can help you build and test local changes.

halostatue commented 1 year ago

I think I don't even need the clipboard module, because I don't use clipboard on AIX. It is probably sufficient to add aix to this line.

Hopefully with two PRs that do the same thing, the upstream provider will take action on it.

The worst are the problems with v2/pkg/shell/shellcgo.go and flock system call on AIX. This error is confusing because the AIX documentation for getpwnam_r states that the third argument to getpwnam_r is a size_t, and chezmoi already uses a size_t. Do you know if the AIX documentation is wrong? Can you find out what the type of the third argument to getpwnam_r is on AIX?

The problem is the size of size_t in this case. I looked at this several months ago and I think that the pointer value is 128bit but size_t is 64 bit or something similar. I can’t find the references now, but it’s worth noting that the Go code for this is more than a bit convoluted.

Callsite: https://github.com/golang/go/blob/40bdc5618053c16e171473b99186affcbda2166c/src/os/user/cgo_lookup_unix.go#L30C2-L31

Main definition: https://github.com/golang/go/blob/40bdc5618053c16e171473b99186affcbda2166c/src/os/user/cgo_lookup_cgo.go#L83-L87

Cgo definition: https://github.com/golang/go/blob/40bdc5618053c16e171473b99186affcbda2166c/src/os/user/cgo_lookup_cgo.go#L32-L39

So it shouldn’t be causing a problem, but note that for some reason, cgo is using what appears to be some sort of trampoline function rather than the direct call.

For flock, I have no idea how to AIX handles file locking and do not know where to start on this one.

Since we can't even build for AIX without a copy of AIX (it requires cgo and no cross-compiler is available AFAIK), you'll have to try and test these changes locally. In case you're not aware, the replace directive in go.mod allows you to use a locally-modified version of a Go module when building locally, which can help you build and test local changes.

The flock changes are, I think, easier than the pointer issue, but they would need to be upstreamed to go-git/go-billy or kept as an override dependency.

The problem is that AIX doesn’t have flock, and this code assumes that unix.Flock will just work:

https://github.com/go-git/go-billy/blob/ed16d7ba55f95b937e995cb7890e91e1e785b7be/osfs/os_posix.go#L11-L22

What needs to be done instead for go-billy is that there needs to be an AIX-specific implementation that uses FcntlFlock instead:

https://github.com/golang/go/blob/e7cfcda67d430a6e3cf4e8dfef27416870a2b007/src/syscall/flock_aix.go#L9-L18

twpayne commented 1 year ago

Gently closing this because fixing this will require a lot of work outside the scope of chezmoi. Very happy to re-open this issue when a few of the upstream issues have been fixed.