os: introduce Executable to return the path of the current executable

minux commented 9 years ago

This proposes to introduce a new API in package os that addresses #4057.

package os

// Executable returns the path name for the executable that starts the
// current process. The result is the path is used to start the the current
// process, but there is no guarantee that the path is still pointing to
// the correct executable.
//
// The main use case is finding resources located relative to an executable.
//
// Not all OS support this, but all popular ones do.
func Executable() (string, error)

This API has been proposed before, but rejected on the ground that it's impossible to formally define its semantic, but that's because the old proposal says the result can be used to re-exec the current process. If we relax the condition that it returns the original path that starts the current process, its semantic is fully defined on all supported platforms. Not being to re-exec the current process is not a big issue as the primary use case is to find relatively positioned resource files (i.e. OS X app bundles.)

The implementation is already available in https://golang.org/cl/6736069, we just have to brought it up to date.

I'd like to thank @kardianos for the suggestion of the new semantic.

bradfitz commented 9 years ago

SGTM

adg commented 9 years ago

SGTM

davecheney commented 9 years ago

If this proposal could be adapted to remove GOROOT, and it seems as if it would, then you have my enthusiastic support

On Tue, 29 Sep 2015 11:37 Andrew Gerrand notifications@github.com wrote:

SGTM

— Reply to this email directly or view it on GitHub https://github.com/golang/go/issues/12773#issuecomment-143919178.

minux commented 9 years ago

I remember that Rob explicitly mentioned that he dislikes commands that behave differently depending on where they are located.

The only problem of using os.Executable to detect GOROOT is that it won't help if the user hardlinks the go command to another place (symlink should be fine as cmd/go can resolve the symlinks in the result of os.Executable.)

That said, I'm also in favor of using this new API to detect GOROOT automatically, at least for the common cases.

adg commented 9 years ago

I think we should leave discussions of GOROOT out of this specific proposal, which is nicely self-contained and easy to evaluate as-is.

davecheney commented 9 years ago

Understood.

On Wed, 30 Sep 2015 13:18 Andrew Gerrand notifications@github.com wrote:

I think we should leave discussions of GOROOT out of this specific proposal, which is nicely self-contained and easy to evaluate as-is.

— Reply to this email directly or view it on GitHub https://github.com/golang/go/issues/12773#issuecomment-144270511.

alexbrainman commented 9 years ago

SGTM

adg commented 8 years ago

Let's do it.

rsc commented 8 years ago

I am very skeptical of this. I will wait until I see the revised CL I guess.

gopherbot commented 8 years ago

CL https://golang.org/cl/16551 mentions this issue.

rsc commented 8 years ago

Based on the discussion on the CL, I've removed the proposal accepted label. I think this needs more thought.

It is worth noting that the proposal is "do X" instead of "solve problem X". If the goal is to find "resources" maybe we need to think about that problem directly.

robpike commented 8 years ago

As @rsc says, it's not clear what the problem is that is being solved. This is a solution, not a problem. I think the problem should be clearly elucidated and from that we can decide if there is something worth doing.

kardianos commented 8 years ago

@robpike There are three primary problems I solve with the Executable call today:

Finding configuration properties for windows services.
Finding resources (CSS, HTML, SQL, ...) for services.
For github.com/kardianos/service, finding the default executable to register in the system service manager such as windows service, systemd, or launchd.

Windows services always start up with an initial working directory of %SYSTEM%, usually C:\windows\system32 (there is no way to configure this) . In this case the os.Args[0] doesn't give the correct path either if memory serves. I often place configuration files for windows services adjacent to their executable for easy access and deployment. The other ways to solve this problem is to pass in arguments when installing the service pointing to a root folder or configuration file, or by reading and writing to the windows registry. I don't like dealing with the registry and prefer simple local config files.

Resources such as CSS, HTML, or other files I either place adjacent to the executable or zip up and embed within the executable. Unless there is some deployment constraint, I prefer placing resources adjacent to the executable directly on the filesystem. I could use on os.Args[0], but I prefer not to rely on it because it can be faked. If I see the executable next to the resources on the filesystem and I use Executable(), it doesn't matter how the thing is started up, it will find the resources.

kostya-sh commented 8 years ago

Another use-case is a program that starts itself possibly with different command line arguments and/or environment.

gocode does that to automatically start the server daemon if it is not running. On Jul 19, 2016 11:00 PM, "Rob Pike" notifications@github.com wrote:

As @rsc https://github.com/rsc says, it's not clear what the problem is that is being solved. This is a solution, not a problem. I think the problem should be clearly elucidated and from that we can decide if there is something worth doing.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/12773#issuecomment-233779693, or mute the thread https://github.com/notifications/unsubscribe-auth/AGy9Ax5_5t_2EKdSza3UqHt40-l5-GmOks5qXUjygaJpZM4GFIez .

robpike commented 8 years ago

@kostya-sh You can do that by examining os.Args[0]. It does not require the full path name. Plus as said above, I strongly dislike this practice as it's unnecessary, tricky, and mysterious in practice. I have never seen a case where there wasn't a cleaner way to achieve the same result.

@kardianos Is it even possible to discover the executable's correct full path on Windows? I have heard claims to the contrary, but I am no expert.

kardianos commented 8 years ago

@robpike Yes, Executable() is reliable on Windows. I've never seen it go wrong with the correct API call.

matrixik commented 8 years ago

I'm using https://github.com/kardianos/osext because os.Args[0] is unreliable on platforms I use: Windows and Linux (getting real exec path with running my programs in cron never worked for me on my VPS).

Simple program for tests: https://github.com/matrixik/executable-example

http://stackoverflow.com/questions/12090170/go-find-the-path-to-the-executable/ http://stackoverflow.com/questions/18537257/golang-how-to-get-the-directory-of-the-currently-running-file/

minux commented 8 years ago

Examine os.Args is unreliable on at least all Unix systems. The proposed implementation on Windows is reliable unless you play tricks to delete the current program while it's still running, but I don't think you can continue to run normal Go code after doing that (the truck requires running from stack to unmap the executable.)

In cases where the Linux implementation gives incorrect path, it's because the user has moved or deleted the program after it gets started. There is nothing we can do about that and I don't think any user of os.Executable will do that.

mdempsky commented 8 years ago

FWIW, OpenBSD has been opposed to adding kernel support for returning the running process's executable path.

minux commented 8 years ago

What's the reason for OpenBSD's opposition? An os function does not need to work on every supported platforms. We can just document the restrictions. If memory serves, Plan 9 also doesn't provide real path to the executable (only a way to re-exec it.)

mdempsky commented 8 years ago

@minux The prevailing arguments in OpenBSD for opposing it as a kernel feature are pretty much the same as @robpike's earlier arguments for opposing it as a standard library feature.

Of course, the OpenBSD ports tree maintainers would like it to be there to simplify their jobs of porting code full of Linux-isms to OpenBSD, but it hasn't been a showstopper to date to my knowledge.

minux commented 8 years ago

How could gdb -p $pid figure out which executable to read symbols from on OpenBSD?

adg commented 8 years ago

@robpike given that there is a reliable mechanism to discover the program executable under Windows, and that it is distinct from examining os.Args[0], does that assuage your worries about this change? After all, package os is supposed to be an abstraction layer between the Go programmer and the operating system. This seems like a fine addition to the package, IMO.

jimmyfrasche commented 8 years ago

I'm not particularly for this, but SDL 2 introduced a pair of functions for this sort of thing. Their documentation contain a number of remarks, use cases, and warnings about special cases that may enlighten the debate:

robpike commented 8 years ago

I still stand by my original point that there's a solution here but the problem is ill defined, and it seems other solutions exist. But I understand that other people think it's important and it seems easy to add.

So I won't approve this, but I won't veto it.

Russ should also weigh in but he is out for a while.

kardianos commented 8 years ago

@robpike Thank you for your feedback.

the problem is ill defined

Could you be more specific here? I'm unsure what you find ill-defined of the above.

it seems other solutions exist

I can agree with this. Let me take the windows service problem for as a poster child of why I use this. There are four ways from a windows service to find a configuration:

Put the configuration file in a known location. This has problems as drive letters are often C:\, but may be any other letter (esp for Program Files). Not a good option on windows.
The official central location for configuration on windows is the windows registry. This works but go has limited support for w/r registry values. Most importantly this means a larger more intrusive install step that isn't cleaned up when you just remove the folder.
Pass in an initial root path or configuration path to the windows service when it is installed. This works and is how the Oracle DB does it on windows. This works, but can be trickier for ops to install then...
Put the configuration into a well known location relative to the executable. Ops like it because it is copy and register the service.

Now I also use the same technique on non-windows platforms because of, habit, same code base, portable, and it remains easy.

I respect your decision and agree that Russ should have buy in. But I did want to try ensure I mapped out the alternatives that I see. If I have missed one or if you feel I mischaracterized an option, let me know. I support it going into "os" because that's where people tend to look for it and people get it "wrong" otherwise:

http://stackoverflow.com/questions/12090170/go-find-the-path-to-the-executable http://stackoverflow.com/questions/18537257/golang-how-to-get-the-directory-of-the-currently-running-file

robpike commented 8 years ago

What I mean by "the problem is undefined" is that it seems the goal here is to have some sort of "application" that is a bundle of items, and that the binary can discover the bundle when it executes. Asserting that this feature will address that is perhaps true, but it seems the real solution could be much better thought out.

alexbrainman commented 8 years ago

What I mean by "the problem is undefined" is that it seems the goal here is to have some sort of "application" that is a bundle of items, and that the binary can discover the bundle when it executes.

Another real life scenario. Windows service is started automatically by OS every time OS boots. For that OS keeps list of all Windows service executables (full paths). So if you want your Windows service to be started by OS, you have to give OS your executable path. You can rely on user to enter correct path, but it is much easier and safer if the service executable could determine its own path.

I didn't know that Windows had an API for it when I wrote this

https://github.com/golang/sys/blob/master/windows/svc/example/install.go#L18

I should have used GetModuleFileName in there instead.

Also nearly all Windows GUI executables include "resources": icons, bitmaps, form dialogues and others. Windows GUIs use resources extensively when they run. But appropriate Windows APIs find resources automatically. So you don't need to worry about finding your executable path.

Alex

minux commented 8 years ago

Right. Not only for Windows, bundled applications (where program locates its resources relative to its executable file) are also the norm on OS X and iOS. It's just uncommon for Linux or other Unix because most software are installed from source and can afford to hardcode paths. However, for commercial software, it's very common to have relocatable packages in that you can install it to any location and it will just work. The ELF file format even has $ORIGIN to support referencing shared libraries relative to the executable. Therefore I won't call the need undefined: It's actually well defined and has well defined (and deployed) solutions too.

adg commented 8 years ago

Approved. @minux do you want to take care of the implementation?

minux commented 8 years ago

I will take care of the implementation. Thanks for the decision.

matrixik commented 8 years ago

Can't https://github.com/kardianos/osext source be used? It's stable from long time and I think it have proper license. @kardianos

minux commented 8 years ago

I think that was based on my first implementation (3 years and 10 months ago) at https://codereview.appspot.com/6736069/.

kardianos commented 8 years ago

The go4.org/ouutil has the correct license.

On Mon, Sep 12, 2016, 22:28 Minux Ma notifications@github.com wrote:

I think that was based on my first implementation (3 years and 10 months ago) at https://codereview.appspot.com/6736069/.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/12773#issuecomment-246578004, or mute the thread https://github.com/notifications/unsubscribe-auth/AAuFsRL3EGhR7IZb8yfvWGrltKIRNnOIks5qpjRqgaJpZM4GFIez .

bradfitz commented 8 years ago

^typo. https://godoc.org/go4.org/osutil

kardianos commented 7 years ago

@minux did you want to push this forward before November 1? I could also send a CL.

rsc commented 7 years ago

@adg, the discussion on here before Sep 12 was basically @robpike and me saying this didn't make sense, and then you marked this proposal accepted without further elaboration. I assume there was a discussion with the proposal reviewers. Can you summarize the rationale? Thanks.

rsc commented 7 years ago

@kardianos, @bradfitz, go4.org/osutil/osutil.go does not have a correct license for this purpose. Both the copyright line (go4 authors) and the license itself (Apache) differs from the Go project. Please do not copy code from that repo into the Go tree without being very careful about checking that the authors have CLAs on file and agree to the code going to Go.

bradfitz commented 7 years ago

@rsc, this was approved after discussion in a proposal review meeting in which @robpike was (I believe) present. Rob, you cool with this?

As for the CLAs & license: all go4.org commits require the Google CLA, just like Go. And the only committer to that directory is @kardianos, who is also a Go contributor. So we're good on both fronts.

robpike commented 7 years ago

I wasn't so much cool with it as saying I wouldn't stand in the way as there seemed to be so much passion for it. Although I still think in general it's the wrong approach, there were arguments about the way various OSes do packaging that I couldn't refute.

Bad existing practice will always defeat good forward-looking design.

ngrilly commented 7 years ago

Bad existing practice will always defeat good forward-looking design.

@robpike For the record, regarding the use case of "bundled" applications, where a program locates its resources relative to its executable file, what alternative/better design would you suggest?

4ad commented 7 years ago

I disagree with this proposal, it's promoting bad practices. If someone wants this "feature", he can get it from a 3rd party package, or he can write his own. This doesn't need to be in package os.

creker commented 7 years ago

@4ad, what's bad for some OSes is the default for others. For example, Linux apps usually store configuration files in well known fixed locations. For me, that's bad practice. But that just an opinion and I will still use the default method for the specific OS.

Windows or macOS, by default, bundle files into the same directory as the executable. On macOS it's actually called bundle and all files are accessed relative to it. You don't know (on iOS bundle path is actually random) and not suppose to know full path to them. It's considered bad practice to store application files outside it's bundle and AppStore applications are not even allowed to do that. That's just one of the reasons. There're more in the discussion above.

And that's why this needs to be in Go by default. Windows and macOS are first class ports and Go should support default and good practice solutions for them, not force how things done in other OSes.

4ad commented 7 years ago

You are right, programs running on specific platforms would best use whatever is the default on that platform, however bad it might be. This does not mean support for the necessary features need to be in the standard library.

On Windows, services are DLLs which answer a common protocol. The Go standard library makes zero effort supporting this configuration. On macOS, services are configured through launchd. The Go standard library makes zero effort to add special support for launchd. Whoever needs Windows services and launchd daemons needs to get all the features required from somewhere else. I could go on forever with things that are very common on some platforms and Go takes no special steps to support.

There is some argument to be made that Go standard library or runtime should support OS-dependent features or configurations where technical limitations prevent a 3rd party providing these features. For example, we should probably support FreeBSD's capsicum because it requires runtime and os/exec coordination and a 3rd party can't implement it by itself. Similarly, we support Linux's unshare(2) mechanism, because 3rd party programs can't use unshare(2) safely otherwise.

This is not the case here. This feature can be implemented in a 3rd party package without any issues.

The argument about macOS .app bundles does not hold water. Even if we ignore the fact that in practice nobody really writes .app bundles in Go on macOS (maybe they do on iOS, I have ivy on my phone), we can't ignore the fact that the APIs for macOS and iOS already provide an interface for the user to get the path to the bundle!

jimmyfrasche commented 7 years ago

@4ad this isn't really about getting the path to the current executable.

It's about getting the location of the current executable's resources. This location is platform specific, if the platform defines such a location at all. On some platforms it happens to be the same as the path to the current executable. Cf. https://wiki.libsdl.org/SDL_GetBasePath

Another reason one would think they need to get the path to the current executable would be to find a location to save data between runs of the executable, which, again, depends on the platform and on some platforms could be the same as the path of the current executable. Cf. https://wiki.libsdl.org/SDL_GetPrefPath

I'm not sure either of those require any special support by the stdlib, but they could definitely use a library, somewhere, to normalize all the differences between platforms.

4ad commented 7 years ago

@4ad this isn't really about getting the path to the current executable.

Yes it is:

os: introduce Executable to return the path of the current executable

If you want some other kind of higher level API, file a new issue or a new proposal. Lets keep this thread focused.

In fact, if you don't think this proposal is about getting path to the current executable, but want something higher level, this is evidence against implementing this particular feature.

kardianos commented 7 years ago

On Windows, services are DLLs which answer a common protocol. The Go standard library makes zero effort supporting this configuration.

Windows services are standard executables (you can load a DLL into another service). Alex made an asm stub to facilitate callbacks which has been added to the /x/sys/windows sub-repo. You're correct it isn't in the std lib.

On macOS, services are configured through launchd. The Go standard library makes zero effort to add special support for launchd.

Launchd needs XML and signals. You are right it doesn't add special support just for it.

@4ad you seem to be making a value judgment on what APIs people should use as they are actively using them and how they should structure their apps. Let me make a different argument. Assume certain application models and environments require a call like this. Does this call categorically go in package "os"? In my experience, finding this call is difficult for people to find in 3rd party packages when they need it. They first look in "os" then they might search on google (godoc isn't as useful due to difficulty of finding correct terms), from google they find a stackoverflow article which then points them to the package.

You are correct this could go into a 3rd party package. There are benefits to having this in the std lib for osx. Categorically I think it does belong in "os", when people need the function it would nice for it to be there. Maintenance will be low, the need is real and present.

jimmyfrasche commented 7 years ago

In fact, if you don't think this proposal is about getting path to the current executable, but want something higher level, this is evidence against implementing this particular feature.

@4ad That was actually the point I was trying to make. People asking for the specific feature proposed in this issue are XY'ing because they're coming from a platform where X=Y.

It was meant as an argument against this proposal. I wasn't being very clear and I apologize for the confusion.

If someone wants to make a counter-proposal for a higher level API, like the SDL's, don't wait on me. I'm ambivalent about that. I just didn't want a solution to the wrong problem to sneak in the stdlib.

minux commented 7 years ago

I'd like to know more about why this is bad practice.

If I put software on shared NFS mount, but I couldn't control where each Unix machine mounts the volume, I think the only way is to locate resources relative to the executable.

ELF and MachO both have dedicated support for locating needed shared library relative the path where the binary is found.

The reason this has to be in the standard library is that the only way to not require cgo for such functionality is to hook into the runtime library.

golang / go

os: introduce Executable to return the path of the current executable #12773