jpillora / overseer

Monitorable, gracefully restarting, self-upgrading binaries in Go (golang)
MIT License
2.32k stars 207 forks source link

Issue when path is symlink #20

Open fmpwizard opened 7 years ago

fmpwizard commented 7 years ago

At work, our deployment goes something like this:

copy go binary to

/srv/tools//binary-here

then create/update an symlink so that we have: /srv/tools/current => /srv/tools/<last-build-id>

This lets us quickly revert to a previous version if things go wrong and we didn't detect any issues during testing.

Because overseer uses github.com/kardianos/osext to detect the "real" path to the binary that is running and needs to be restarted, sending a restart signal doesn't restart the new binary, but it restarts the "old" binary, the one with the specific build-id, instead of the one pointed by current

Possible solutions:

  1. I could update our deployment to make a "copy" of the binary to .../current/, and duplicate it at /srv/tools/, but doesn't seem too clean
  2. I see that you have this env variable envBinPath = "OVERSEER_BIN_PATH" , how about checking if this is set, and if is is, then use that instead of calling :
func (mp *master) checkBinary() error {
    //get path to binary and confirm its writable
    binPath, err := osext.Executable()
    if err != nil {
        return fmt.Errorf("failed to find binary path (%s)", err)
    }

so, it would be something like

mp.binPath = os.GetEnv(envBinPath)

I would be happy to send a PR if you find this idea useful, or maybe there is already a work around this.

jpillora commented 7 years ago

Since you're deploying to /srv/tools/<last-build-id>, does that mean you're just using overseer for graceful restarts? Or do you want to use it for upgrades (i.e. deploys) too?

jpillora commented 7 years ago

In the past, I've just used S3 with versioning enabled to store binaries, and an S3 rollback will result in an overseer "upgrade" (actually a downgrade). #19 will add a File Fetcher, though it also just checks a single path so it still wouldn't match your workflow. Maybe the easiest thing to do is make a custom Fetcher the interface is quite simple.

fmpwizard commented 7 years ago

yes, I'm only using overseer as a graceful restart that plays well with upstart/systemd/etc

I'll look into the Fetcher idea, I assume it means, I keep deploying to /.../<build-id, but instead of using a symlink that is created by our deployment scripts (ansible in our case), I let overseer make the actual copy of the file, interesting. I'll try it and if I run into any issues I'll post back, thanks!

jpillora commented 7 years ago

Okay no worries. Note: you can also omit the fetcher to just use graceful restarts:

    overseer.Run(overseer.Config{
        Program: prog,
        Address: ":3000",
    })

Then in your app, you could poll the symlink for changes, or you could expose an HTTP endpoint, and then call overseer.Restart().

fmpwizard commented 7 years ago

I remembered that the way our servers are setup, the actual go binary doesn't have write access to the path where it is deployed, which means I cannot go with the Fetcher idea (unless I update our folder permissions, which I would like to avoid).

Because our go app isn't the only app we run, we would like to keep ansible as the one place where deployments happen from, and keep our apps away from the deployment logic.

I ended up making these changes to proc_master.go

func (mp *master) checkBinary() error {
    // If you run your app from a synlink, you can
    // pass this env variable to set where overseer should
    // look for the binary to start a new instance
    binPath := os.Getenv(envBinPath)
    var err error
    if os.Getenv(envBinPath) == "" {
        //get path to binary and confirm its writable
        binPath, err = osext.Executable()
        if err != nil {
            return fmt.Errorf("failed to find binary path (%s)", err)
        }
    }
        mp.binPath = binPath

and now upstart passes the env variable to our go app telling it to always run from /srv/tools/web/current/<binary name>

I can keep this in our own fork if it doesn't seen aligned with your original idea for overseer.

Thanks for a great tool!