Launcher passes CWD and Environment between Steps

[stjohnjohnson]

Currently we don't pass environment variables or cwd between steps because we execute all commands separately in sub-shells, this causes fun issues like:

$ eval `ssh-agent -s`
$ ssh-add -L
Error connecting to agent: No such file or directory

This happens because SSH_AUTH_SOCK is set in the first step, but not passed into the second one.

---

The proposed solution to this is to follow something similar to Wercker:

• Start /bin/sh with control of STDOUT, STDERR, STDIN
• Setup all environment variables via export FOO=bar (how to handle quoting here?)
• Create unique ID for each step
• Write step as shell script to a temporary file (with #!/bin/sh -e)
• Run source $STEP_SCRIPT ; echo "UNIQUE_ID $?"
• Wait for UNIQUE_ID to come back from STDOUT along with the exit code
• Act appropriately when the exit code comes back

[d2lam]

This makes sense to me.

(how to handle quoting here?) What do you mean by this? Shouldn't it work similarly to currently: https://github.com/screwdriver-cd/launcher/blob/master/launch.go#L237-L246

So for each step we need to write to a temporary file, would that add too much overhead if someone has a lot of steps?

[stjohnjohnson]

@d2lam previously we could just spawn the command with environment set. In this case we're running a shell and executing export in the shell. This will allow users to do more "creative" environment setting, but at the risk of shell exploiting.

Maybe you're right, we should not support "creative" environment setting via the environment feature. Instead they need to use a step that calls export

[jer]

I agree on not making it easy to do weird environment things. Not for security and shell exploits or anything, it just makes for a confusing interface.

This seems like a reasonable plan. We just need to make sure the random id doesn't end up in logs as well.

[FenrirUnbound]

Task map

• [ ] All-in-one shipment
  • [x] /bin/sh with the ability to manipulate STDIN
    • Instead of "run echo", it's "start in a shell & run a command when we tell you"
    • [x] Complete incomplete prototype
    • [ ] ~io.Writer~
    • [x] Terminate shell
    • [x] Notify API on step start
  • [x] Execute command from file
    • [x] Save command to file
      • [x] File starts with #!/bin/sh -e
    • [x] Modify execution to source the file
  • [x] Unique Identifier
    • [x] Generate one
    • [x] Suffix step execution with unique identifier & exit code
  • [ ] ~Update Emitter~
    • [x] Terminate step based on sentinel value
    • [x] Notify API on step finish
    • [x] Start the next step
    • [x] Class/function to notify step completion
  • [ ] Error Handling
    • ?
• [x] Stronger understanding of the current workflow & pending modifications
  • [x] Learn more Go

[minzcmu]

Summary

Instead of Starting /bin/sh with control of STDOUT, STDERR, STDIN, our current implementation for that part is as follows:

1. Start /bin/sh in a pseudo-terminal using https://github.com/kr/pty
2. Write and execute commands in the pty:
   • Run setup commands (e.g. set -e)
   • Source step script for each step
3. Pipe the output from pty to emitter directly

Progress

• [x] Integration test with in-a-box is working
• [x] One unit test(TestEnv) is failing due to ## in the env output
  • Root cause: interactive mode by default is on

[d2lam]

Fixed the unit test.

What we tried:

• c := exec.Command("sh") then turned off interactive by set +o interactive. Didn't work
• c := exec.Command("sh", "+o interactive"). Still didn't work

What worked

• Add PS1="" to list of default environment variables.

[d2lam]

Using export

Using another script

[d2lam]

Done!

Blog post: http://blog.screwdriver.cd/post/156911122972/launcher-passes-cwd-and-environment-variables Documentation: https://docs.screwdriver.cd/user-guide/configuration/environment