stackstorm and chatops behind a proxy

ytjohn commented 7 years ago

I'm going to open this as an issue first to get some notes in place. Maybe after some thinking and discussion, we can shape this into PR against the proxy readme. I'm new to stackstorm, and new to npm/nodejs modules. So I'm trying to take knowledge from working in linux and python and apply it to this arena. That doesn't always work. For example, one of the first thing I found is that hubot uses an npm http library that doesn't honor HTTP_PROXY type environment variables.

uppercase vs lowercase

My first comment is that most of the docs reference HTTP_PROXY and HTTPS_PROXY capital, which is ok. But I think we should make mention that the lowercase variants of these often take precedence in linux. Another variable that is useful, but does not seem to be implemented very well through npm modules is NO_PROXY or no_proxy variable, which is standard in linux to exclude sites from hitting the proxy. Great for excluding localhost. See my example in ansible-st2.

Upper vs lowercase:

getting hubot to use proxies in general

For hubot, they provide instructions for getting hubot to work behind a proxy. To make that work, simply create a file /opt/stackstorm/chatops/scripts/proxy.coffee with the following, and that will cause hubot and most scripts to work through a proxy. It does not have provisions like NO_PROXY to exclude localhost. There is no need to edit external-scripts.json, as hubot will load any file in this scripts directory.

proxy = require 'proxy-agent'
module.exports = (robot) ->
  robot.globalHttpOptions.httpAgent  = proxy('http://my-proxy-server.internal', false)
  robot.globalHttpOptions.httpsAgent = proxy('http://my-proxy-server.internal', true)

In addition to npm install proxy-agent, I had to npm install proxy, which probably should be a dependency, but wasn't.

getting slack to work through proxy.

You do not need to create the hubot scripts/proxy.coffee script to get slack to work.

The slack adapter (and probably all of the adapters) load before the scripts are loaded. I learned that hubot-slack added a fix to read the https_proxy environment variable (lowercase only) in slack-client@3.6.1. My freshly installed version had slack-client@1.4.1.

npm upgrade hubot-slack upgraded the depency modules and brought in slack/client@3.8.1 under hubot-slack. Once this is done, you can add https_proxy=http://proxy:3128/ to your st2chatops.env file and restart st2chatops service.

ytjohn commented 7 years ago

A follow up note. I specifically removed the scripts/proxy.coffee on my side because that prevents hubot-stackstorm from hitting localhost and authenticating. However, if you setup your st2chatops.env to use your server's fqdn or ip address, as long as the proxy can reach back to it, it should work.

I was looking at this issue under hubot-proxy-loader and tried to implement proxy-agent-patch to accept NO_PROXY, but ran into issues. Now that I have slack connectivity working and a 2% better understanding of the hubot ecosystem, I may try this again and update here.

ytjohn commented 7 years ago

for ~st2/.gitconfig:

[http]
    proxy = http://proxy:port/
[https]
    proxy = http://proxy:port/
[url "https://"]
    insteadOf = git://

LindsayHill commented 7 years ago

We've seen some initial reports that there may be some issues with Slack getting the execution results when using newer versions of hubot-slack. Has not been fully investigated yet. Further data would be appreciated if anyone has it.

ytjohn commented 7 years ago

@LindsayHill

Here's what I've observed (and this is ran behind a proxy). I can run an action like chaptops.post_message, or a pack executes chatops.post_result and it appears to operate correct, but the message never makes it to slack. Hubot and slack integration work just fine, I can send messages to hubot, it can respond with help and available commands. But stackstorm -> hubot -> slack does not seem to work.

At debug logging, the only message hubot leaves in the logs is receipt of the chatops message, as shown below. I see "chatops message received", but I don't see anything about "sending to".

Apr  1 03:56:01 stackstorm hubot[13595]: [Sat Apr 01 2017 03:56:01 GMT+0000 (UTC)] DEBUG Chatops message received: {"trace_context": {"id_": "58c22adc02ebd52bc7e1c347", "trace_tag": null}, "payload": {"whisper": false, "message": "  Successful deployment of *aws* pack!\n", "user": "john", "channel": "G4FLVFYJX", "extra": {}}}
Apr  1 03:57:14 stackstorm hubot[13595]: [Sat Apr 01 2017 03:57:14 GMT+0000 (UTC)] DEBUG Chatops message received: {"trace_context": {"id_": "58c22adc02ebd52bc7e1c347", "trace_tag": null}, "payload": {"whisper": false, "message": "this is a test", "user": null, "channel": "metallica-stackstorm", "extra": {}}}

If I send a help command in the channel. It works.

Apr  1 03:53:00 stackstorm hubot[13595]: [Sat Apr 01 2017 03:53:00 GMT+0000 (UTC)] DEBUG Received message: '@metalstorm help' in channel: metallica-stackstorm, from: john
Apr  1 03:53:00 stackstorm hubot[13595]: [Sat Apr 01 2017 03:53:00 GMT+0000 (UTC)] DEBUG Message '@metalstorm help' matched regex //^\s*[@]?(?:!metalstorm[:,]?|metalstorm[:,]?)\s*(?:([\s\S]+?)$)/i/; listener.options = { id: null }
Apr  1 03:53:00 stackstorm hubot[13595]: [Sat Apr 01 2017 03:53:00 GMT+0000 (UTC)] DEBUG Executing listener callback for Message '@metalstorm help'
Apr  1 03:53:00 stackstorm hubot[13595]: [Sat Apr 01 2017 03:53:00 GMT+0000 (UTC)] DEBUG Message '@metalstorm help' matched regex //^\s*[@]?(?:!metalstorm[:,]?|metalstorm[:,]?)\s*(?:help(?:\s+(.*))?$)/i/; listener.options = { id: null }
Apr  1 03:53:00 stackstorm hubot[13595]: [Sat Apr 01 2017 03:53:00 GMT+0000 (UTC)] DEBUG Executing listener callback for Message '@metalstorm help'
Apr  1 03:53:00 stackstorm hubot[13595]: [Sat Apr 01 2017 03:53:00 GMT+0000 (UTC)] DEBUG Sending to G4FLVFYJX: !metalstorm help - Displays all of the help commands that Hubot knows about.
Apr  1 03:53:00 stackstorm hubot[13595]: !metalstorm help <query> - Displays all help commands that match <query>.
Apr  1 03:53:00 stackstorm hubot[13595]: !metalstorm pack get <pack> - Get information about installed StackStorm pack.
Apr  1 03:53:00 stackstorm hubot[13595]: !metalstorm pack install <gitUrl>[,<gitUrl>] - Install/upgrade StackStorm packs.
Apr  1 03:53:00 stackstorm hubot[13595]: !metalstorm pack install <pack>[,<pack>] - Install/upgrade StackStorm packs.
Apr  1 03:53:00 stackstorm hubot[13595]: !metalstorm pack search <query> - Search for packs in StackStorm Exchange and other directories.
Apr  1 03:53:00 stackstorm hubot[13595]: !metalstorm pack show <pack> - Show information about the pack from StackStorm Exchange.

root@stackstorm:/opt/stackstorm/chatops# npm ls | egrep '(hubot|slack|st2)'
├── hubot@2.18.0
├── hubot-diagnostics@0.0.1
├── hubot-flowdock@0.7.6
├── hubot-help@0.1.3
├── hubot-hipchat@2.12.0-5 (git+https://github.com/StackStorm/hubot-hipchat.git#c36a50d067b33e5e0e10c12190a63039815066ec)
├── hubot-irc@0.2.8 (git+https://github.com/nandub/hubot-irc.git#51d1f4b418fc039a456b7891c7f9bf836699cfdc)
├─┬ hubot-proxy-loader@0.0.1
├── hubot-redis-brain@0.0.3
├── hubot-scripts@2.16.2
├─┬ hubot-slack@4.3.3
│ ├─┬ @slack/client@3.8.1
├─┬ hubot-spark@1.7.0
├─┬ hubot-stackstorm@0.4.5
├─┬ hubot-xmpp@0.1.18 (git+https://github.com/markstory/hubot-xmpp.git#94c3438e42778c53e38f6909939c514da50a0dca)
├── hubot-yammer@0.2.1 (git+https://github.com/athieriot/hubot-yammer.git#fde43a79a977203370ac7f2595702ad60406cffb)
├─┬ slack-client@1.4.1
├── st2client@0.4.4

A larger log is available here: https://gist.github.com/ytjohn/d06c786d8755b1a9c8dd015c39f841c1

ytjohn commented 7 years ago

adding this in because it never did get reported here:

mting: slack - proxy require hubot-slack > 4.0 but hubot-stackstorm just support hubot-slack < 4.0 hubot slack using new slack-client that hubot-stackstorm doesnot supporting

mting: i done a quick check days before, slack-client < 4.0 , channel name return the name , after > 4.0 it return channel ID that maybe cause st2 cant send right command execution result to right channel thats why we could get help info, but cant get the execution result

arm4b commented 6 years ago

Update: current st2chatops relies on hubot-slack >= 4.3.4. See: https://github.com/StackStorm/st2chatops/blob/master/package.json#L22

StackStorm / st2docs