search

poshbotio / PoshBot

Powershell-based bot framework
MIT License
536 stars 108 forks source link

PoshBot as a Service Restart error if plugins are installed. #156

Closed mgeorgebrown89 closed 4 years ago

mgeorgebrown89 commented 5 years ago

I think I've isolated the issue I've been having with my PoshBot running as a windows service not always surviving service restarts. If I install plugins, then restart the service, the service starts and stops every five seconds infinitely unless I kill it. I've tested this with the other psd1 files, logs, groups, roles, and permissions. Those can be created and will survive the restart, but plugins seem to make it crash. I added groups, roles, permissions, and plugins and saw that they were added to the psd1 files like expected, and then tried removing the files before restarting. Each one worked except for the plugins. So it seems like if there are plugins installed it will cause some sort of collision. I'm not sure if this is how I've set things up or what.

Expected Behavior

PoshBot as a service should be able to survive a service restart with plugins installed.

Current Behavior

It seems like the only thing that causes the start/stop loop is if a plugin other than the builtin has been installed.

Steps to Reproduce (for bugs)

  1. Setup PoshBot as a windows service.
  2. Install a plugin via Slack.
  3. Restart PoshBot service.
  4. View the service from the services app, view it's properties, and it's service status should flash back and forth between stopping and starting.

Your Environment

I'm assuming this is probably related to how I've set PoshBot up. I have the service call a script that creates a new configuration each time, pulling the credentials from an outside source. 

Add-Type -AssemblyName System.Security
#Install-Module -Name PoshBot -Repository PSGallery -Force
Import-Module -Name PoshBot -Force
Import-Module 'C:\Repositories\PoshBot-Secrets\Get-PoshBotCredentials.psm1' -Force

# Define Bot Configuration
$Token = Get-SlackToken
$BotName = 'Vectoria' # The name of the bot we created
$BotAdmin = 'michael.g.brown' # My account name in Slack
$PoshbotPath = 'C:\Repositories\PoshBot'
$PoshbotConfig = Join-Path $PoshbotPath config.psd1
$PoshbotPlugins = Join-Path $PoshbotPath plugins
$PoshbotLogs = Join-Path $PoshbotPath logs

$BotParams = @{
    ApprovalExpireMinutes = 30
    ApprovalCommandConfigurations = @(
        @{
            Expression = 'neverfail:*'
            Groups = @('admin', 'Cloud Ops Engineers')
            PeerApproval = $true
        }
        @{
            Expression = 'amazon:New-AWS_R53Entry:*'
            Groups = @('admin', 'Cloud Ops Engineers')
            PeerApproval = $true
        }
        @{
            Expression = 'tests:Test-Approvals:*'
            Groups = @('admin', 'Cloud Ops Engineers')
            PeerApproval = $true
        }
    )
    Name                     = $BotName
    BotAdmins                = $BotAdmin
    CommandPrefix            = '!'
    LogLevel                 = 'Verbose'
    BackendConfiguration     = @{
        Name  = 'SlackBackend'
        Token = $Token
    }
    AlternateCommandPrefixes = @('vec', 'VEC', 'Vec', 'VEc')
    ConfigurationDirectory   = $PoshbotPath
    LogDirectory             = $PoshbotLogs
    PluginDirectory          = $PoshbotPlugins
    PluginConfiguration      = @{
        vTM       = @{
            Credential = Get-PluginCredentials -Plugin "vTM" -Name "Credential"
        }
        neverfail = @{
            Credential = Get-PluginCredentials -Plugin "neverfail" -Name "Credential"
        }
        pingdom   = @{
            Credential = Get-PluginCredentials -Plugin "pingdom" -Name "Credential"
            AppKey     = Get-PluginCredentials -Plugin "pingdom" -Name "AppKey"
        }
        amazon    = @{
            SecretKey = Get-PluginCredentials -Plugin "amazon" -name "SecretKey"
            AccessKey = Get-PluginCredentials -Plugin "amazon" -Name "AccessKey"
        }
    }
}

# Set up folders for logging and plugins, save the config
$null = mkdir $PoshbotPath, $PoshbotPlugins, $PoshbotLogs -Force
$pbc = New-PoshBotConfiguration @BotParams 

Save-PoshBotConfiguration -InputObject $pbc -Path $PoshbotConfig -Force

$pbc = Get-PoshBotConfiguration -Path $PoshbotConfig 
$backend = New-PoshBotSlackBackend -Configuration $pbc.BackendConfiguration
$bot = New-PoshBotInstance -Configuration $pbc -Backend $backend

Send-SlackMessage -Token $pbc.BackendConfiguration.Token -Channel 'bot_test' -Text "New bot instance configured. Starting now..." -AsUser

while ($True) {
    try {
        $err = $null
        $bot | Start-PoshBot -Verbose -ErrorVariable err
        if ($err) {
            throw $err
        }
    }
    catch {
        $_ | Format-List -Force | Out-String | Out-File (Join-Path $pbc.LogDirectory Service.Error)
    }
}

Send-SlackMessage -Token $pbc.BackendConfiguration.Token -Channel 'bot_test' -Text "PoshBot Service stopped...goodbye." -AsUser
devblackops commented 5 years ago

@mgeorgebrown89 Couple questions:

  1. What account are you running the service under? Local system or a service account?
  2. In the context of that account, is PSGallery (or whatever PS repository you've configured) set to Trusted?
mgeorgebrown89 commented 5 years ago

@devblackops

  1. Its a user account called 'poshbot' that has the access I need it to.
  2. Yes.
mgeorgebrown89 commented 5 years ago

I wonder if this is related to update-plugin not working for me either. From an earlier post, I know you mentioned using credentials to run AD stuff, but that I could try running it as a user. I've been doing the latter, but do you think I should try running it as the local system and pass it the correct credentials each time I use something in the AD module? @devblackops

Or do I need to add another value to the PluginRepository value of the config file? Right now it's just PSGallery, but I thought the plugin directory took care of my own modules.

Also, I noticed this in the verbose output: WARNING: {"DataTime":"2019-03-03 22:24:30Z","Class":"PluginManager","Method":"GetPluginConfig","Severity":"Warning","LogLevel":"Debug","Message":"Plugin [Builtin] not defined in plugins.psd1","Data":{}}

mgeorgebrown89 commented 5 years ago

Another thing I noticed: It seems to only crash if I install my plugins. If I install a plugin like PoshBot.Giphy and restart the service, there's no problem. I tried seeing if it was my psd1 and made it match the PoshBot.Giphy.psd1 and the same thing happens. The only other thing I can think of is maybe this is happening because I'm not using the default plugin directory or I haven't published it to a psrepo.

devblackops commented 5 years ago

So are you manually saving your custom modules to the plugin directory defined in $PoshbotPlugins?

PoshBot will prepend that path to $end:PSModulePath so when you tell it to install a plugin, it will look in that directory first. If present, it will load it into PoshBot, if not present, it will attempt to install the module from the repository defined in the PluginRepository property of the bot configuration (normally PSGallery unless changed). It does not save modules to this directory though, they are always saved using Install-Module -Scope CurrentUser so <homedir>/Documents/PowerShell/Modules using a user account and using the system account this will end up being C:/Windows/system32/config/systemprofile/Documents/PowerShell/Modules.

devblackops commented 5 years ago

@mgeorgebrown89 Are you by chance using the PoshBot.8ball plugin? I just noticed a problem with that one. Notice that when installing this plugin, the command name is 8ball. In plugins.psd1 we store what plugins are registered along with any permissions attached to the commands. CommandPermissions is a hashtable will every command in the plugin and it's permissions. A hashtable cannot start with a number unless enclosed in quotes.

'PoshBot.8ball' = @{
    '1.0.0' = @{
      Name = 'PoshBot.8ball'
      AdhocPermissions = @()
      ManifestPath = 'C:\Users\poshbot\Documents\PowerShell\Modules\PoshBot.8ball\1.0.0\PoshBot.8ball.psd1'
      CommandPermissions = @{
        8ball = @()
      }
      Enabled = $True
      Version = '1.0.0'
    }
  }

This surfaces a minor bug in PoshBot which I think we can fix by ensuring we store this using quotes.

mgeorgebrown89 commented 5 years ago

@devblackops , nope. I haven't used PoshBot.8Ball.

If by manually, you mean installing them via PowerShell, then no. I install them from the Slack channel. $PoshbotPlugins = 'C:\Repositories\PoshBot\plugins' and I haven't changed the PluginRepository.

So, I don't see any of my modules in 'C:\Users\poshbot\Documents\WindowsPowerShell\Modules' just the ones that are Poshbot.*

Is this because my start-poshbot.ps1 script specifies a different plugin directory?

mgeorgebrown89 commented 5 years ago

@devblackops So do you recommend the service that runs poshbot logon as the local system, and pass in the credentials necessary? It seems like that's how it was meant to be, but I'm not sure.
Edit: and I've just confirmed that when I !Install-Module mymodule it doesn't save it to the directories mentioned above, either as a user or as the local system. Only modules from the PSGallery seem to get saved that way. I haven't published modules to either the PSGallery or my own before, but is this possible the reason this is happening?

devblackops commented 5 years ago

@mgeorgebrown89 I'd recommend running the service as a non-privileged service account and have any required credentials for the plugins passed in as parameters. Running PoshBot itself as a privileged user could present security concerns as ALL commands are executed under that context and you may introduce a plugin that does more than you think. Best to keep the blast area as small a possible.

As far as installing the plugins into PoshBot, where they get saved depends on if they are already present in $env:PSModulePath. If the module is already on the system, PoshBot doesn't move it anywhere, it will just load it from its current path. For custom developed modules that aren't published anywhere, you can put them in the directory defined by PluginDirectory. When PoshBot starts, it prepends that path to $env:PSModulePath and will look there first for modules on the system.

If the module is not found in $env:PSModulePath, it will install the module from whatever PS repository is defined and save it using Install-Module -Scope CurrentUser. If using the system account this will be C:/Windows/system32/config/systemprofile/Documents/PowerShell/Modules and for any other account it will be C:/Users/<username>/Documents/PowerShell/Modules.

You can verify where the modules it uses are by looking at the Plugins.psd1 file in the configuration directory. That file contains the list of plugins, their version, and their specific location on the system. PoshBot will load that file at startup (if present) to re-hydrate the system after restarts.

mgeorgebrown89 commented 5 years ago

Hmmm. All of that seems to be true in this case. The signs seem to point to it being a service error. I tried spinning up a new machine and running it from there but I just got different service errors. Edit: @devblackops, it seems to be the case. If I just run my start-poshbot.ps1 script in the console, install plugins from slack, end the powershell session, and run the script again, all works as expected. So there's something wrong either with my system, nssm, or what account I'm using or having the service log on as. It's definitely the plugin.psd1 file that's the problem, at least as far as when running it as a service, and it doesn't seem to matter who or what it logs on as, whether the local system or a user.

mgeorgebrown89 commented 5 years ago

@devblackops , well, what I had hoped would be an easy fix did not seem to help. Currently running into the exact same issue as before, but now I'm on a Windows 2016 server. It still comes down to that plugins.psd1 file, which is so weird. I have verified that this file does contain the modules I installed and that their location on the system is correct. The only other thing I can think to try is to create the service by means other than nssm, but it seems like I'm the only person with this issue... I wonder if this is related to the update plugin command also not working for me. Edit: just tried running on my own system, and when I started the service but had my start-script open it gave me this WARNING: Did not find config file C:\Users\mgb11\AppData\Local\Temp\mgb11-MICHAEL-DESKTOP-PSSlack.xml, attempting to create. Curious. Edit2: Ah. what is going on? I even published my module to the PSGallery, removed it completely from the bots config files and directories, installed it from there, and it still caused my bot to go into that start/stop loop.

Edit3: more context. this time from event viewer, which may be out of the scope. 

Faulting application name: powershell.exe, version: 10.0.14409.1005, time stamp: 0x584a185c
Faulting module name: mscorlib.ni.dll, version: 4.7.3324.0, time stamp: 0x5c09b1b7
Exception code: 0xc00000fd
Fault offset: 0x00000000004adf96
Faulting process id: 0x1e2c
Faulting application start time: 0x01d4d641ff90c447
Faulting application path: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
Faulting module path: C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\a4c029035a52b21a293c249a889b6925\mscorlib.ni.dll
Report Id: 400083a9-4235-11e9-810b-d6ed4d449357
Faulting package full name: 
Faulting package-relative application ID:
mgeorgebrown89 commented 5 years ago

!!!!!!!!! @devblackops. I may have figured it out. I started with a brand new bot on a brand new system, and began the set up, slowly and meticulously, to try and see exactly when this strange issue happened. Early on, there were no problems. When I copied the example modules and restarted the bot, they were still there, which was great but also frustrating. Then I noticed something. I had a lot of stuff in the same folder as all the config psd1 files. As soon as I changed their location to an isolated folder of their own, things seem to be working as expected. I won't close the case yet, as I want to verify, but things look promising so far.

devblackops commented 5 years ago

Good to hear @mgeorgebrown89. What were the extra files? Given a brand new setup, PoshBot will create a few files at startup if they are not present. This includes:

mgeorgebrown89 commented 5 years ago

@devblackops, The folder structure I had was like this:

The top four were folders. After thinking about this a bit more, this could also have to do with user permissions actually and some of our environment's GPOs affecting what files the user had access to. The new setup is in the user directory, and the service/task is running as that user currently. I will continue to investigate.

EDIT: Now I'm wondering if it's because I explicitly Import-Module PoshBot in my modules. I ran into the same problem again this morning, and for some reason decided not to import it and see, and it restarted and loaded the modules no problem. I have marked the required modules part of the psd1 file with 'PoshBot' but it still causes a crash. What's going on here I wonder...

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.