Agent MSI support - Githubissues

amitkanfer commented 1 year ago

[x] Support basic functionality e2e (copy files and execute agent install from the target folder)
[x] Support uninstall
[x] Check return code from agent install and fail execution if code is different than 0
[x] Support custom installation folder (EDIT: not sure this is needed. See comment here)
[ ] Implement the buildkite logic to trigger an MSI build whenever there's a new Agent version (similar to the logic we have with beats)
[x] Cleanup all temp files after agent install is executed successfully
[ ] check installation with endpoint
[x] check upgrade flow

To test, run from a command prompt with administrator permissions:

elastic-agent.msi INSTALLARGS="--url=<URL> --enrollment-token=<TOKEN>"

Another option, with logs collection:

msiexec -i elastic-agent.msi INSTALLARGS="--url=<URL> --enrollment-token=<TOKEN>" -L*V "log.txt"

Few notes:

If INSTALLARGS are not provided, the MSI will copy the files to a temp folder and finish.
If INSTALLARGS are provided, the MSI will copy the files to a temp folder and then run the elastic-agent install command with the provided args. If the install flow is successful, the temp folder is deleted.
If INSTALLARGS are provided but the elastic-agent install command fails - the top folder is NOT deleted to allow further troubleshooting.
MSI uninstall will call elatic-agent uninstall on a "best effort" basis and will always finish successfully (if not, a 3rd party tool will be needed to remove the windows service)
upgrades are not supported, and in fact prevented as implemented here. Upgrades shall happen via fleet.
if the elastic-agent install command fails (for any reason), the MSI will rollback all changes.

nimarezainia commented 1 year ago

@amitkanfer is it possible to have the MSI set the --base-path option for the agent install command? (as @cmacknz mentions here)?

Setting the path was a big requirement from enterprises and I can't see any reason why it wouldn't also be a requirement for the same set of users who would want to utilize the MSI (aside from Defend users of course). it would be great not to have diverging product behaviors if possible. (of course I am saying this not knowing what the amount of effort involved exactly is)

amitkanfer commented 1 year ago

i'll look into that @nimarezainia . My concern is more around the uninstall flow. Not sure how the new base path will be available to the "add/remove features" and the other native Windows applications. Probably a solved problem, but i just need to understand them.

amitkanfer commented 1 year ago

I also need to check and implement upgrade flows. editing the description

amitkanfer commented 11 months ago

@cmacknz

When this gets merged, at what point is it included in the stack release process?

There will be a follow up PR to make the necessary changes in the pipelines. @dliappis is will be reviewing the work. We should be able to first merge this PR, make sure there's no impact on existing (beats) MSIs, and then trigger the agent MSI flow.

At the moment, i'm focused on testing the install / upgrade / uninstall flows. Once they work 100% i'll cleanup the PR and go through the review cycle

amitkanfer commented 11 months ago

Will try following https://stackoverflow.com/a/17790056 to show error messages to the end user. Note that this is valid only when running the installer w/o MDMs

mbudge commented 10 months ago

Will the MSI support initial communication to Fleet Server through a proxy?

Set the proxy for retrieving agent policies from Fleet If the initial agent communication with Fleet (i.e control plane) needs to traverse the proxy server, then the agent needs to be configured to do so using the –proxy-url command line flag which is applied during the agent installation. Once connectivity to Fleet is established, proxy server details can be managed through the UI.

https://www.elastic.co/guide/en/fleet/current/fleet-agent-proxy-managed.html

mbudge commented 10 months ago

It looks like we have to set up on-prem Fleet servers in each network region, as it's not possible to configure multiple proxies to connect to the Default Fleet Server.

This means we need to set up Fleet servers in production mode. When we do this we need to deploy a cert when we install Elastic-Agent, so agent can connect to the on-prem Fleet server. This might mean we need to deploy a cert with the MSI.

Production – Provide your own certificates. This option will require agents to specify a cert key when enrolling with Fleet

https://www.elastic.co/guide/en/fleet/current/add-fleet-server-mixed.html

amitkanfer commented 10 months ago

@mbudge - any params you'll pass to the MSI using the INSTALLARGS command line argument will get passed to the elastic-agent install command, so i believe both of your questions are answered (proxy and certificate).

nimarezainia commented 10 months ago

@mbudge putting aside the MSI requirement you have, you should be able to create multiple proxies that are on-prem and have all forward to the same Fleet Server, if I understand your situation correctly. This fleet server maybe your default.

On the Fleet settings page:

first create your proxies (bottom of the page), these would be the addresses for the physical proxies you have in each network. Agent's in that network NEED to have a path to the proxy.

Second, for each network that has a proxy create a Fleet Server, and configure the proxy for that network. The fleet server host address may be the same (indeed for your example they all will be the default Fleet Server but different proxies):

in your case you would probably need to create multiples of these one for each network. Certs for the proxy can be configured also when the proxy definition is created. One thing to note is that we currently don't support mTLS for the agent --> Fleet Server connection which is something we are planning on working on soon.

mbudge commented 10 months ago

--delay-enroll

Yes consider this. You might get customers who install elastic-agent on a Virtual Desktop Infrastructure (VDI) hosts, and sometimes those hosts might not have internet access until a user logs in. This is when elastic-agent is running on a pool of VDI hosts, and the VDI hosts are deleted/rebuilt when the user logs off. This might happen then the customer isn't using a dedicated proxy for elastic-agent/fleet connections. This might be because the proxy settings are downloaded with the windows user profile (via group policy) when the user logs into the VDI host.

cmacknz commented 10 months ago

Something seems odd with how the elastic-agent command symlink is getting setup. I downloaded the elastic agent .msi from https://buildkite.com/elastic/elastic-stack-installers/builds/3433 and installed it standalone by double clicking on the .msi file on a Windows VM as the "default" installation path.

The agent installs, but I can't interact with it via the elastic-agent command.

C:\Windows\system32>elastic-agent status
Error: unknown flag: --path.data
Usage:
  elastic-agent status [flags]

Flags:
  -h, --help            help for status
...
unknown flag: --path.data

It seems like the --path.data argument is being unconditionally passed in. This seems to be because the MSI is installing the agent twice, once in C:\Program Files\Elastic\Agent and once in C:\Program Files\Elastic\Beats with the Beats directory taking precedence in the shell:

C:\Program Files\Elastic>where elastic-agent
C:\Program Files\Elastic\Beats\8.13.0\elastic-agent.cmd

The contents of elastic-agent.cmd are:

@echo off
set args=%*

if "%args%" == "" (
    set args=--help
)

set beat_bin=%~dp0%~n0
set beat_data=%ProgramData%\Elastic\Beats\%~n0

"%beat_bin%\%~n0.exe" ^
    --path.home "%beat_bin%" ^
    --path.config "%beat_data%" ^
    --path.data "%beat_data%\data" ^
    --path.logs "%beat_data%\logs" ^
    --E logging.files.redirect_stderr=true ^
    %args%

If I directly invoke C:\Program Files\Elastic\Agent>elastic-agent.exe then it works as expected:

C:\Program Files\Elastic\Agent>elastic-agent.exe status
┌─ fleet
│  └─ status: (STOPPED) Not enrolled into Fleet
└─ elastic-agent
   └─ status: (HEALTHY) Running

C:\Program Files\Elastic\Agent>

cmacknz commented 10 months ago

Double clicking on the .msi again and following the Remove path didn't seem to do anything. The agent remains installed for me.

cmacknz commented 10 months ago

If I run the MSI with an invalid argument the installer fails but doesn't tell me why. Is there anything we can do to change this?

I ran elastic-agent.msi INSTALLARGS="--not-a-real-arg" and the installer UI eventually just told me there was an error after rolling back the installation.

strawgate commented 10 months ago

If I run the MSI with an invalid argument the installer fails but doesn't tell me why. Is there anything we can do to change this?

I ran elastic-agent.msi INSTALLARGS="--not-a-real-arg" and the installer UI eventually just told me there was an error after rolling back the installation.

I think this might just be kind of how it is with an MSI installer -- they aren't really meant to provide feedback via UI

cmacknz commented 10 months ago

I think this might just be kind of how it is with an MSI installer -- they aren't really meant to provide feedback via UI

Is there a standard way to get error information after failures for MSIs? Right now errors are intended to go to the console. Should we start writing a log file, writing to the windows event log, something else?

This will eventually lead to support cases if we don't have some way to troubleshoot.

amitkanfer commented 10 months ago

Good catch @cmacknz - if i run the MSI installer with 'msiexec' and logging turned on - i do see this log line:

MSI (s) (A0:70) [22:02:07:165]: Executing op: UpdateEnvironmentStrings(Name=PATH,Value=C:\Program Files\Elastic\Beats\8.11.4,Delimiter=;,Action=1073741825,)
WriteEnvironmentStrings: Name: PATH, Value: C:\Program Files\Elastic\Beats\8.11.4, Action 1073741825

Something that needs to be taken care of

amitkanfer commented 10 months ago

Is there a standard way to get error information after failures for MSIs? Right now errors are intended to go to the console. Should we start writing a log file, writing to the windows event log, something else?

This will eventually lead to support cases if we don't have some way to troubleshoot.

yes, look at the description. There's an option to use msiexec with logging. The installers redirects stderr to the that specified log file.

mbudge commented 7 months ago

When will this be GA with supporting documentation? Says it's beta.

nimarezainia commented 7 months ago

@mbudge we are looking at 8.14 for GA and docs will be ready then.

kilfoyle commented 7 months ago

@amitkanfer , @nimarezainia I've opened the issue above for the docs.

Amit, I'd really appreciate if you, or anyone, can provide the install steps and any other details users will need. Also, are there any UI changes planned? Thanks!

elastic / elastic-stack-installers

Agent MSI support #212