java packaging tests

[andyb-elastic]

This issue is following the work from https://github.com/elastic/elasticsearch/issues/26741 and tracks what tests from the old bats project need to be ported to the new java project, as well as what new test coverage needs to be added

See here for info about how the java tests are structured (this may be a little out of date as some things have changed)

See here for instructions about how to run these tests, particularly this section about iterating more quickly which will save you a lot of time developing these tests

porting bats tests

Bats migration has been spun off into a separate issue: https://github.com/elastic/elasticsearch/issues/46005

executable coverage

We currently don't have coverage for all executables that ship with the distributions. Note that the ones marked checked may only have coverage in a bats test that has not yet been ported

oss executables

• [x] elasticsearch - Used in most tests
• [x] elasticsearch-env - We check for its existence and it's used by bin/elasticsearch so I think coverage here is fine
• [x] elasticsearch-plugin - used in 25_tar_plugins.bats and 50_modules_and_plugins.bats
• [x] elasticsearch-keystore - We test for keystore usage in the basic tests which have been ported
• [ ] elasticsearch-translog - We check for its existence but there's no usage
• [ ] elasticsearch-cli - We don't check for its existence nor is there any usage

windows-specific executables

• [ ] elasticsearch-service.bat - we check for its existence but no usage
• [ ] elasticsearch-service-mgr.exe - we check for its existence but no usage
• [ ] elasticsearch-service-x64.exe - we check for its existence but no usage

default executables

• [x] elasticsearch-certgen - used in 40_tar_certgen.bats and 45_package_certgen.bats
• [ ] elasticsearch-certutil - we check for existence but no usage
• [x] elasticsearch-coneval - used in x-pack's 10_basic.bats
• [ ] elasticsearch-migrate - we check for existence but no usage
• [ ] elasticsearch-saml-metadata - we check for existence but no usage (@jkakavas)
• [x] elasticsearch-setup-passwords - used in 30_tar_setup_password.bats and 35_package_setup_password.bats
• [x] elasticsearch-sql-cli - used in 20_tar_bootstrap_password.bats and 25_package_bootstrap_password.bats
• [ ] elasticsearch-syskeygen - we check for existence but no usage
• [ ] elasticsearch-users - we check for existence but not run (@jkakavas)
• [x] x-pack-env - we check for existence and it's used by other x-pack executables so I think we have coverage here
• [x] x-pack-security-env - we check for existence and it's used by other x-pack executables so I think we have coverage here
• [x] x-pack-watcher-env - we check for existence and it's used by other x-pack executables so I think we have coverage here

Other major things to do

• [ ] Collecting logging so we can debug these tests when they fail. Most of them are transient and not reproducible, so what we currently do is collect the logs from the installed distribution and dump them to the console (#28608). We can port this by adding a junit test rule to the packaging test classes that does the same when a test method fails, which will have the added benefit of applying to every test (in the bash tests we would need to manually add a function call for it)
• [x] Test installing elasticsearch as a service on Windows - using the elasticsearch-service.bat executable

• [ ] On Windows we should run the server as a role user because it works around some issues with how we're using windows powershell remoting (and because it's what we recommend). The remote shell runs as administrator, so to avoid running e.g. the server as administrator we have to "impersonate" the vagrant user to run commands not as-administrator. There's some more details about this here and here.

  An example of one way to handle local windows users in powershell is in this feature branch. The relevant cmdlets are New-LocalUser, Get-LocalUser, and Remove-LocalUser. The server 2012r2 box may need an upgraded version of powershell and WMF to use these.

  When running other powershell commands as a role user, (e.g. elasticsearch-keystore, not the server) it's probably best to use Invoke-Command (roughly equivalent to sudo -u elasticsearch foo). For example

  # here the elasticsearch user's password is 'elasticsearch' - you'll have to change some local security policies to actually be able to use that
  $password = ConvertTo-SecureString 'elasticsearch' -AsPlainText -Force
  $credential = New-Object System.Management.Automation.PSCredential 'elasticsearch', $password
  Invoke-Command -ComputerName localhost -Credential $credential -ScriptBlock { <your command goes here }

[elasticmachine]

Pinging @elastic/es-core-infra

[andyb-elastic]

@rjernst I think we can check off "Test installing elasticsearch as a service on Windows" right, does https://github.com/elastic/elasticsearch/pull/33729 cover all of that

[rjernst]

Yes, #33729 covered basic windows service tests.

[jaymode]

@elastic/es-security there are security executables listed above that do not have test coverage in our packaging tests, so I'd like to ask for your help in adding coverage for these tests.

[mark-vieira]

Existing BATS tests have all been migrated.