search

opensearch-project / opensearch-build

🧰 OpenSearch / OpenSearch-Dashboards Build Systems
Apache License 2.0
137 stars 272 forks source link

[Bug]: All integration tests for RPM, Debian and Windows are failing #4370

Closed gaiksaya closed 7 months ago

gaiksaya commented 8 months ago

Describe the bug

Note: This is an observation for OpenSearch only. OSD might be a different story Recently so many changes went in related to security plugin as well as in general. We enabled integration testing to be triggered via distribution pipeline for all distributions. (Earlier it was just running for tarball)

To reproduce

All windows failed: https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/7136/pipeline All debian failed: https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/7135/pipeline All RPM failed: https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/7134/pipeline

Expected behavior

All tests should run. The infrastructure and test set up should be fixed.

Screenshots

If applicable, add screenshots to help explain your problem.

Host / Environment

No response

Additional context

No response

Relevant log output

No response

peterzhuamazon commented 8 months ago

Will check soon, most likely the deb/rpm is due to sudo permission not added to the ci-runner images.

PRs:

peterzhuamazon commented 8 months ago

k-NN: Test with @junqiu-lei on tar for concurrent search, resolved https://github.com/opensearch-project/k-NN/pull/1419.

peterzhuamazon commented 8 months ago

Resolve rpm:

2024-01-26 23:15:19 INFO     rpm installation requires sudo, script will exit if current user does not have sudo access
LANG=C.UTF-8
HOSTNAME=1c85fa7fa882
TERM=xterm
MAIL=/var/mail/root
PATH=/sbin:/bin:/usr/sbin:/usr/bin
LOGNAME=root
USER=root
HOME=/root
SHELL=/bin/bash
SUDO_COMMAND=/bin/env OPENSEARCH_INITIAL_ADMIN_PASSWORD=myStrongPassword123!
SUDO_USER=ci-runner
SUDO_UID=1000
SUDO_GID=1000
XDG_SESSION_ID=c1
XDG_RUNTIME_DIR=/run/user/0
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus
OPENSEARCH_INITIAL_ADMIN_PASSWORD=myStrongPassword123!
No match for argument: opensearch
No packages marked for removal.
Dependencies resolved.
Nothing to do.
Complete!
Rocky Linux 8 - AppStream                                                                                                                                                                                                                                                                   7.1 MB/s |  10 MB     00:01
Rocky Linux 8 - BaseOS                                                                                                                                                                                                                                                                      4.6 MB/s | 5.1 MB     00:01
Rocky Linux 8 - Extras                                                                                                                                                                                                                                                                       29 kB/s |  14 kB     00:00
Extra Packages for Enterprise Linux 8 - x86_64                                                                                                                                                                                                                                              8.6 MB/s |  16 MB     00:01
packages for the GitHub CLI                                                                                                                                                                                                                                                                  15 kB/s | 2.6 kB     00:00
Dependencies resolved.
============================================================================================================================================================================================================================================================================================================================
 Package                                                                       Architecture                                                              Version                                                                      Repository                                                                       Size
============================================================================================================================================================================================================================================================================================================================
Installing:
 opensearch                                                                    x86_64                                                                    2.12.0-1                                                                     @commandline                                                                    660 M

Transaction Summary
============================================================================================================================================================================================================================================================================================================================
Install  1 Package

Total size: 660 M
Installed size: 931 M
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
peterzhuamazon commented 8 months ago 

Removed:
  opensearch-2.12.0-1.x86_64

Complete!
2024-01-26 23:33:17 INFO     Recording component test results for geospatial at /home/ci-runner/opensearch-build/test-results/57995af2c3444c6d86f67ce3fbad526e/integ-test/geospatial/with-security
2024-01-26 23:33:17 INFO     Walking tree from /home/ci-runner/opensearch-build/test-results/57995af2c3444c6d86f67ce3fbad526e/integ-test/geospatial/with-security
2024-01-26 23:33:17 INFO     Recording component test results for geospatial at /home/ci-runner/opensearch-build/test-results/57995af2c3444c6d86f67ce3fbad526e/integ-test/geospatial/without-security
2024-01-26 23:33:17 INFO     Walking tree from /home/ci-runner/opensearch-build/test-results/57995af2c3444c6d86f67ce3fbad526e/integ-test/geospatial/without-security
2024-01-26 23:33:17 INFO     Skipping ml-commons
2024-01-26 23:33:17 INFO     Skipping neural-search
2024-01-26 23:33:17 INFO     Skipping notifications-core
2024-01-26 23:33:17 INFO     Skipping notifications
2024-01-26 23:33:17 INFO     Skipping opensearch-observability
2024-01-26 23:33:17 INFO     Skipping anomaly-detection
2024-01-26 23:33:17 INFO     Skipping alerting
2024-01-26 23:33:17 INFO     Skipping security-analytics
2024-01-26 23:33:17 INFO     Skipping index-management
2024-01-26 23:33:17 INFO     Skipping flow-framework
2024-01-26 23:33:17 INFO     Removing /tmp/tmpun19ylg0
2024-01-26 23:33:17 INFO     | geospatial           | with-security        | PASS  |
2024-01-26 23:33:17 INFO     | geospatial           | without-security     | PASS  |
gaiksaya commented 8 months ago

Looks like debian is fixed: https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/7376/pipeline RPM too: https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/7375/pipeline Windows not yet: https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/7371/pipeline

Thanks @peterzhuamazon

peterzhuamazon commented 8 months ago

Today I learn that bash shell inside the windows container using git bash is actually giving

$ echo $SHELL
/sbin/nologin

causing failures due to ftrepo specifically checking if it is a bash shell……

peterzhuamazon commented 8 months ago

More changes needed due to this:

More issues:

peterzhuamazon commented 8 months ago

IntegTest should be fully fixed now on Windows.

gaiksaya commented 8 months ago

Windows looks good too now: https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/7681/pipeline Thanks @peterzhuamazon 🙌

peterzhuamazon commented 8 months ago

See issues: https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test-opensearch-dashboards/detail/integ-test-opensearch-dashboards/5530/pipeline/ x64, linux, osd-2.12.0-7254, os-2.12.0-9373, rpm

https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test-opensearch-dashboards/detail/integ-test-opensearch-dashboards/5527/pipeline/ arm64, linux, osd-2.12.0-7254, os-2.12.0-9373, tar

{"type":"log","@timestamp":"2024-02-09T18:57:36Z","tags":["fatal","root"],"pid":521,"message":"Error: Cannot find module '../../types'\nRequire stack:\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/plugins/securityAnalyticsDashboards/server/utils/constants.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/plugins/securityAnalyticsDashboards/server/clusters/createSecurityAnalyticsCluster.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/plugins/securityAnalyticsDashboards/server/plugin.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/plugins/securityAnalyticsDashboards/server/index.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/plugins/plugin.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/plugins/discovery/plugins_discovery.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/plugins/discovery/index.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/plugins/plugins_service.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/plugins/index.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/index.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/cli/serve/serve.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/cli/cli.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/cli/dist.js\n at Function.Module._resolveFilename (node:internal/modules/cjs/loader:1075:15)\n at Module.Hook._require.Module.require (/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/node_modules/require-in-the-middle/index.js:61:29)\n at require (node:internal/modules/cjs/helpers:110:18)\n at Object.<anonymous> (/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/plugins/securityAnalyticsDashboards/server/utils/constants.ts:6:1)\n at Module._compile (node:internal/modules/cjs/loader:1254:14)\n at Object.Module._extensions..js (node:internal/modules/cjs/loader:1308:10)\n at Module.load (node:internal/modules/cjs/loader:1117:32)\n at Function.Module._load (node:internal/modules/cjs/loader:958:12)\n at Module.require (node:internal/modules/cjs/loader:1141:19)\n at Module.Hook._require.Module.require (/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/node_modules/require-in-the-middle/index.js:80:39)\n at Module.Hook._require.Module.require (/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/node_modules/require-in-the-middle/index.js:80:39)\n at Module.Hook._require.Module.require (/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/node_modules/require-in-the-middle/index.js:80:39)\n at Module.Hook._require.Module.require (/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/node_modules/require-in-the-middle/index.js:80:39)\n at Module.Hook._require.Module.require (/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/node_modules/require-in-the-middle/index.js:80:39)\n at require (node:internal/modules/cjs/helpers:110:18)\n at Object.<anonymous> (/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/plugins/securityAnalyticsDashboards/server/clusters/createSecurityAnalyticsCluster.ts:6:1) {\n code: 'MODULE_NOT_FOUND',\n requireStack: [\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/plugins/securityAnalyticsDashboards/server/utils/constants.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/plugins/securityAnalyticsDashboards/server/clusters/createSecurityAnalyticsCluster.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/plugins/securityAnalyticsDashboards/server/plugin.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/plugins/securityAnalyticsDashboards/server/index.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/plugins/plugin.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/plugins/discovery/plugins_discovery.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/plugins/discovery/index.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/plugins/plugins_service.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/plugins/index.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/index.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/cli/serve/serve.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/cli/cli.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/cli/dist.js'\n ]\n}"}

peterzhuamazon commented 8 months ago

Should be solved by this PR:

peterzhuamazon commented 8 months ago

New issue:

peterzhuamazon commented 7 months ago

There seems to be a bug where Windows will run better if:

# OpenSearch:
http.host: 0.0.0.0
network.host: 127.0.0.1

# OpenSearch-Dashboards:
server.host: 0.0.0.0

If not this then it will nearly 100% fail the manual run of Windows ftrepo directly.

Very weird.

peterzhuamazon commented 7 months ago

Resolved now.