OpenSearch / OpenSearch Dashboards - macOS distribution (X64)

bbarani commented 3 years ago

Is your feature request related to a problem? Please describe. No

Describe the solution you'd like Provide signed macOS (X64) artifact for OpenSearch / OpenSearch Dashboards

Describe alternatives you've considered Generating the artifact using the source code

Additional context Currently there are macOS (X64) artifacts for OpenSearch / OpenSearch Dashboards

skkosuri-amzn commented 2 years ago

Please prioritize this to speed up development on mac.

dblock commented 2 years ago

With https://github.com/opensearch-project/opensearch-build/pull/751 you can produce a darwin bundle for OpenSearch. This creates bundle/opensearch-1.1.0-darwin-x64.tar.gz.

./build.sh manifests/1.1.0/opensearch-1.1.0.manifest.yml
./assemble.sh artifacts/manifest.yml
cd artifacts
./opensearch-tar-install.sh

$ curl -u admin:admin --insecure https://localhost:9200
{
  "name" : "...",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "04husTTqQi6cd2bPSRvuqw",
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.1.0",
    "build_type" : "tar",
    "build_hash" : "15e9f137622d878b79103df8f82d78d782b686a1",
    "build_date" : "2021-10-15T13:14:16.973135Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

All plugins except PA seem to load, opened https://github.com/opensearch-project/performance-analyzer/issues/73

dbbaughe commented 2 years ago

@bbarani @saratvemulapalli @peternied @dblock

Is this scheduled for 1.3 release?

This comment seems to imply it, but the roadmap for infra seemed to not have it on there or scheduled for 1.3 release.

Can we get some clarification on when this is going to be completed?

saratvemulapalli commented 2 years ago

My comment is based of the roadmap: https://github.com/orgs/opensearch-project/projects/1#column-14092180. Assuming it's up to date.

dbbaughe commented 2 years ago

@saratvemulapalli Thanks Sarat, looks like it is scheduled for 1.3 unless @bbarani mentions otherwise.

dblock commented 2 years ago

Someone contributed a formulae to Homebrew that builds and runs OpenSearch-min and OpenSearch Dashboards from source.

https://github.com/Homebrew/homebrew-core/blob/master/Formula/opensearch.rb

See https://github.com/opensearch-project/opensearch-build/issues/1314#issuecomment-993789585

bbarani commented 2 years ago

We are currently working on creating Linux based distributions (RPM and Deb) at this point in time. Please feel free to provide your inputs on this issue #741 to prioritize other non Linux distributions for us to pickup post that effort. As always, we welcome any contribution towards the development of additional distributions.

saratvemulapalli commented 2 years ago

@bbarani with the launch of RPM/DEB, are we looking to add support for MacOS? Coming from: https://github.com/opensearch-project/OpenSearch/issues/1676

bbarani commented 2 years ago

@saratvemulapalli We are currently working on supporting Windows distribution and you can track the status of other distributions here

CC: @setiah

jmazanec15 commented 2 years ago

@bbarani is it possible to release an ARCHIVE testDistribution for Mac? Related to https://github.com/opensearch-project/OpenSearch/issues/1676.

setiah commented 2 years ago

@bbarani is it possible to release an ARCHIVE testDistribution for Mac? Related to opensearch-project/OpenSearch#1676.

I like this. Unblocking developers would also help individual plugins move faster on this meta issue. I see this is happening already - https://github.com/opensearch-project/OpenSearch/issues/1676#issuecomment-1161611756. Thanks!

msfroh commented 1 year ago

I have put together a basic script that will:

Fork a repo and clone the fork,
Find any GitHub workflows that run on windows-latest but not macos-latest, and add macos-latest to the matrix.os
Commit + push that change back to the fork. (The commit message has a pointer to this issue.)
Cut a PR against the source repo. (We'll be able to see from the PR checks if the macos-latest build succeeds or fails.)

I'm going to run it against all repositories under opensearch-project.

dblock commented 1 year ago

@msfroh btw, you might be interested in https://github.com/opensearch-project/project-meta and things like https://github.com/opensearch-project/opensearch-clients/blob/main/META.md#open-a-pull-request-in-each-repo

msfroh commented 1 year ago

I did an audit of all repos (see https://gist.github.com/msfroh/5071d8be1b5408526b6fce0440fc27c0), cutting some automated PRs to see what would happen if we just add macos-latest everywhere that has ubuntu-latest and windows-latest.

The repos where that "just worked" were common-utils, k-NN, and oui.

Here are the repos that require more complicated actions, with recommendations:

OpenSearch-Dashboard: Has Linux and Windows builds in https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/.github/workflows/build_and_test_workflow.yml. Needs a dedicated MacOS build or needs to refactor to come up with a cross-platform build.
dashboards-reporting: Has builds for Linux, WIndows, and Mac, but the release workflow (https://github.com/opensearch-project/dashboards-reporting/blob/main/.github/workflows/dashboards-reports-release-workflow.yml) only uploads for Linux and Windows.
ml-commons: The CI workflow (https://github.com/opensearch-project/ml-commons/blob/main/.github/workflows/CI-workflow.yml) has jobs for Linux and Windows, but no job for MacOS.
opensearch-dashboards-functional-tests: This is an interesting one, since it looks like we only run e2e tests on Linux. There's a template for Windows, but it's currently unused. I think the first step is to make sure we run e2e tests on Windows (since people do run OpenSearch on Windows in production), and then maybe consider adding e2e testing for MacOS (which I don't think anyone would use in production).
neural-search and job-scheduler: Both fail because their builds try to download OpenSearch artifacts for MacOS, which are not available, like https://artifacts.opensearch.org/releases/core/opensearch/2.6.0/opensearch-min-2.6.0-darwin-x64.tar.gz. I think we need the OpenSearch core repo to build and publish those artifacts.
security: Build fails on MacOS because it uses https://github.com/peternied/action-sleep/blob/main/action.yml, which adds s suffix on the sleep argument, which is not Posix-compliant.
notifications and dashboard-notifications: Builds already seem to be broken, unrelated to my attempts to add MacOS builds. They need to be fixed. Example failures: notifications, dashboards-notifications.
dashboards-maps and anomaly-detection-dashboards-plugin: Both fail on their integration test step where they try to launch OpenSearch wrapped in the timeout command, which doesn't seem to be available on MacOS. If we have GNU coreutils installed on the Mac build machines, it should be available as gtimeout. Alternatively, we might be able to use a while loop that counts the total sleep time and aborts when it runs out.
security-analytics-dashboards-plugin: Fails when caching Cypress on MacOS. I haven't been able to figure out why this one fails when other repos have succeeded.

Across the board, I've struggled to get Cypress integration tests passing for OpenSearch Dashboards plugins on MacOS. I've tried several things in https://github.com/opensearch-project/dashboards-search-relevance/pull/143, like increasing start-up time for OSD, but nothing seems to work. In general, the effort for OSD plugins seems to be higher than for OS core plugins.

msfroh commented 1 year ago

Oh, also I'm working from the assumption that any MacOS distribution at this point would need ARM support to be useful. My 1.5 year-old Mac laptop uses an M1 processor and I gather that more and more folks will be in the same boat as time goes on.

peternied commented 1 year ago

Recently the security team dropped support for MacOS builds because builds were slower and network infrastructure was flaky. We should investigate the reliability of this platform as this will have an impact on maintenance of repositories https://github.com/opensearch-project/security/issues/2467

dblock commented 1 year ago

Looks like we're doing some work in this space with https://github.com/opensearch-project/opensearch-build/pull/3670. @gaiksaya is there a plan to make a macos distribution? Are we going to do the items listed above too?

gaiksaya commented 1 year ago

Not that I know of currently. @bbarani Might be able to get more information. The above PR was to support signing clients and driver and other standalone artifacts that have macos as one of the distribution. 😄

bbarani commented 1 year ago

@gaiksaya @dblock we haven't prioritized it yet. We are planning to look in to it in Q3 / Q4 of 2023.

lpsinger commented 7 months ago

Naive question: OpenSearch is written in Java, isn't it? Is the bundled JRE/JDK the only platform-dependent part of the distributions?

msfroh commented 7 months ago

Naive question: OpenSearch is written in Java, isn't it? Is the bundled JRE/JDK the only platform-dependent part of the distributions?

Unfortunately not, we have some native code in the ml-commons plugin, for example (though I think it can build for MacOS even on ARM). Also, for OpenSearch Dashboards, there's a lot of NodeJS stuff going on.

Looking back at the experimentation I did in February, OpenSearch Dashboards was where most of the headaches came in. (OpenSearch core builds and runs just fine on my M1 laptop, for example.)

opensearch-project / opensearch-build

OpenSearch / OpenSearch Dashboards - macOS distribution (X64) #38