search

opensearch-project / opensearch-build

🧰 OpenSearch / OpenSearch-Dashboards Build Systems
Apache License 2.0
135 stars 271 forks source link

Support `ppc64le` OpenSearch distribution for `POWER` architecture. #3122

Open SonalMahambrey1 opened 1 year ago

SonalMahambrey1 commented 1 year ago

Opensearch snapshot for X64 is available at https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-linux-x64-latest.tar.gz As power architecture is supported now, any update when the similar artifacts will be published for ppc64le as well ?

dblock commented 1 year ago

I am not sure whether the build team intends to publish these, or whether the team at IBM (cc: @seth-priya) wants to maintain this port, but let's move this into opensearch-build.

seth-priya commented 1 year ago

@dblock it would be great if the build team can publish these and we will continue to watch for Power specific issues and fix them, on our side.

dblock commented 1 year ago

@bbarani @peterzhuamazon what would it take to add those to automation? Maybe @seth-priya can contribute?

zelinh commented 1 year ago

[Triages] First step is to have the IBM server in order to move forward. Adding @elfisher and @setiah for discussion.

seth-priya commented 1 year ago

There are a few places to get access to ppc64le VMs, one of them is https://osuosl.org/services/powerdev/request_hosting/

Sunidhi-Gaonkar1 commented 1 year ago

Hi Team, any thoughts on above?

jordarlu commented 1 year ago

Adding @bbarani to comment on future supporting PowerPC 64 Little Endian ..thanks!

seth-priya commented 1 year ago

thanks @jordarlu , we can definitely contribute the changes that are needed for building the artifacts via automation and we can also discuss about the infrastructure that's needed for the same (if needed on a call?). As of now - we already have the Power changes up-streamed for the main opensearch repository and have been able to locally build the images and test them and also have a working build and image for opensearch-dashboards. Currently, we are building/validating all of the opensearch plugins separately on Power to make sure all of them work smoothly (though from what we have seen so far, they should). I am also adding our overall technical lead/architect @gerrith3 here.

gerrith3 commented 1 year ago

Yeah - we would love to help support getting a CI environment set up for ppc64le. I'd love to know what the options are from your side, what we would need to integrate with, etc. We have some challenges with broadly available CI platforms but we do have several solutions, alternatives, and work arounds to integrate with most/many environments.

Sapana-Khemkar commented 9 months ago

@dblock, @bbarani, @jordarlu and others, I am from @seth-priya's team. We can help adding/maintaining ppc64le support to CI environment. Let us know how can we move further on this.

bbarani commented 9 months ago

@Sapana-Khemkar Thanks for your support. How do you want to proceed with next steps? Would a kick-off call help to get things started soon? It would be really easy to collaborate via OpenSearch Slack workspace.

Sapana-Khemkar commented 9 months ago

@Sapana-Khemkar Thanks for your support. How do you want to proceed with next steps? Would a kick-off call help to get things started soon? It would be really easy to collaborate via OpenSearch Slack workspace.

@bbarani joined slack and sent you message on devops channel.

prudhvigodithi commented 9 months ago

Had an offline sync up with @Sapana-Khemkar, since OpenSearch supports ppc64le, having a Power9 server and running the build and assemble workflow should give us an artifact for ppc64le. While we request for the server and integrate with the CI/CD @Sapana-Khemkar can you please post the results of the steps (manually executing the workflows on your server inside the opensearchstaging/ci-runner:ci-runner-centos7-opensearch-build-v3 docker) we discussed over the call? Thank you

gerrith3 commented 9 months ago

Hey for the OpenSearch folks, if you do request a free VM at OSL per the link earlier in this thread, reference my name as your IBM support contact and we will make sure it gets expedited for you. We would love to see this integrated and we encounter more users of OpenSearch on Power all the time, so this would be a great step for all of us!!

gerrith3 commented 9 months ago

And, apologies if this should be in a thread of its own, but does anyone know if OpenSearch is keeping up with Elasticsearch's ability to be used as a vector database? I've been seeing that question come up several times and at the moment, I have no clue. ;) thanks, and apologies for hijacking another thread, although this is really yet another Power use case for us. ;)

Sunidhi-Gaonkar1 commented 9 months ago

Hi @prudhvigodithi , I am from @seth-priya and @Sapana-Khemkar's team. I was able to manually execute the build and assemble workflows on Power server and create an artifact for ppc64le. The docker image opensearchstaging/ci-runner:ci-runner-centos7-opensearch-build-v3 is not available for power, hence was not able to execute workflows inside the container. There were a few changes required to be made. I have attached logs of output for execution of build and assemble workflows, also attached the changes that were made to build the artifact.

OpenSearch_assemble.txt OpenSearch_build.txt OpenSearch_Patch.txt

bbarani commented 9 months ago

And, apologies if this should be in a thread of its own, but does anyone know if OpenSearch is keeping up with Elasticsearch's ability to be used as a vector database? I've been seeing that question come up several times and at the moment, I have no clue. ;) thanks, and apologies for hijacking another thread, although this is really yet another Power use case for us. ;)

OpenSearch indeed has made great strides in Vector DB space. It uses K-NN functionality as documented in this blog to implement vector db functionality. The issues related to Vector DB functionalities are tracked here.

Sapana-Khemkar commented 8 months ago

Had an offline sync up with @Sapana-Khemkar, since OpenSearch supports ppc64le, having a Power9 server and running the build and assemble workflow should give us an artifact for ppc64le. While we request for the server and integrate with the CI/CD @Sapana-Khemkar can you please post the results of the steps (manually executing the workflows on your server inside the opensearchstaging/ci-runner:ci-runner-centos7-opensearch-build-v3 docker) we discussed over the call? Thank you

@prudhvigodithi any update from your side? Have you raised request for ppc64le vm?

prudhvigodithi commented 8 months ago

Hey @Sapana-Khemkar I have the form submitted NAME: OpenSearch Project, support.osuosl.org #33270. Thank you @bbarani @peterzhuamazon

Sapana-Khemkar commented 8 months ago

Hey @Sapana-Khemkar I have the form submitted NAME: OpenSearch Project, support.osuosl.org #33270. Thank you @bbarani @peterzhuamazon

@prudhvigodithi your request 33270 is approved. Please let me know if you face any issue.

prudhvigodithi commented 8 months ago

Thanks, @Sapana-Khemkar what is the next step here? I just got an email as Approved for the request I created. Let me know we can discuss the next steps in public slack on how to securely connect the ppc64le VM as jenkins agent.

peterzhuamazon commented 8 months ago

Have a discussion with @prudhvigodithi and we need to add a test ci-runner image layer of linux/ppc64le, then try to start it up on the requested server before trying out the built.

If successful, then we will proceed with a manual build, before adding the changes to all the ci-runner images.

I will start on creating the new ci-runner image soon.

Sapana-Khemkar commented 8 months ago

Thanks, @Sapana-Khemkar what is the next step here? I just got an email as Approved for the request I created. Let me know we can discuss the next steps in public slack on how to securely connect the ppc64le VM as jenkins agent.

@prudhvigodithi your request is approved from IBM. Now Lance Albertson from OSU will create VM. But he have few questions. I am pasting it here. Can you please respond to these?

Distribution: CentOS

Will AlmaLinux work for this or do you want CentOS Stream? Also do you want The 8 or 9 release?

Est Size Of User Community: NA

Expected Usage Model: The POWER is used to build and assemble the OpenSearch software for ppc64le architecture.

Flavor: xlarge

Number Of Nodes: 5

Is this quantity a hard requirement is are you able to reduce this to something smaller? I want to make sure we only allocate what's really needed.

cc: @ramereth

peterzhuamazon commented 8 months ago

@Sapana-Khemkar since we are using docker here for the build, as long as there is docker installed for base OS on that machine, we good.

peterzhuamazon commented 8 months ago

There are issues generating the ppc64le layer on docker, mainly due to pip installation here:

 => ERROR [linux/ppc64le 15/22] RUN ln -sfn /usr/local/bin/python3.9 /usr/bin/python3 &&     ln -sfn /usr/local/bin/pip3.9 /usr/bin/pip &&     ln -sfn /usr/local/bin/pip3.9 /usr/local/bin/pip &&     ln -sfn /usr/local/bin/pip3.9 /usr/bin/pip3 &&     pip3 install pip==23.1.2 && pip3 install pipenv==2023.6.12  81.0s
------
 > [linux/ppc64le 15/22] RUN ln -sfn /usr/local/bin/python3.9 /usr/bin/python3 &&     ln -sfn /usr/local/bin/pip3.9 /usr/bin/pip &&     ln -sfn /usr/local/bin/pip3.9 /usr/local/bin/pip &&     ln -sfn /usr/local/bin/pip3.9 /usr/bin/pip3 &&     pip3 install pip==23.1.2 && pip3 install pipenv==2023.6.12 awscli==1.22.12:
#0 6.242 Collecting pip==23.1.2
#0 6.365   Downloading pip-23.1.2-py3-none-any.whl (2.1 MB)
#0 7.345 Installing collected packages: pip
#0 7.345   Attempting uninstall: pip
#0 7.347     Found existing installation: pip 21.2.3
#0 8.631     Uninstalling pip-21.2.3:
#0 9.266       Successfully uninstalled pip-21.2.3
#0 19.95 Successfully installed pip-23.1.2
#0 19.95 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#0 25.83 Collecting pipenv==2023.6.12
#0 26.07   Downloading pipenv-2023.6.12-py3-none-any.whl (2.9 MB)
#0 26.77      ���������������������������������������� 2.9/2.9 MB 4.4 MB/s eta 0:00:00
#0 34.13 Collecting awscli==1.22.12
#0 34.16   Downloading awscli-1.22.12-py3-none-any.whl (3.8 MB)
#0 35.04      ���������������������������������������� 3.8/3.8 MB 4.4 MB/s eta 0:00:00
#0 35.97 Collecting certifi (from pipenv==2023.6.12)
#0 36.00   Downloading certifi-2023.11.17-py3-none-any.whl (162 kB)
#0 36.07      ��������������������������������������� 162.5/162.5 kB 3.6 MB/s eta 0:00:00
#0 39.04 Collecting setuptools>=67.0.0 (from pipenv==2023.6.12)
#0 39.07   Downloading setuptools-69.0.3-py3-none-any.whl (819 kB)
#0 39.29      ��������������������������������������� 819.5/819.5 kB 4.3 MB/s eta 0:00:00
#0 39.50 Collecting virtualenv-clone>=0.2.5 (from pipenv==2023.6.12)
#0 39.53   Downloading virtualenv_clone-0.5.7-py3-none-any.whl (6.6 kB)
#0 40.75 Collecting virtualenv>=20.17.1 (from pipenv==2023.6.12)
#0 40.78   Downloading virtualenv-20.25.0-py3-none-any.whl (3.8 MB)
#0 41.66      ���������������������������������������� 3.8/3.8 MB 4.4 MB/s eta 0:00:00
#0 49.00 Collecting botocore==1.23.12 (from awscli==1.22.12)
#0 49.03   Downloading botocore-1.23.12-py3-none-any.whl (8.2 MB)
#0 50.91      ���������������������������������������� 8.2/8.2 MB 4.4 MB/s eta 0:00:00
#0 51.51 Collecting docutils<0.16,>=0.10 (from awscli==1.22.12)
#0 51.54   Downloading docutils-0.15.2-py3-none-any.whl (547 kB)
#0 51.70      ��������������������������������������� 547.6/547.6 kB 4.2 MB/s eta 0:00:00
#0 51.93 Collecting s3transfer<0.6.0,>=0.5.0 (from awscli==1.22.12)
#0 51.96   Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB)
#0 52.00      ���������������������������������������� 79.5/79.5 kB 3.2 MB/s eta 0:00:00
#0 52.46 Collecting PyYAML<5.5,>=3.10 (from awscli==1.22.12)
#0 52.49   Downloading PyYAML-5.4.1.tar.gz (175 kB)
#0 52.56      ��������������������������������������� 175.1/175.1 kB 3.7 MB/s eta 0:00:00
#0 53.91   Installing build dependencies: started
#0 77.90   Installing build dependencies: finished with status 'done'
#0 77.91   Getting requirements to build wheel: started
#0 80.14   Getting requirements to build wheel: finished with status 'error'
#0 80.22   error: subprocess-exited-with-error
#0 80.22
#0 80.22   � Getting requirements to build wheel did not run successfully.
#0 80.22   � exit code: 1
#0 80.22   ��> [48 lines of output]
#0 80.22       running egg_info
#0 80.22       writing lib3/PyYAML.egg-info/PKG-INFO
#0 80.22       writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
#0 80.22       writing top-level names to lib3/PyYAML.egg-info/top_level.txt
#0 80.22       Traceback (most recent call last):
#0 80.22         File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
#0 80.22           main()
#0 80.22         File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
#0 80.22           json_out['return_val'] = hook(**hook_input['kwargs'])
#0 80.22         File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
#0 80.22           return hook(config_settings)
#0 80.22         File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
#0 80.22           return self._get_build_requires(config_settings, requirements=['wheel'])
#0 80.22         File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
#0 80.22           self.run_setup()
#0 80.22         File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 311, in run_setup
#0 80.22           exec(code, locals())
#0 80.22         File "<string>", line 271, in <module>
#0 80.22         File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 103, in setup
#0 80.22           return distutils.core.setup(**attrs)
#0 80.22         File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
#0 80.22           return run_commands(dist)
#0 80.22         File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
#0 80.22           dist.run_commands()
#0 80.22         File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
#0 80.22           self.run_command(cmd)
#0 80.22         File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 963, in run_command
#0 80.22           super().run_command(command)
#0 80.22         File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
#0 80.22           cmd_obj.run()
#0 80.22         File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 321, in run
#0 80.22           self.find_sources()
#0 80.22         File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 329, in find_sources
#0 80.22           mm.run()
#0 80.22         File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 551, in run
#0 80.22           self.add_defaults()
#0 80.22         File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
#0 80.22           sdist.add_defaults(self)
#0 80.22         File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/command/sdist.py", line 112, in add_defaults
#0 80.22           super().add_defaults()
#0 80.22         File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
#0 80.22           self._add_defaults_ext()
#0 80.22         File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
#0 80.22           self.filelist.extend(build_ext.get_source_files())
#0 80.22         File "<string>", line 201, in get_source_files
#0 80.22         File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
#0 80.22           raise AttributeError(attr)
#0 80.22       AttributeError: cython_sources
#0 80.22       [end of output]
#0 80.22
#0 80.22   note: This error originates from a subprocess, and is likely not a problem with pip.
#0 80.25 error: subprocess-exited-with-error
#0 80.25
#0 80.25 � Getting requirements to build wheel did not run successfully.
#0 80.25 � exit code: 1
#0 80.25 ��> See above for output.
#0 80.25
#0 80.25 note: This error originates from a subprocess, and is likely not a problem with pip.
#0 80.28
#0 80.28 [notice] A new release of pip is available: 23.1.2 -> 23.3.2
#0 80.28 [notice] To update, run: pip install --upgrade pip
janani66 commented 8 months ago

Thanks, @Sapana-Khemkar what is the next step here? I just got an email as Approved for the request I created. Let me know we can discuss the next steps in public slack on how to securely connect the ppc64le VM as jenkins agent.

@prudhvigodithi your request is approved from IBM. Now Lance Albertson from OSU will create VM. But he have few questions. I am pasting it here. Can you please respond to these?

Distribution: CentOS

Will AlmaLinux work for this or do you want CentOS Stream? Also do you want The 8 or 9 release?

Est Size Of User Community: NA Expected Usage Model: The POWER is used to build and assemble the OpenSearch software for ppc64le architecture. Flavor: xlarge Number Of Nodes: 5

Is this quantity a hard requirement is are you able to reduce this to something smaller? I want to make sure we only allocate what's really needed.

cc: @ramereth

@peterzhuamazon / @prudhvigodithi -- so I take it AlmaLinux will work for you. What about the number of Nodes -- is 5 a hard requirement?

Can the responses please be posted to: https://support.osuosl.org/Ticket/Display.html?id=33270. in response to the question from Lance Albertson. Once that is done Lance will assign someone to create a VM for you'll to use.

prudhvigodithi commented 8 months ago

Hey @janani66, the link asks for login details https://support.osuosl.org/Ticket/Display.html?id=33270 so we can continue to discuss on this issue. The requirement can be adjusted later based on how frequently we want to build the ppc64le distribution (scale up and scale down as required), since we are at POC phase, its ok to get a basic VM that works to build and assemble the OpenSearch distribution. For linux x64 distribution build the existing setup uses C54xlarge (Ref: https://github.com/opensearch-project/opensearch-ci/blob/main/lib/compute/agent-nodes.ts#L74-L86) which can handle multiple parallel OpenSearch distribution builds. So please adjust the requirement that can be given for testing and flexible enough to update if required. Thank you Adding @peterzhuamazon

peterzhuamazon commented 8 months ago

Hey @janani66, the link asks for login details https://support.osuosl.org/Ticket/Display.html?id=33270 so we can continue to discuss on this issue. The requirement can be adjusted later based on how frequently we want to build the ppc64le distribution (scale up and scale down as required), since we are at POC phase, its ok to get a basic VM that works to build and assemble the OpenSearch distribution. For linux x64 distribution build the existing setup uses C54xlarge (Ref: https://github.com/opensearch-project/opensearch-ci/blob/main/lib/compute/agent-nodes.ts#L74-L86) which can handle multiple parallel OpenSearch distribution builds. So please adjust the requirement that can be given for testing and flexible enough to update if required. Thank you Adding @peterzhuamazon

As of now either Rocky/Alma/Ubuntu based server is fine, as we are using docker to build.

We can have 5 nodes to start with, each having 16CPU 32GB setup, as each will be loaded with 4 docker containers (4CPU/8GB each). That means a total of 20 executors with containers which is sufficient enough for now.

Tho keep in mind our current setup is utilizing EC2 to dynamically provision runners, which means if there are a lot of builds, then 20 executors may be execeeded later. If there is a way to provision ppc64le server dynamically through EC2 that would be even better.

Thanks.

ramereth commented 8 months ago

@peterzhuamazon should be able to use the OpenStack API to automatically provision what you need, however @janani66 mentioned in the ticket that you wanted us to create the VMs for you instead of getting API access.

@janani66 if you can get email addresses for @peterzhuamazon and whomever else on this issue, I can add them to our ticket so we can discuss this further without going back and forth here.

peterzhuamazon commented 8 months ago

I will try out the 6.0.1 version of PyYAML today to see if that will pass the ci runner creation.

peterzhuamazon commented 8 months ago

@peterzhuamazon should be able to use the OpenStack API to automatically provision what you need, however @janani66 mentioned in the ticket that you wanted us to create the VMs for you instead of getting API access.

@janani66 if you can get email addresses for @peterzhuamazon and whomever else on this issue, I can add them to our ticket so we can discuss this further without going back and forth here.

I dont think I have a login credential for the ticket now, assuming this one? https://support.osuosl.org/Ticket/Display.html?id=33270

I can give you more information once I can login and comment there.

ramereth commented 8 months ago

@peterzhuamazon it's best if we just add you to the ticket directly and move the conversation to email only. We don't typically open up logins directly to the tickets.

peterzhuamazon commented 8 months ago

@peterzhuamazon it's best if we just add you to the ticket directly and move the conversation to email only. We don't typically open up logins directly to the tickets.

Got it. I will reply through email soon after.

peterzhuamazon commented 8 months ago

Able to login to the initial almalinux8 test server.

Server Info

# uname -a
Linux opensearch-1.novalocal 4.18.0-513.9.1.el8_9.ppc64le #1 SMP Sat Dec 2 10:18:07 UTC 2023 ppc64le ppc64le ppc64le GNU/Linux

Alma Linux 8 16CPU 32GB MEM 300GB Storage

Will use RockyLinux 8 docker setups to verify if all the packages can be installed.

peterzhuamazon commented 8 months ago

Issue: gh client has no ppc64le support.

Error:
 Problem: cannot install the best candidate for the job
  - package gh-2.42.0-1.i386 from gh-cli does not have a compatible architecture
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
# yum list gh --showduplicates
Last metadata expiration check: 0:01:43 ago on Thu 11 Jan 2024 09:57:38 PM UTC.
Available Packages
gh.aarch64                                                                                                                                                  2.42.0-1                                                                                                                                                  gh-cli
gh.armv6hl                                                                                                                                                  2.42.0-1                                                                                                                                                  gh-cli
gh.i386                                                                                                                                                     2.42.0-1                                                                                                                                                  gh-cli
gh.x86_64                                                                                                                                                   2.42.0-1                                                                                                                                                  gh-cli

As of https://github.com/cli/cli/releases/tag/v2.42.0 we are not seeing support on ppc64le for gh client.

peterzhuamazon commented 8 months ago

Add a few tweaks before making maven run:

# mvn --version
Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae)
Maven home: /usr/local/apache-maven
Java version: 11.0.15, vendor: Eclipse Adoptium, runtime: /usr/lib/jvm/temurin-11-jdk
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.18.0-513.9.1.el8_9.ppc64le", arch: "ppc64le", family: "unix"
peterzhuamazon commented 8 months ago

My PRs:

IBM PRs:

Sunidhi-Gaonkar1 commented 8 months ago

Issue: gh client has no ppc64le support.

Error:
 Problem: cannot install the best candidate for the job
  - package gh-2.42.0-1.i386 from gh-cli does not have a compatible architecture
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
# yum list gh --showduplicates
Last metadata expiration check: 0:01:43 ago on Thu 11 Jan 2024 09:57:38 PM UTC.
Available Packages
gh.aarch64                                                                                                                                                  2.42.0-1                                                                                                                                                  gh-cli
gh.armv6hl                                                                                                                                                  2.42.0-1                                                                                                                                                  gh-cli
gh.i386                                                                                                                                                     2.42.0-1                                                                                                                                                  gh-cli
gh.x86_64                                                                                                                                                   2.42.0-1                                                                                                                                                  gh-cli

As of https://github.com/cli/cli/releases/tag/v2.42.0 we are not seeing support on ppc64le for gh client.

Hi @peterzhuamazon, We are working on enabling power support for gh-cli, as a workaround gh-cli can be installed using following command : go install github.com/cli/cli/v2/cmd/gh@latest

peterzhuamazon commented 7 months ago

Issue: gh client has no ppc64le support.

Error:
 Problem: cannot install the best candidate for the job
  - package gh-2.42.0-1.i386 from gh-cli does not have a compatible architecture
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
# yum list gh --showduplicates
Last metadata expiration check: 0:01:43 ago on Thu 11 Jan 2024 09:57:38 PM UTC.
Available Packages
gh.aarch64                                                                                                                                                  2.42.0-1                                                                                                                                                  gh-cli
gh.armv6hl                                                                                                                                                  2.42.0-1                                                                                                                                                  gh-cli
gh.i386                                                                                                                                                     2.42.0-1                                                                                                                                                  gh-cli
gh.x86_64                                                                                                                                                   2.42.0-1                                                                                                                                                  gh-cli

As of https://github.com/cli/cli/releases/tag/v2.42.0 we are not seeing support on ppc64le for gh client.

Hi @peterzhuamazon, We are working on enabling power support for gh-cli, as a workaround gh-cli can be installed using following command : go install github.com/cli/cli/v2/cmd/gh@latest

Seems like it needs go version 1.21 or above? I will take a look on that.

Thanks.

peterzhuamazon commented 7 months ago

20240122: Have meeting with IBM team on current status:

  1. ml-commons still have some issues and IBM team is still testing on their end.
  2. IBM team will raise the PR to have gh cli supported officially on gh cli repo, with prebuild binaries.
  3. IBM team is aiming for TAR distribution to start with, and later RPM. Only build/assemble now, not integTest, see 6.
  4. IBM Team needs to make changes on these files to start with:
  5. IBM Team will try to resolve the pytest errors due to PyYAML 6.0.1 upgrades: https://github.com/opensearch-project/opensearch-build/pull/4348 (✅ #4413)
  6. IBM Team will try to figure out some ways of allowing Jenkins to dynamically provision servers for its agents. So we will not have resource exhaustion problem or have to manually setup the runners everytime there is a Jenkins maintenance.
    • Current expectation: At least 10x Servers with 16CPU 32GB setups to start with.
  7. @peterzhuamazon will get this PR reviewed and merged soon, and building new images with linux/ppc64le layer: https://github.com/opensearch-project/opensearch-build/pull/4336

cc: @Sapana-Khemkar @seth-priya @Sunidhi-Gaonkar1 @bbarani @prudhvigodithi

Thanks.

Sapana-Khemkar commented 7 months ago
  1. ml-commons - we are working on this.
  2. Official support from gh-cli

We have raised PR for ppc64le support. But it may take a little time. Meanwhile can you install go v1.21.0 using below steps

wget https://go.dev/dl/go1.21.0.linux-ppc64le.tar.gz
tar -C /usr/local -xzf go1.21.0.linux-ppc64le.tar.gz
export GOROOT=/usr/local/go
export GOPATH=$HOME/go
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin:/usr/local/bin
go version
  1. @Sunidhi-Gaonkar1 will raise PR for build.sh and update you. For changes in Jenkins file we will wait for your PR to get merged.
  2. @Sunidhi-Gaonkar1 will be looking into pytest errors due to PyYAML 6.0.1.
  3. For dynamic provision of ppc64le servers, you can raise request with OpenStack Access. While raising vm request select "I'do like to have access to the Openstack GUI/API" option (as highlighted in below image) and use Openstack APIs for dynamically provision ppc64le servers. image
peterzhuamazon commented 7 months ago

Thanks @Sapana-Khemkar ,

On 7, @prudhvigodithi has completed the review of my ci-runners changes.

Since building these images takes a lot of time, so I plan to reduce the load if possible.

This PR is in review to add gh cli ppc64le support, and I will give it a few days to see gh team reply.

If their review and support takes a lot of time, I will merge #4336 and start building the images next week.

Thanks.

peterzhuamazon commented 7 months ago

On 6, I am still a bit confused as that seems like still manual approaches to request servers, without internally connect through EC2 APIs.

Or is there is any plugin that can directly connect Jenkins provision agents with openstack, we still need to look into that in details.

seth-priya commented 7 months ago

Or is there is any plugin that can directly connect Jenkins provision agents with openstack, we still need to look into that in details.

@peterzhuamazon yes, there is an openstack plugin for jenkins that can help with dynamic provisioning of VMs once you have openstack access - https://plugins.jenkins.io/openstack-cloud/ , https://plugins.jenkins.io/openstack-cloud/releases/

peterzhuamazon commented 7 months ago

Sent OpenStack Request.

peterzhuamazon commented 7 months ago

Due to #4241 #4370, we will delay the image generation for ppc64le, to unblock 2.12 release issues with IntegTests.

Sunidhi-Gaonkar1 commented 7 months ago

@peterzhuamazon, I was able to resolve the pytest errors faced for #4348. The errors were thrown due to code linting issue. The sorting order was fixed after running command pipenv run isort . as mentioned in https://github.com/opensearch-project/opensearch-build/blob/main/DEVELOPER_GUIDE.md#code-linting.

peterzhuamazon commented 7 months ago

@peterzhuamazon, I was able to resolve the pytest errors faced for #4348. The errors were thrown due to code linting issue. The sorting order was fixed after running command pipenv run isort . as mentioned in https://github.com/opensearch-project/opensearch-build/blob/main/DEVELOPER_GUIDE.md#code-linting.

Hi @Sunidhi-Gaonkar1 if you resolve the sort issue and actually run pipenv run pytest you will see new issues. The isort issues is expected and not related to the pytest issues.

Thanks.

peterzhuamazon commented 7 months ago

Receive credentials on accessing to OpenStack and its endpoints. Will work on connecting to Jenkins.

Sunidhi-Gaonkar1 commented 7 months ago

Raised PR #4413 to fix pytests and update build.sh to add ppc64le build target. Please review, Thank you.