Open SonalMahambrey1 opened 1 year ago
I am not sure whether the build team intends to publish these, or whether the team at IBM (cc: @seth-priya) wants to maintain this port, but let's move this into opensearch-build.
@dblock it would be great if the build team can publish these and we will continue to watch for Power specific issues and fix them, on our side.
@bbarani @peterzhuamazon what would it take to add those to automation? Maybe @seth-priya can contribute?
[Triages] First step is to have the IBM server in order to move forward. Adding @elfisher and @setiah for discussion.
There are a few places to get access to ppc64le VMs, one of them is https://osuosl.org/services/powerdev/request_hosting/
Hi Team, any thoughts on above?
Adding @bbarani to comment on future supporting PowerPC 64 Little Endian ..thanks!
thanks @jordarlu , we can definitely contribute the changes that are needed for building the artifacts via automation and we can also discuss about the infrastructure that's needed for the same (if needed on a call?). As of now - we already have the Power changes up-streamed for the main opensearch repository and have been able to locally build the images and test them and also have a working build and image for opensearch-dashboards. Currently, we are building/validating all of the opensearch plugins separately on Power to make sure all of them work smoothly (though from what we have seen so far, they should). I am also adding our overall technical lead/architect @gerrith3 here.
Yeah - we would love to help support getting a CI environment set up for ppc64le. I'd love to know what the options are from your side, what we would need to integrate with, etc. We have some challenges with broadly available CI platforms but we do have several solutions, alternatives, and work arounds to integrate with most/many environments.
@dblock, @bbarani, @jordarlu and others, I am from @seth-priya's team. We can help adding/maintaining ppc64le support to CI environment. Let us know how can we move further on this.
@Sapana-Khemkar Thanks for your support. How do you want to proceed with next steps? Would a kick-off call help to get things started soon? It would be really easy to collaborate via OpenSearch Slack workspace.
@Sapana-Khemkar Thanks for your support. How do you want to proceed with next steps? Would a kick-off call help to get things started soon? It would be really easy to collaborate via OpenSearch Slack workspace.
@bbarani joined slack and sent you message on devops channel.
Had an offline sync up with @Sapana-Khemkar, since OpenSearch supports ppc64le
, having a Power9 server and running the build and assemble workflow should give us an artifact for ppc64le. While we request for the server and integrate with the CI/CD @Sapana-Khemkar can you please post the results of the steps (manually executing the workflows on your server inside the opensearchstaging/ci-runner:ci-runner-centos7-opensearch-build-v3
docker) we discussed over the call?
Thank you
Hey for the OpenSearch folks, if you do request a free VM at OSL per the link earlier in this thread, reference my name as your IBM support contact and we will make sure it gets expedited for you. We would love to see this integrated and we encounter more users of OpenSearch on Power all the time, so this would be a great step for all of us!!
And, apologies if this should be in a thread of its own, but does anyone know if OpenSearch is keeping up with Elasticsearch's ability to be used as a vector database? I've been seeing that question come up several times and at the moment, I have no clue. ;) thanks, and apologies for hijacking another thread, although this is really yet another Power use case for us. ;)
Hi @prudhvigodithi , I am from @seth-priya and @Sapana-Khemkar's team. I was able to manually execute the build and assemble workflows on Power server and create an artifact for ppc64le.
The docker image opensearchstaging/ci-runner:ci-runner-centos7-opensearch-build-v3
is not available for power, hence was not able to execute workflows inside the container.
There were a few changes required to be made. I have attached logs of output for execution of build and assemble workflows, also attached the changes that were made to build the artifact.
OpenSearch_assemble.txt OpenSearch_build.txt OpenSearch_Patch.txt
And, apologies if this should be in a thread of its own, but does anyone know if OpenSearch is keeping up with Elasticsearch's ability to be used as a vector database? I've been seeing that question come up several times and at the moment, I have no clue. ;) thanks, and apologies for hijacking another thread, although this is really yet another Power use case for us. ;)
OpenSearch indeed has made great strides in Vector DB space. It uses K-NN functionality as documented in this blog to implement vector db functionality. The issues related to Vector DB functionalities are tracked here.
Had an offline sync up with @Sapana-Khemkar, since OpenSearch supports
ppc64le
, having a Power9 server and running the build and assemble workflow should give us an artifact for ppc64le. While we request for the server and integrate with the CI/CD @Sapana-Khemkar can you please post the results of the steps (manually executing the workflows on your server inside theopensearchstaging/ci-runner:ci-runner-centos7-opensearch-build-v3
docker) we discussed over the call? Thank you
@prudhvigodithi any update from your side? Have you raised request for ppc64le vm?
Hey @Sapana-Khemkar I have the form submitted NAME: OpenSearch Project
, support.osuosl.org #33270
.
Thank you
@bbarani @peterzhuamazon
Hey @Sapana-Khemkar I have the form submitted
NAME: OpenSearch Project
,support.osuosl.org #33270
. Thank you @bbarani @peterzhuamazon
@prudhvigodithi your request 33270 is approved. Please let me know if you face any issue.
Thanks, @Sapana-Khemkar what is the next step here? I just got an email as Approved
for the request I created. Let me know we can discuss the next steps in public slack on how to securely connect the ppc64le VM as jenkins agent.
Have a discussion with @prudhvigodithi and we need to add a test ci-runner image layer of linux/ppc64le
, then try to start it up on the requested server before trying out the built.
If successful, then we will proceed with a manual build, before adding the changes to all the ci-runner images.
I will start on creating the new ci-runner image soon.
Thanks, @Sapana-Khemkar what is the next step here? I just got an email as
Approved
for the request I created. Let me know we can discuss the next steps in public slack on how to securely connect the ppc64le VM as jenkins agent.
@prudhvigodithi your request is approved from IBM. Now Lance Albertson from OSU will create VM. But he have few questions. I am pasting it here. Can you please respond to these?
Distribution: CentOS
Will AlmaLinux work for this or do you want CentOS Stream? Also do you want The 8 or 9 release?
Est Size Of User Community: NA
Expected Usage Model: The POWER is used to build and assemble the OpenSearch software for ppc64le architecture.
Flavor: xlarge
Number Of Nodes: 5
Is this quantity a hard requirement is are you able to reduce this to something smaller? I want to make sure we only allocate what's really needed.
cc: @ramereth
@Sapana-Khemkar since we are using docker here for the build, as long as there is docker installed for base OS on that machine, we good.
There are issues generating the ppc64le layer on docker, mainly due to pip installation here:
=> ERROR [linux/ppc64le 15/22] RUN ln -sfn /usr/local/bin/python3.9 /usr/bin/python3 && ln -sfn /usr/local/bin/pip3.9 /usr/bin/pip && ln -sfn /usr/local/bin/pip3.9 /usr/local/bin/pip && ln -sfn /usr/local/bin/pip3.9 /usr/bin/pip3 && pip3 install pip==23.1.2 && pip3 install pipenv==2023.6.12 81.0s
------
> [linux/ppc64le 15/22] RUN ln -sfn /usr/local/bin/python3.9 /usr/bin/python3 && ln -sfn /usr/local/bin/pip3.9 /usr/bin/pip && ln -sfn /usr/local/bin/pip3.9 /usr/local/bin/pip && ln -sfn /usr/local/bin/pip3.9 /usr/bin/pip3 && pip3 install pip==23.1.2 && pip3 install pipenv==2023.6.12 awscli==1.22.12:
#0 6.242 Collecting pip==23.1.2
#0 6.365 Downloading pip-23.1.2-py3-none-any.whl (2.1 MB)
#0 7.345 Installing collected packages: pip
#0 7.345 Attempting uninstall: pip
#0 7.347 Found existing installation: pip 21.2.3
#0 8.631 Uninstalling pip-21.2.3:
#0 9.266 Successfully uninstalled pip-21.2.3
#0 19.95 Successfully installed pip-23.1.2
#0 19.95 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#0 25.83 Collecting pipenv==2023.6.12
#0 26.07 Downloading pipenv-2023.6.12-py3-none-any.whl (2.9 MB)
#0 26.77 ���������������������������������������� 2.9/2.9 MB 4.4 MB/s eta 0:00:00
#0 34.13 Collecting awscli==1.22.12
#0 34.16 Downloading awscli-1.22.12-py3-none-any.whl (3.8 MB)
#0 35.04 ���������������������������������������� 3.8/3.8 MB 4.4 MB/s eta 0:00:00
#0 35.97 Collecting certifi (from pipenv==2023.6.12)
#0 36.00 Downloading certifi-2023.11.17-py3-none-any.whl (162 kB)
#0 36.07 ��������������������������������������� 162.5/162.5 kB 3.6 MB/s eta 0:00:00
#0 39.04 Collecting setuptools>=67.0.0 (from pipenv==2023.6.12)
#0 39.07 Downloading setuptools-69.0.3-py3-none-any.whl (819 kB)
#0 39.29 ��������������������������������������� 819.5/819.5 kB 4.3 MB/s eta 0:00:00
#0 39.50 Collecting virtualenv-clone>=0.2.5 (from pipenv==2023.6.12)
#0 39.53 Downloading virtualenv_clone-0.5.7-py3-none-any.whl (6.6 kB)
#0 40.75 Collecting virtualenv>=20.17.1 (from pipenv==2023.6.12)
#0 40.78 Downloading virtualenv-20.25.0-py3-none-any.whl (3.8 MB)
#0 41.66 ���������������������������������������� 3.8/3.8 MB 4.4 MB/s eta 0:00:00
#0 49.00 Collecting botocore==1.23.12 (from awscli==1.22.12)
#0 49.03 Downloading botocore-1.23.12-py3-none-any.whl (8.2 MB)
#0 50.91 ���������������������������������������� 8.2/8.2 MB 4.4 MB/s eta 0:00:00
#0 51.51 Collecting docutils<0.16,>=0.10 (from awscli==1.22.12)
#0 51.54 Downloading docutils-0.15.2-py3-none-any.whl (547 kB)
#0 51.70 ��������������������������������������� 547.6/547.6 kB 4.2 MB/s eta 0:00:00
#0 51.93 Collecting s3transfer<0.6.0,>=0.5.0 (from awscli==1.22.12)
#0 51.96 Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB)
#0 52.00 ���������������������������������������� 79.5/79.5 kB 3.2 MB/s eta 0:00:00
#0 52.46 Collecting PyYAML<5.5,>=3.10 (from awscli==1.22.12)
#0 52.49 Downloading PyYAML-5.4.1.tar.gz (175 kB)
#0 52.56 ��������������������������������������� 175.1/175.1 kB 3.7 MB/s eta 0:00:00
#0 53.91 Installing build dependencies: started
#0 77.90 Installing build dependencies: finished with status 'done'
#0 77.91 Getting requirements to build wheel: started
#0 80.14 Getting requirements to build wheel: finished with status 'error'
#0 80.22 error: subprocess-exited-with-error
#0 80.22
#0 80.22 � Getting requirements to build wheel did not run successfully.
#0 80.22 � exit code: 1
#0 80.22 ��> [48 lines of output]
#0 80.22 running egg_info
#0 80.22 writing lib3/PyYAML.egg-info/PKG-INFO
#0 80.22 writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
#0 80.22 writing top-level names to lib3/PyYAML.egg-info/top_level.txt
#0 80.22 Traceback (most recent call last):
#0 80.22 File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
#0 80.22 main()
#0 80.22 File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
#0 80.22 json_out['return_val'] = hook(**hook_input['kwargs'])
#0 80.22 File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
#0 80.22 return hook(config_settings)
#0 80.22 File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
#0 80.22 return self._get_build_requires(config_settings, requirements=['wheel'])
#0 80.22 File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
#0 80.22 self.run_setup()
#0 80.22 File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 311, in run_setup
#0 80.22 exec(code, locals())
#0 80.22 File "<string>", line 271, in <module>
#0 80.22 File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 103, in setup
#0 80.22 return distutils.core.setup(**attrs)
#0 80.22 File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
#0 80.22 return run_commands(dist)
#0 80.22 File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
#0 80.22 dist.run_commands()
#0 80.22 File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
#0 80.22 self.run_command(cmd)
#0 80.22 File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 963, in run_command
#0 80.22 super().run_command(command)
#0 80.22 File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
#0 80.22 cmd_obj.run()
#0 80.22 File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 321, in run
#0 80.22 self.find_sources()
#0 80.22 File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 329, in find_sources
#0 80.22 mm.run()
#0 80.22 File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 551, in run
#0 80.22 self.add_defaults()
#0 80.22 File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
#0 80.22 sdist.add_defaults(self)
#0 80.22 File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/command/sdist.py", line 112, in add_defaults
#0 80.22 super().add_defaults()
#0 80.22 File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
#0 80.22 self._add_defaults_ext()
#0 80.22 File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
#0 80.22 self.filelist.extend(build_ext.get_source_files())
#0 80.22 File "<string>", line 201, in get_source_files
#0 80.22 File "/tmp/pip-build-env-pp05v2oy/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
#0 80.22 raise AttributeError(attr)
#0 80.22 AttributeError: cython_sources
#0 80.22 [end of output]
#0 80.22
#0 80.22 note: This error originates from a subprocess, and is likely not a problem with pip.
#0 80.25 error: subprocess-exited-with-error
#0 80.25
#0 80.25 � Getting requirements to build wheel did not run successfully.
#0 80.25 � exit code: 1
#0 80.25 ��> See above for output.
#0 80.25
#0 80.25 note: This error originates from a subprocess, and is likely not a problem with pip.
#0 80.28
#0 80.28 [notice] A new release of pip is available: 23.1.2 -> 23.3.2
#0 80.28 [notice] To update, run: pip install --upgrade pip
Thanks, @Sapana-Khemkar what is the next step here? I just got an email as
Approved
for the request I created. Let me know we can discuss the next steps in public slack on how to securely connect the ppc64le VM as jenkins agent.@prudhvigodithi your request is approved from IBM. Now Lance Albertson from OSU will create VM. But he have few questions. I am pasting it here. Can you please respond to these?
Distribution: CentOS
Will AlmaLinux work for this or do you want CentOS Stream? Also do you want The 8 or 9 release?
Est Size Of User Community: NA Expected Usage Model: The POWER is used to build and assemble the OpenSearch software for ppc64le architecture. Flavor: xlarge Number Of Nodes: 5
Is this quantity a hard requirement is are you able to reduce this to something smaller? I want to make sure we only allocate what's really needed.
cc: @ramereth
@peterzhuamazon / @prudhvigodithi -- so I take it AlmaLinux will work for you. What about the number of Nodes -- is 5 a hard requirement?
Can the responses please be posted to: https://support.osuosl.org/Ticket/Display.html?id=33270. in response to the question from Lance Albertson. Once that is done Lance will assign someone to create a VM for you'll to use.
Hey @janani66, the link asks for login details https://support.osuosl.org/Ticket/Display.html?id=33270 so we can continue to discuss on this issue. The requirement can be adjusted later based on how frequently we want to build the ppc64le distribution (scale up and scale down as required), since we are at POC phase, its ok to get a basic VM that works to build and assemble the OpenSearch distribution. For linux x64 distribution build the existing setup uses C54xlarge (Ref: https://github.com/opensearch-project/opensearch-ci/blob/main/lib/compute/agent-nodes.ts#L74-L86) which can handle multiple parallel OpenSearch distribution builds. So please adjust the requirement that can be given for testing and flexible enough to update if required. Thank you Adding @peterzhuamazon
Hey @janani66, the link asks for login details https://support.osuosl.org/Ticket/Display.html?id=33270 so we can continue to discuss on this issue. The requirement can be adjusted later based on how frequently we want to build the ppc64le distribution (scale up and scale down as required), since we are at POC phase, its ok to get a basic VM that works to build and assemble the OpenSearch distribution. For linux x64 distribution build the existing setup uses C54xlarge (Ref: https://github.com/opensearch-project/opensearch-ci/blob/main/lib/compute/agent-nodes.ts#L74-L86) which can handle multiple parallel OpenSearch distribution builds. So please adjust the requirement that can be given for testing and flexible enough to update if required. Thank you Adding @peterzhuamazon
As of now either Rocky/Alma/Ubuntu based server is fine, as we are using docker to build.
We can have 5 nodes to start with, each having 16CPU 32GB setup, as each will be loaded with 4 docker containers (4CPU/8GB each). That means a total of 20 executors with containers which is sufficient enough for now.
Tho keep in mind our current setup is utilizing EC2 to dynamically provision runners, which means if there are a lot of builds, then 20 executors may be execeeded later. If there is a way to provision ppc64le server dynamically through EC2 that would be even better.
Thanks.
@peterzhuamazon should be able to use the OpenStack API to automatically provision what you need, however @janani66 mentioned in the ticket that you wanted us to create the VMs for you instead of getting API access.
@janani66 if you can get email addresses for @peterzhuamazon and whomever else on this issue, I can add them to our ticket so we can discuss this further without going back and forth here.
I will try out the 6.0.1 version of PyYAML today to see if that will pass the ci runner creation.
@peterzhuamazon should be able to use the OpenStack API to automatically provision what you need, however @janani66 mentioned in the ticket that you wanted us to create the VMs for you instead of getting API access.
@janani66 if you can get email addresses for @peterzhuamazon and whomever else on this issue, I can add them to our ticket so we can discuss this further without going back and forth here.
I dont think I have a login credential for the ticket now, assuming this one? https://support.osuosl.org/Ticket/Display.html?id=33270
I can give you more information once I can login and comment there.
@peterzhuamazon it's best if we just add you to the ticket directly and move the conversation to email only. We don't typically open up logins directly to the tickets.
@peterzhuamazon it's best if we just add you to the ticket directly and move the conversation to email only. We don't typically open up logins directly to the tickets.
Got it. I will reply through email soon after.
Able to login to the initial almalinux8 test server.
Server Info
# uname -a
Linux opensearch-1.novalocal 4.18.0-513.9.1.el8_9.ppc64le #1 SMP Sat Dec 2 10:18:07 UTC 2023 ppc64le ppc64le ppc64le GNU/Linux
Alma Linux 8 16CPU 32GB MEM 300GB Storage
Will use RockyLinux 8 docker setups to verify if all the packages can be installed.
Issue: gh client has no ppc64le support.
Error:
Problem: cannot install the best candidate for the job
- package gh-2.42.0-1.i386 from gh-cli does not have a compatible architecture
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
# yum list gh --showduplicates
Last metadata expiration check: 0:01:43 ago on Thu 11 Jan 2024 09:57:38 PM UTC.
Available Packages
gh.aarch64 2.42.0-1 gh-cli
gh.armv6hl 2.42.0-1 gh-cli
gh.i386 2.42.0-1 gh-cli
gh.x86_64 2.42.0-1 gh-cli
As of https://github.com/cli/cli/releases/tag/v2.42.0 we are not seeing support on ppc64le for gh client.
Add a few tweaks before making maven run:
# mvn --version
Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae)
Maven home: /usr/local/apache-maven
Java version: 11.0.15, vendor: Eclipse Adoptium, runtime: /usr/lib/jvm/temurin-11-jdk
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.18.0-513.9.1.el8_9.ppc64le", arch: "ppc64le", family: "unix"
My PRs:
IBM PRs:
Issue: gh client has no ppc64le support.
Error: Problem: cannot install the best candidate for the job - package gh-2.42.0-1.i386 from gh-cli does not have a compatible architecture (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
# yum list gh --showduplicates Last metadata expiration check: 0:01:43 ago on Thu 11 Jan 2024 09:57:38 PM UTC. Available Packages gh.aarch64 2.42.0-1 gh-cli gh.armv6hl 2.42.0-1 gh-cli gh.i386 2.42.0-1 gh-cli gh.x86_64 2.42.0-1 gh-cli
As of https://github.com/cli/cli/releases/tag/v2.42.0 we are not seeing support on ppc64le for gh client.
Hi @peterzhuamazon, We are working on enabling power support for gh-cli, as a workaround gh-cli can be installed using following command :
go install github.com/cli/cli/v2/cmd/gh@latest
Issue: gh client has no ppc64le support.
Error: Problem: cannot install the best candidate for the job - package gh-2.42.0-1.i386 from gh-cli does not have a compatible architecture (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
# yum list gh --showduplicates Last metadata expiration check: 0:01:43 ago on Thu 11 Jan 2024 09:57:38 PM UTC. Available Packages gh.aarch64 2.42.0-1 gh-cli gh.armv6hl 2.42.0-1 gh-cli gh.i386 2.42.0-1 gh-cli gh.x86_64 2.42.0-1 gh-cli
As of https://github.com/cli/cli/releases/tag/v2.42.0 we are not seeing support on ppc64le for gh client.
Hi @peterzhuamazon, We are working on enabling power support for gh-cli, as a workaround gh-cli can be installed using following command :
go install github.com/cli/cli/v2/cmd/gh@latest
Seems like it needs go version 1.21 or above? I will take a look on that.
Thanks.
20240122: Have meeting with IBM team on current status:
cc: @Sapana-Khemkar @seth-priya @Sunidhi-Gaonkar1 @bbarani @prudhvigodithi
Thanks.
We have raised PR for ppc64le support. But it may take a little time. Meanwhile can you install go v1.21.0 using below steps
wget https://go.dev/dl/go1.21.0.linux-ppc64le.tar.gz
tar -C /usr/local -xzf go1.21.0.linux-ppc64le.tar.gz
export GOROOT=/usr/local/go
export GOPATH=$HOME/go
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin:/usr/local/bin
go version
Thanks @Sapana-Khemkar ,
On 7, @prudhvigodithi has completed the review of my ci-runners changes.
Since building these images takes a lot of time, so I plan to reduce the load if possible.
This PR is in review to add gh cli ppc64le support, and I will give it a few days to see gh team reply.
If their review and support takes a lot of time, I will merge #4336 and start building the images next week.
Thanks.
On 6, I am still a bit confused as that seems like still manual approaches to request servers, without internally connect through EC2 APIs.
Or is there is any plugin that can directly connect Jenkins provision agents with openstack, we still need to look into that in details.
Or is there is any plugin that can directly connect Jenkins provision agents with openstack, we still need to look into that in details.
@peterzhuamazon yes, there is an openstack plugin for jenkins that can help with dynamic provisioning of VMs once you have openstack access - https://plugins.jenkins.io/openstack-cloud/ , https://plugins.jenkins.io/openstack-cloud/releases/
Sent OpenStack Request.
Due to #4241 #4370, we will delay the image generation for ppc64le, to unblock 2.12 release issues with IntegTests.
@peterzhuamazon, I was able to resolve the pytest errors faced for #4348. The errors were thrown due to code linting issue. The sorting order was fixed after running command pipenv run isort .
as mentioned in https://github.com/opensearch-project/opensearch-build/blob/main/DEVELOPER_GUIDE.md#code-linting.
@peterzhuamazon, I was able to resolve the pytest errors faced for #4348. The errors were thrown due to code linting issue. The sorting order was fixed after running command
pipenv run isort .
as mentioned in https://github.com/opensearch-project/opensearch-build/blob/main/DEVELOPER_GUIDE.md#code-linting.
Hi @Sunidhi-Gaonkar1 if you resolve the sort issue and actually run pipenv run pytest you will see new issues. The isort issues is expected and not related to the pytest issues.
Thanks.
Receive credentials on accessing to OpenStack and its endpoints. Will work on connecting to Jenkins.
Raised PR #4413 to fix pytests and update build.sh to add ppc64le build target. Please review, Thank you.
Opensearch snapshot for X64 is available at https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-linux-x64-latest.tar.gz As power architecture is supported now, any update when the similar artifacts will be published for ppc64le as well ?