Open peterzhuamazon opened 1 month ago
In short term:
In long term:
With the success of the AL2 compilation, we will go ahead with the S3 approach for now, and see if we can bring it up for 2.15.0. If not, we will fall back to use AL2 for all the builds. Else, we will use Almalinux8 for the build and AL2 for knnlib.
https://github.com/opensearch-project/opensearch-build/issues/4379#issuecomment-2140954823
Start checking the jenkins lib and write a POC for the knnlib pipeline.
After discussion with team on the current progress, here are some of the points suggested by comments:
Welcome comments,
Thanks.
Coming from #4379 #4687 and we are looking at possibilities to move knnlib compilation to its own stage or pipeline.
As of now, knnlib build is part of the knn build.gradle task, and will be run before the knn java plugin getting built.
Due to the uniqueness of this native lib, we could explore options to build it outside of the scope of core/plugin build.
As of now the knn team is suggesting building the lib, upload to S3, then download the lib during distribution build/assemble workflow.
There are also some limitations with the existing distribution build pipeline on Jenkins that needs to be taken care of.
Thanks.