search

opensearch-project / opensearch-build

🧰 OpenSearch / OpenSearch-Dashboards Build Systems
Apache License 2.0
135 stars 271 forks source link

[RPM M1] Add a new block to call the generation code for RPM #1545

Closed peterzhuamazon closed 2 years ago

peterzhuamazon commented 2 years ago
Tasks Estimate Status Notes Dependencies
The generation code should pull the artifacts from the build workflow to a temporary location 1 Complete   Build workflow must provide usable artifacts
The code will call existing install function to install plugins on min artifacts 1 Complete    
After installation, the code will execute a tool or utility to wrap all the content into a RPM package 5 Complete Require writing a script to utilize FPM to start with and later implement in pure python code.

20220204: We might change to rpmbuild directly without using FPM. See comments.		 FPM usages
The code will also add dependencies to the RPM package so that things like JDK and additional libs for plugins can be installed and pulled separately 5 Complete Need to study on RPM dependency setups RPM Build Dependencies and the locations of each dependent artifact
The code will move the RPM package from the temp location to dist folder 2 Complete    
peterzhuamazon commented 2 years ago

20220204: FPM is having some issues along the way:

Overall, it is a great tool for quick building an RPM without spec file. However, for our production usages I think using rpmbuild directly probably better here.

Still testing.

peterzhuamazon commented 2 years ago

It can finally run now with rpmbuild integrated:

Test Logs ``` % ./assemble.sh builds/opensearch/manifest.yml Installing dependencies in . ... Installing dependencies from Pipfile.lock (16e10a)... 🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00 To activate this project's virtualenv, run pipenv shell. Alternatively, run a command inside the virtualenv with pipenv run. Running ./src/run_assemble.py builds/opensearch/manifest.yml ... 2022-03-09 02:25:41 INFO Bundling OpenSearch (x64) on linux into /home/opensearch-build-peterzhuamazon/dist/opensearch ... 2022-03-09 02:25:41 INFO Copied min bundle to /tmp/tmphii43sg4/opensearch-min-1.3.0-x64.rpm. 2022-03-09 02:25:41 INFO Start creating distribution rpm for OpenSearch. 2022-03-09 02:25:41 INFO Extracting dist into /tmp/tmphii43sg4. 2022-03-09 02:25:41 INFO Convert rpm to cpio for extraction: /tmp/tmphii43sg4/opensearch-min-1.3.0-x64.rpm to /tmp/tmphii43sg4/opensearch-min-1.3.0-x64.cpio 2022-03-09 02:25:44 INFO Extract cpio /tmp/tmphii43sg4/opensearch-min-1.3.0-x64.cpio content to /tmp/tmphii43sg4 2022-03-09 02:25:45 INFO Move /tmp/tmphii43sg4/usr/share/opensearch to /tmp/tmphii43sg4/opensearch-1.3.0 for plugin installation 2022-03-09 02:25:45 INFO Extracted dist into /tmp/tmphii43sg4. 2022-03-09 02:25:45 INFO Executing "bash /home/opensearch-build-peterzhuamazon/scripts/components/OpenSearch/install.sh -v 1.3.0 -p linux -a x64 -d rpm -f /home/opensearch-build-peterzhuamazon/builds/opensearch -o /tmp/tmphii43sg4/opensearch-1.3.0" in /tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:d:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:s:o:p:a:d:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:s:o:p:a:d:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:s:o:p:a:d:f: arg + case $arg in + DISTRIBUTION=rpm + getopts :h:v:s:o:p:a:d:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:s:o:p:a:d:f: arg + case $arg in + OUTPUT=/tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:d:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + '[' -z rpm ']' + '[' rpm = tar ']' + '[' rpm = rpm ']' ++ dirname /home/opensearch-build-peterzhuamazon/scripts/components/OpenSearch/install.sh + DIR=/home/opensearch-build-peterzhuamazon/scripts/components/OpenSearch + echo /home/opensearch-build-peterzhuamazon/scripts/components/OpenSearch /home/opensearch-build-peterzhuamazon/scripts/components/OpenSearch + cd /home/opensearch-build-peterzhuamazon/scripts/components/OpenSearch + cp -a ../../../scripts/pkg/service_templates/opensearch/etc ../../../scripts/pkg/service_templates/opensearch/usr /tmp/tmphii43sg4/opensearch-1.3.0/../ + cp -a ../../../scripts/pkg/build_templates/opensearch/opensearch.rpm.spec /tmp/tmphii43sg4/opensearch-1.3.0/../ 2022-03-09 02:25:45 INFO Installing job-scheduler 2022-03-09 02:25:45 INFO Executing "/tmp/tmphii43sg4/opensearch-1.3.0/bin/opensearch-plugin install --batch file:/tmp/tmphii43sg4/opensearch-job-scheduler-1.3.0.0.zip" in /tmp/tmphii43sg4/opensearch-1.3.0 -> Installing file:/tmp/tmphii43sg4/opensearch-job-scheduler-1.3.0.0.zip -> Downloading file:/tmp/tmphii43sg4/opensearch-job-scheduler-1.3.0.0.zip -> Installed opensearch-job-scheduler with folder name opensearch-job-scheduler 2022-03-09 02:25:47 INFO Executing "bash /home/opensearch-build-peterzhuamazon/scripts/default/install.sh -v 1.3.0 -p linux -a x64 -f /home/opensearch-build-peterzhuamazon/builds/opensearch -o /tmp/tmphii43sg4/opensearch-1.3.0" in /tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:s:o:p:a:f: arg + case $arg in + OUTPUT=/tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + exit 0 2022-03-09 02:25:47 INFO Installing alerting 2022-03-09 02:25:47 INFO Executing "/tmp/tmphii43sg4/opensearch-1.3.0/bin/opensearch-plugin install --batch file:/tmp/tmphii43sg4/opensearch-alerting-1.3.0.0.zip" in /tmp/tmphii43sg4/opensearch-1.3.0 -> Installing file:/tmp/tmphii43sg4/opensearch-alerting-1.3.0.0.zip -> Downloading file:/tmp/tmphii43sg4/opensearch-alerting-1.3.0.0.zip @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission createClassLoader * java.lang.RuntimePermission getClassLoader * java.net.NetPermission getProxySelector * java.net.SocketPermission * connect,resolve See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. -> Installed opensearch-alerting with folder name opensearch-alerting 2022-03-09 02:25:48 INFO Executing "bash /home/opensearch-build-peterzhuamazon/scripts/default/install.sh -v 1.3.0 -p linux -a x64 -f /home/opensearch-build-peterzhuamazon/builds/opensearch -o /tmp/tmphii43sg4/opensearch-1.3.0" in /tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:s:o:p:a:f: arg + case $arg in + OUTPUT=/tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + exit 0 2022-03-09 02:25:48 INFO Installing asynchronous-search 2022-03-09 02:25:48 INFO Executing "/tmp/tmphii43sg4/opensearch-1.3.0/bin/opensearch-plugin install --batch file:/tmp/tmphii43sg4/opensearch-asynchronous-search-1.3.0.0.zip" in /tmp/tmphii43sg4/opensearch-1.3.0 -> Installing file:/tmp/tmphii43sg4/opensearch-asynchronous-search-1.3.0.0.zip -> Downloading file:/tmp/tmphii43sg4/opensearch-asynchronous-search-1.3.0.0.zip @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission createClassLoader * java.lang.RuntimePermission getClassLoader * java.net.NetPermission getProxySelector * java.net.SocketPermission * connect,resolve See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. -> Installed opensearch-asynchronous-search with folder name opensearch-asynchronous-search 2022-03-09 02:25:49 INFO Executing "bash /home/opensearch-build-peterzhuamazon/scripts/default/install.sh -v 1.3.0 -p linux -a x64 -f /home/opensearch-build-peterzhuamazon/builds/opensearch -o /tmp/tmphii43sg4/opensearch-1.3.0" in /tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:s:o:p:a:f: arg + case $arg in + OUTPUT=/tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + exit 0 2022-03-09 02:25:49 INFO Installing index-management 2022-03-09 02:25:49 INFO Executing "/tmp/tmphii43sg4/opensearch-1.3.0/bin/opensearch-plugin install --batch file:/tmp/tmphii43sg4/opensearch-index-management-1.3.0.0.zip" in /tmp/tmphii43sg4/opensearch-1.3.0 -> Installing file:/tmp/tmphii43sg4/opensearch-index-management-1.3.0.0.zip -> Downloading file:/tmp/tmphii43sg4/opensearch-index-management-1.3.0.0.zip @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission createClassLoader * java.lang.RuntimePermission getClassLoader * java.net.NetPermission getProxySelector * java.net.SocketPermission * connect,resolve See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. -> Installed opensearch-index-management with folder name opensearch-index-management 2022-03-09 02:25:51 INFO Executing "bash /home/opensearch-build-peterzhuamazon/scripts/default/install.sh -v 1.3.0 -p linux -a x64 -f /home/opensearch-build-peterzhuamazon/builds/opensearch -o /tmp/tmphii43sg4/opensearch-1.3.0" in /tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:s:o:p:a:f: arg + case $arg in + OUTPUT=/tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + exit 0 2022-03-09 02:25:51 INFO Installing k-NN 2022-03-09 02:25:51 INFO Executing "/tmp/tmphii43sg4/opensearch-1.3.0/bin/opensearch-plugin install --batch file:/tmp/tmphii43sg4/opensearch-knn-1.3.0.0.zip" in /tmp/tmphii43sg4/opensearch-1.3.0 -> Installing file:/tmp/tmphii43sg4/opensearch-knn-1.3.0.0.zip -> Downloading file:/tmp/tmphii43sg4/opensearch-knn-1.3.0.0.zip @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission loadLibrary.opensearchknn_faiss * java.lang.RuntimePermission loadLibrary.opensearchknn_nmslib * java.net.SocketPermission * connect,resolve See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. -> Installed opensearch-knn with folder name opensearch-knn 2022-03-09 02:25:52 INFO Executing "bash /home/opensearch-build-peterzhuamazon/scripts/default/install.sh -v 1.3.0 -p linux -a x64 -f /home/opensearch-build-peterzhuamazon/builds/opensearch -o /tmp/tmphii43sg4/opensearch-1.3.0" in /tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:s:o:p:a:f: arg + case $arg in + OUTPUT=/tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + exit 0 2022-03-09 02:25:52 INFO Installing security 2022-03-09 02:25:52 INFO Executing "/tmp/tmphii43sg4/opensearch-1.3.0/bin/opensearch-plugin install --batch file:/tmp/tmphii43sg4/opensearch-security-1.3.0.0.zip" in /tmp/tmphii43sg4/opensearch-1.3.0 -> Installing file:/tmp/tmphii43sg4/opensearch-security-1.3.0.0.zip -> Downloading file:/tmp/tmphii43sg4/opensearch-security-1.3.0.0.zip @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.io.FilePermission /proc/sys/net/core/somaxconn#plus read * java.io.SerializablePermission enableSubstitution * java.lang.RuntimePermission accessClassInPackage.com.sun.jndi.* * java.lang.RuntimePermission accessClassInPackage.sun.misc * java.lang.RuntimePermission accessClassInPackage.sun.nio.ch * java.lang.RuntimePermission accessClassInPackage.sun.security.x509 * java.lang.RuntimePermission accessDeclaredMembers * java.lang.RuntimePermission accessUserInformation * java.lang.RuntimePermission createClassLoader * java.lang.RuntimePermission getClassLoader * java.lang.RuntimePermission setContextClassLoader * java.lang.RuntimePermission shutdownHooks * java.lang.reflect.ReflectPermission suppressAccessChecks * java.net.NetPermission getNetworkInformation * java.net.NetPermission getProxySelector * java.net.SocketPermission * connect,accept,resolve * java.security.SecurityPermission getProperty.ssl.KeyManagerFactory.algorithm * java.security.SecurityPermission insertProvider.BC * java.security.SecurityPermission org.apache.xml.security.register * java.security.SecurityPermission putProviderProperty.BC * java.security.SecurityPermission setProperty.ocsp.enable * java.util.PropertyPermission * read,write * java.util.PropertyPermission org.apache.xml.security.ignoreLineBreaks write * javax.security.auth.AuthPermission doAs * javax.security.auth.AuthPermission modifyPrivateCredentials * javax.security.auth.kerberos.ServicePermission * accept See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. -> Installed opensearch-security with folder name opensearch-security 2022-03-09 02:25:54 INFO Executing "bash /home/opensearch-build-peterzhuamazon/scripts/default/install.sh -v 1.3.0 -p linux -a x64 -f /home/opensearch-build-peterzhuamazon/builds/opensearch -o /tmp/tmphii43sg4/opensearch-1.3.0" in /tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:s:o:p:a:f: arg + case $arg in + OUTPUT=/tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + exit 0 2022-03-09 02:25:54 INFO Installing performance-analyzer 2022-03-09 02:25:54 INFO Executing "/tmp/tmphii43sg4/opensearch-1.3.0/bin/opensearch-plugin install --batch file:/tmp/tmphii43sg4/opensearch-performance-analyzer-1.3.0.0.zip" in /tmp/tmphii43sg4/opensearch-1.3.0 -> Installing file:/tmp/tmphii43sg4/opensearch-performance-analyzer-1.3.0.0.zip -> Downloading file:/tmp/tmphii43sg4/opensearch-performance-analyzer-1.3.0.0.zip @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * com.sun.tools.attach.AttachPermission attachVirtualMachine * com.sun.tools.attach.AttachPermission createAttachProvider * java.io.FilePermission /-#plus read,write,execute,delete,readlink * java.io.FilePermission build/tmp/junit_metrics#plus read * java.lang.RuntimePermission accessClassInPackage.jdk.internal.misc * java.lang.RuntimePermission accessClassInPackage.sun.tools.attach * java.lang.RuntimePermission accessDeclaredMembers * java.lang.RuntimePermission createClassLoader * java.lang.RuntimePermission defineClass * java.lang.RuntimePermission loadLibrary.attach * java.lang.RuntimePermission manageProcess * java.lang.reflect.ReflectPermission suppressAccessChecks See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. -> Installed opensearch-performance-analyzer with folder name opensearch-performance-analyzer 2022-03-09 02:25:57 INFO Executing "bash /home/opensearch-build-peterzhuamazon/scripts/components/performance-analyzer/install.sh -v 1.3.0 -p linux -a x64 -f /home/opensearch-build-peterzhuamazon/builds/opensearch -o /tmp/tmphii43sg4/opensearch-1.3.0" in /tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:s:o:p:a:f: arg + case $arg in + OUTPUT=/tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + mv /tmp/tmphii43sg4/opensearch-1.3.0/plugins/opensearch-performance-analyzer/performance-analyzer-rca /tmp/tmphii43sg4/opensearch-1.3.0/ + mv /tmp/tmphii43sg4/opensearch-1.3.0/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli /tmp/tmphii43sg4/opensearch-1.3.0/bin + rm -rf /tmp/tmphii43sg4/opensearch-1.3.0/bin/opensearch-performance-analyzer 2022-03-09 02:25:57 INFO Installing anomaly-detection 2022-03-09 02:25:57 INFO Executing "/tmp/tmphii43sg4/opensearch-1.3.0/bin/opensearch-plugin install --batch file:/tmp/tmphii43sg4/opensearch-anomaly-detection-1.3.0.0.zip" in /tmp/tmphii43sg4/opensearch-1.3.0 -> Installing file:/tmp/tmphii43sg4/opensearch-anomaly-detection-1.3.0.0.zip -> Downloading file:/tmp/tmphii43sg4/opensearch-anomaly-detection-1.3.0.0.zip @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission accessClassInPackage.sun.misc * java.lang.RuntimePermission accessDeclaredMembers * java.lang.RuntimePermission getClassLoader * java.lang.RuntimePermission setContextClassLoader * java.lang.reflect.ReflectPermission suppressAccessChecks * java.net.SocketPermission * connect,resolve * javax.management.MBeanPermission org.apache.commons.pool2.impl.GenericObjectPool#-[org.apache.commons.pool2:name=pool,type=GenericObjectPool] registerMBean * javax.management.MBeanPermission org.apache.commons.pool2.impl.GenericObjectPool#-[org.apache.commons.pool2:name=pool,type=GenericObjectPool] unregisterMBean * javax.management.MBeanServerPermission createMBeanServer * javax.management.MBeanTrustPermission register See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. -> Installed opensearch-anomaly-detection with folder name opensearch-anomaly-detection 2022-03-09 02:25:58 INFO Executing "bash /home/opensearch-build-peterzhuamazon/scripts/default/install.sh -v 1.3.0 -p linux -a x64 -f /home/opensearch-build-peterzhuamazon/builds/opensearch -o /tmp/tmphii43sg4/opensearch-1.3.0" in /tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:s:o:p:a:f: arg + case $arg in + OUTPUT=/tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + exit 0 2022-03-09 02:25:58 INFO Installing cross-cluster-replication 2022-03-09 02:25:58 INFO Executing "/tmp/tmphii43sg4/opensearch-1.3.0/bin/opensearch-plugin install --batch file:/tmp/tmphii43sg4/opensearch-cross-cluster-replication-1.3.0.0.zip" in /tmp/tmphii43sg4/opensearch-1.3.0 -> Installing file:/tmp/tmphii43sg4/opensearch-cross-cluster-replication-1.3.0.0.zip -> Downloading file:/tmp/tmphii43sg4/opensearch-cross-cluster-replication-1.3.0.0.zip -> Installed opensearch-cross-cluster-replication with folder name opensearch-cross-cluster-replication 2022-03-09 02:26:00 INFO Executing "bash /home/opensearch-build-peterzhuamazon/scripts/default/install.sh -v 1.3.0 -p linux -a x64 -f /home/opensearch-build-peterzhuamazon/builds/opensearch -o /tmp/tmphii43sg4/opensearch-1.3.0" in /tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:s:o:p:a:f: arg + case $arg in + OUTPUT=/tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + exit 0 2022-03-09 02:26:00 INFO Installing sql 2022-03-09 02:26:00 INFO Executing "/tmp/tmphii43sg4/opensearch-1.3.0/bin/opensearch-plugin install --batch file:/tmp/tmphii43sg4/opensearch-sql-1.3.0.0.zip" in /tmp/tmphii43sg4/opensearch-1.3.0 -> Installing file:/tmp/tmphii43sg4/opensearch-sql-1.3.0.0.zip -> Downloading file:/tmp/tmphii43sg4/opensearch-sql-1.3.0.0.zip @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission accessDeclaredMembers * java.lang.RuntimePermission defineClass * java.lang.reflect.ReflectPermission suppressAccessChecks See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. -> Installed opensearch-sql with folder name opensearch-sql 2022-03-09 02:26:02 INFO Executing "bash /home/opensearch-build-peterzhuamazon/scripts/default/install.sh -v 1.3.0 -p linux -a x64 -f /home/opensearch-build-peterzhuamazon/builds/opensearch -o /tmp/tmphii43sg4/opensearch-1.3.0" in /tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:s:o:p:a:f: arg + case $arg in + OUTPUT=/tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + exit 0 2022-03-09 02:26:02 INFO Installing dashboards-reports 2022-03-09 02:26:02 INFO Executing "/tmp/tmphii43sg4/opensearch-1.3.0/bin/opensearch-plugin install --batch file:/tmp/tmphii43sg4/opensearch-reports-scheduler-1.3.0.0.zip" in /tmp/tmphii43sg4/opensearch-1.3.0 -> Installing file:/tmp/tmphii43sg4/opensearch-reports-scheduler-1.3.0.0.zip -> Downloading file:/tmp/tmphii43sg4/opensearch-reports-scheduler-1.3.0.0.zip @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission createClassLoader * java.lang.RuntimePermission getClassLoader * java.net.NetPermission getProxySelector * java.net.SocketPermission * connect,resolve See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. -> Installed opensearch-reports-scheduler with folder name opensearch-reports-scheduler 2022-03-09 02:26:04 INFO Executing "bash /home/opensearch-build-peterzhuamazon/scripts/default/install.sh -v 1.3.0 -p linux -a x64 -f /home/opensearch-build-peterzhuamazon/builds/opensearch -o /tmp/tmphii43sg4/opensearch-1.3.0" in /tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:s:o:p:a:f: arg + case $arg in + OUTPUT=/tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + exit 0 2022-03-09 02:26:04 INFO Installing opensearch-observability 2022-03-09 02:26:04 INFO Executing "/tmp/tmphii43sg4/opensearch-1.3.0/bin/opensearch-plugin install --batch file:/tmp/tmphii43sg4/opensearch-observability-1.3.0.0.zip" in /tmp/tmphii43sg4/opensearch-1.3.0 -> Installing file:/tmp/tmphii43sg4/opensearch-observability-1.3.0.0.zip -> Downloading file:/tmp/tmphii43sg4/opensearch-observability-1.3.0.0.zip @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission createClassLoader * java.lang.RuntimePermission getClassLoader * java.net.NetPermission getProxySelector * java.net.SocketPermission * connect,resolve See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. -> Installed opensearch-observability with folder name opensearch-observability 2022-03-09 02:26:05 INFO Executing "bash /home/opensearch-build-peterzhuamazon/scripts/default/install.sh -v 1.3.0 -p linux -a x64 -f /home/opensearch-build-peterzhuamazon/builds/opensearch -o /tmp/tmphii43sg4/opensearch-1.3.0" in /tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:s:o:p:a:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:s:o:p:a:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:s:o:p:a:f: arg + case $arg in + OUTPUT=/tmp/tmphii43sg4/opensearch-1.3.0 + getopts :h:v:s:o:p:a:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + exit 0 2022-03-09 02:26:05 INFO Installed plugins: ['opensearch-cross-cluster-replication', 'opensearch-observability', 'opensearch-anomaly-detection', 'opensearch-alerting', 'opensearch-index-management', 'opensearch-asynchronous-search', 'opensearch-knn', 'opensearch-security', 'opensearch-performance-analyzer', 'opensearch-job-scheduler', 'opensearch-reports-scheduler', 'opensearch-sql'] 2022-03-09 02:26:05 INFO Organize folder structure before generating rpm 2022-03-09 02:26:05 INFO Execute rpmbuild -bb --define '_topdir /tmp/tmphii43sg4' opensearch.rpm.spec in /tmp/tmphii43sg4 Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.Q3vOb2 + umask 022 + cd /tmp/tmphii43sg4/BUILD + exit 0 Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.tWvXqO + umask 022 + cd /tmp/tmphii43sg4/BUILD + exit 0 Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.s98mGA + umask 022 + cd /tmp/tmphii43sg4/BUILD + '[' /tmp/tmphii43sg4/BUILDROOT/opensearch-1.3.0-1.x86_64 '!=' / ']' + rm -rf /tmp/tmphii43sg4/BUILDROOT/opensearch-1.3.0-1.x86_64 ++ dirname /tmp/tmphii43sg4/BUILDROOT/opensearch-1.3.0-1.x86_64 + mkdir -p /tmp/tmphii43sg4/BUILDROOT + mkdir /tmp/tmphii43sg4/BUILDROOT/opensearch-1.3.0-1.x86_64 + cd /tmp/tmphii43sg4 + pwd /tmp/tmphii43sg4 + mkdir -p /tmp/tmphii43sg4/BUILDROOT/opensearch-1.3.0-1.x86_64/var/run/opensearch + cp -a etc usr var /tmp/tmphii43sg4/BUILDROOT/opensearch-1.3.0-1.x86_64 + ln -s /var/lib/opensearch /tmp/tmphii43sg4/BUILDROOT/opensearch-1.3.0-1.x86_64/usr/share/opensearch/data + ln -s /var/log/opensearch /tmp/tmphii43sg4/BUILDROOT/opensearch-1.3.0-1.x86_64/usr/share/opensearch/logs + echo true + echo true + echo true + chmod -Rf a+rX,u+w,g-w,o-w /tmp/tmphii43sg4/BUILDROOT/opensearch-1.3.0-1.x86_64/etc /tmp/tmphii43sg4/BUILDROOT/opensearch-1.3.0-1.x86_64/usr /tmp/tmphii43sg4/BUILDROOT/opensearch-1.3.0-1.x86_64/var + exit 0 Processing files: opensearch-1.3.0-1.x86_64 Provides: opensearch = 1.3.0-1 opensearch(x86-64) = 1.3.0-1 Requires(interp): /bin/sh /bin/sh /bin/sh /bin/sh Requires(rpmlib): rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(CompressedFileNames) <= 3.0.4-1 Requires(pre): /bin/sh Requires(post): /bin/sh Requires(preun): /bin/sh Requires(postun): /bin/sh Checking for unpackaged file(s): /usr/lib/rpm/check-files /tmp/tmphii43sg4/BUILDROOT/opensearch-1.3.0-1.x86_64 Wrote: /tmp/tmphii43sg4/RPMS/x86_64/opensearch-1.3.0-1.x86_64.rpm Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.ef1m5x + umask 022 + cd /tmp/tmphii43sg4/BUILD + /usr/bin/rm -rf /tmp/tmphii43sg4/BUILDROOT/opensearch-1.3.0-1.x86_64 + exit 0 2022-03-09 02:28:29 INFO Published /home/opensearch-build-peterzhuamazon/dist/opensearch/opensearch-1.3.0-linux-x64.rpm. 2022-03-09 02:28:29 INFO Done. ```
peterzhuamazon commented 2 years ago

PR add rpm block in assemble workflows: https://github.com/opensearch-project/opensearch-build/pull/1726

peterzhuamazon commented 2 years ago

PR add rpm block in assemble workflows: https://github.com/opensearch-project/opensearch-build/pull/1726

peterzhuamazon commented 2 years ago

List of issues:

peterzhuamazon commented 2 years ago

20220324:

peterzhuamazon commented 2 years ago

20220324:

peterzhuamazon commented 2 years ago

OSD now support building from opensearch-build for rpm:

Logs: ``` $ ./assemble.sh builds/opensearch-dashboards/manifest.yml Installing dependencies in . ... Installing dependencies from Pipfile.lock (16e10a)... 🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00 To activate this project's virtualenv, run pipenv shell. Alternatively, run a command inside the virtualenv with pipenv run. Running ./src/run_assemble.py builds/opensearch-dashboards/manifest.yml ... 2022-03-29 22:45:06 INFO Bundling OpenSearch Dashboards (x64) on linux into <>/opensearch-build-peterzhuamazon/dist/opensearch-dashboards ... 2022-03-29 22:45:06 INFO Copied min bundle to /tmp/tmpi4vmv5ll/opensearch-dashboards-min-1.3.0-x64.rpm. 2022-03-29 22:45:06 INFO Start creating distribution rpm for OpenSearch-Dashboards. 2022-03-29 22:45:06 INFO Extracting dist into /tmp/tmpi4vmv5ll. 2022-03-29 22:45:06 INFO Convert rpm to cpio for extraction: /tmp/tmpi4vmv5ll/opensearch-dashboards-min-1.3.0-x64.rpm to /tmp/tmpi4vmv5ll/opensearch-dashboards-min-1.3.0-x64.cpio 2022-03-29 22:45:07 INFO Extract cpio /tmp/tmpi4vmv5ll/opensearch-dashboards-min-1.3.0-x64.cpio content to /tmp/tmpi4vmv5ll 2022-03-29 22:45:09 INFO Move /tmp/tmpi4vmv5ll/usr/share/opensearch-dashboards to /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 for plugin installation 2022-03-29 22:45:09 INFO Extracted dist into /tmp/tmpi4vmv5ll. 2022-03-29 22:45:09 INFO Executing "bash <>/opensearch-build-peterzhuamazon/scripts/components/OpenSearch-Dashboards/install.sh -v 1.3.0 -p linux -a x64 -d rpm -f <>/opensearch-build-peterzhuamazon/builds/opensearch-dashboards -o /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0" in /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 + getopts :h:v:s:o:p:a:d:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:s:o:p:a:d:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:s:o:p:a:d:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:s:o:p:a:d:f: arg + case $arg in + DISTRIBUTION=rpm + getopts :h:v:s:o:p:a:d:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:s:o:p:a:d:f: arg + case $arg in + OUTPUT=/tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 + getopts :h:v:s:o:p:a:d:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + '[' -z rpm ']' ++ dirname <>/opensearch-build-peterzhuamazon/scripts/components/OpenSearch-Dashboards/install.sh + DIR=<>/opensearch-build-peterzhuamazon/scripts/components/OpenSearch-Dashboards + echo <>/opensearch-build-peterzhuamazon/scripts/components/OpenSearch-Dashboards <>/opensearch-build-peterzhuamazon/scripts/components/OpenSearch-Dashboards + cd <>/opensearch-build-peterzhuamazon/scripts/components/OpenSearch-Dashboards + '[' rpm = tar ']' + '[' rpm = rpm ']' + cp ../../../config/opensearch_dashboards.yml /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0/../etc/opensearch-dashboards/ + cp -a ../../../scripts/pkg/service_templates/opensearch-dashboards/usr /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0/../ + cp -a ../../../scripts/pkg/build_templates/opensearch-dashboards/opensearch-dashboards.rpm.spec /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0/../ 2022-03-29 22:45:09 INFO Recording functionalTestDashboards 2022-03-29 22:45:09 INFO Installing queryWorkbenchDashboards 2022-03-29 22:45:09 INFO Executing "/tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0/bin/opensearch-dashboards-plugin --allow-root install file:/tmp/tmpi4vmv5ll/queryWorkbenchDashboards-1.3.0.zip" in /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 Attempting to transfer from file:/tmp/tmpi4vmv5ll/queryWorkbenchDashboards-1.3.0.zip Transferring 2631850 bytes.................... Transfer complete Retrieving metadata from plugin archive Extracting plugin archive Extraction complete Plugin installation complete 2022-03-29 22:45:09 INFO Executing "bash <>/opensearch-build-peterzhuamazon/scripts/default/install.sh -v 1.3.0 -p linux -a x64 -f <>/opensearch-build-peterzhuamazon/builds/opensearch-dashboards -o /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0" in /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + OUTPUT=/tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + exit 0 2022-03-29 22:45:09 INFO Installing observabilityDashboards 2022-03-29 22:45:09 INFO Executing "/tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0/bin/opensearch-dashboards-plugin --allow-root install file:/tmp/tmpi4vmv5ll/observabilityDashboards-1.3.0.zip" in /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 Attempting to transfer from file:/tmp/tmpi4vmv5ll/observabilityDashboards-1.3.0.zip Transferring 36259056 bytes.................... Transfer complete Retrieving metadata from plugin archive Extracting plugin archive Extraction complete Plugin installation complete 2022-03-29 22:45:13 INFO Executing "bash <>/opensearch-build-peterzhuamazon/scripts/default/install.sh -v 1.3.0 -p linux -a x64 -f <>/opensearch-build-peterzhuamazon/builds/opensearch-dashboards -o /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0" in /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + OUTPUT=/tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + exit 0 2022-03-29 22:45:13 INFO Installing securityDashboards 2022-03-29 22:45:13 INFO Executing "/tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0/bin/opensearch-dashboards-plugin --allow-root install file:/tmp/tmpi4vmv5ll/securityDashboards-1.3.0.zip" in /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 Attempting to transfer from file:/tmp/tmpi4vmv5ll/securityDashboards-1.3.0.zip Transferring 1105820 bytes.................... Transfer complete Retrieving metadata from plugin archive Extracting plugin archive Extraction complete Plugin installation complete 2022-03-29 22:45:14 INFO Executing "bash <>/opensearch-build-peterzhuamazon/scripts/default/install.sh -v 1.3.0 -p linux -a x64 -f <>/opensearch-build-peterzhuamazon/builds/opensearch-dashboards -o /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0" in /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + OUTPUT=/tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + exit 0 2022-03-29 22:45:14 INFO Installing anomalyDetectionDashboards 2022-03-29 22:45:14 INFO Executing "/tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0/bin/opensearch-dashboards-plugin --allow-root install file:/tmp/tmpi4vmv5ll/anomalyDetectionDashboards-1.3.0.zip" in /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 Attempting to transfer from file:/tmp/tmpi4vmv5ll/anomalyDetectionDashboards-1.3.0.zip Transferring 16680173 bytes.................... Transfer complete Retrieving metadata from plugin archive Extracting plugin archive Extraction complete Plugin installation complete 2022-03-29 22:45:16 INFO Executing "bash <>/opensearch-build-peterzhuamazon/scripts/default/install.sh -v 1.3.0 -p linux -a x64 -f <>/opensearch-build-peterzhuamazon/builds/opensearch-dashboards -o /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0" in /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + OUTPUT=/tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + exit 0 2022-03-29 22:45:16 INFO Installing reportsDashboards 2022-03-29 22:45:16 INFO Executing "/tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0/bin/opensearch-dashboards-plugin --allow-root install file:/tmp/tmpi4vmv5ll/reportsDashboards-1.3.0.zip" in /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 Attempting to transfer from file:/tmp/tmpi4vmv5ll/reportsDashboards-1.3.0.zip Transferring 69841365 bytes.................... Transfer complete Retrieving metadata from plugin archive Extracting plugin archive Extraction complete Plugin installation complete 2022-03-29 22:45:22 INFO Executing "bash <>/opensearch-build-peterzhuamazon/scripts/default/install.sh -v 1.3.0 -p linux -a x64 -f <>/opensearch-build-peterzhuamazon/builds/opensearch-dashboards -o /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0" in /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + OUTPUT=/tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + exit 0 2022-03-29 22:45:22 INFO Installing ganttChartDashboards 2022-03-29 22:45:22 INFO Executing "/tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0/bin/opensearch-dashboards-plugin --allow-root install file:/tmp/tmpi4vmv5ll/ganttChartDashboards-1.3.0.zip" in /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 Attempting to transfer from file:/tmp/tmpi4vmv5ll/ganttChartDashboards-1.3.0.zip Transferring 5646464 bytes.................... Transfer complete Retrieving metadata from plugin archive Extracting plugin archive Extraction complete Plugin installation complete 2022-03-29 22:45:23 INFO Executing "bash <>/opensearch-build-peterzhuamazon/scripts/default/install.sh -v 1.3.0 -p linux -a x64 -f <>/opensearch-build-peterzhuamazon/builds/opensearch-dashboards -o /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0" in /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + OUTPUT=/tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + exit 0 2022-03-29 22:45:23 INFO Installing indexManagementDashboards 2022-03-29 22:45:23 INFO Executing "/tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0/bin/opensearch-dashboards-plugin --allow-root install file:/tmp/tmpi4vmv5ll/indexManagementDashboards-1.3.0.zip" in /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 Attempting to transfer from file:/tmp/tmpi4vmv5ll/indexManagementDashboards-1.3.0.zip Transferring 602755 bytes.................... Transfer complete Retrieving metadata from plugin archive Extracting plugin archive Extraction complete Plugin installation complete 2022-03-29 22:45:23 INFO Executing "bash <>/opensearch-build-peterzhuamazon/scripts/default/install.sh -v 1.3.0 -p linux -a x64 -f <>/opensearch-build-peterzhuamazon/builds/opensearch-dashboards -o /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0" in /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + OUTPUT=/tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + exit 0 2022-03-29 22:45:23 INFO Installing alertingDashboards 2022-03-29 22:45:23 INFO Executing "/tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0/bin/opensearch-dashboards-plugin --allow-root install file:/tmp/tmpi4vmv5ll/alertingDashboards-1.3.0.zip" in /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 Attempting to transfer from file:/tmp/tmpi4vmv5ll/alertingDashboards-1.3.0.zip Transferring 7020130 bytes.................... Transfer complete Retrieving metadata from plugin archive Extracting plugin archive Extraction complete Plugin installation complete 2022-03-29 22:45:25 INFO Executing "bash <>/opensearch-build-peterzhuamazon/scripts/default/install.sh -v 1.3.0 -p linux -a x64 -f <>/opensearch-build-peterzhuamazon/builds/opensearch-dashboards -o /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0" in /tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + VERSION=1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + PLATFORM=linux + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + ARCHITECTURE=x64 + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + ARTIFACTS= + getopts :h:v:q:s:o:p:a:f: arg + case $arg in + OUTPUT=/tmp/tmpi4vmv5ll/opensearch-dashboards-1.3.0 + getopts :h:v:q:s:o:p:a:f: arg + '[' -z 1.3.0 ']' + '[' -z '' ']' + SNAPSHOT=false + '[' -z linux ']' + '[' -z x64 ']' + exit 0 2022-03-29 22:45:25 INFO Installed plugins: ['securityDashboards', 'reportsDashboards', 'alertingDashboards', 'ganttChartDashboards', 'anomalyDetectionDashboards', 'queryWorkbenchDashboards', 'observabilityDashboards', 'indexManagementDashboards'] 2022-03-29 22:45:25 INFO Organize folder structure before generating rpm 2022-03-29 22:45:25 INFO Execute rpmbuild -bb --define '_topdir /tmp/tmpi4vmv5ll' --define '_version 1.3.0' --define '_architecture x86_64' opensearch-dashboards.rpm.spec in /tmp/tmpi4vmv5ll Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.plFP0m + umask 022 + cd /tmp/tmpi4vmv5ll/BUILD + exit 0 Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.9vmTUi + umask 022 + cd /tmp/tmpi4vmv5ll/BUILD + exit 0 Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.XCUdPe + umask 022 + cd /tmp/tmpi4vmv5ll/BUILD + '[' /tmp/tmpi4vmv5ll/BUILDROOT/opensearch-dashboards-1.3.0-1.x86_64 '!=' / ']' + rm -rf /tmp/tmpi4vmv5ll/BUILDROOT/opensearch-dashboards-1.3.0-1.x86_64 ++ dirname /tmp/tmpi4vmv5ll/BUILDROOT/opensearch-dashboards-1.3.0-1.x86_64 + mkdir -p /tmp/tmpi4vmv5ll/BUILDROOT + mkdir /tmp/tmpi4vmv5ll/BUILDROOT/opensearch-dashboards-1.3.0-1.x86_64 + set -e + cd /tmp/tmpi4vmv5ll + pwd /tmp/tmpi4vmv5ll + mkdir -p /tmp/tmpi4vmv5ll/BUILDROOT/opensearch-dashboards-1.3.0-1.x86_64/var/run/opensearch-dashboards + mkdir -p /tmp/tmpi4vmv5ll/BUILDROOT/opensearch-dashboards-1.3.0-1.x86_64/usr/share/opensearch-dashboards/assets + mkdir -p /tmp/tmpi4vmv5ll/BUILDROOT/opensearch-dashboards-1.3.0-1.x86_64/usr/share/opensearch-dashboards/plugins + mkdir -p /tmp/tmpi4vmv5ll/BUILDROOT/opensearch-dashboards-1.3.0-1.x86_64/var/log/opensearch-dashboards + rm -rvf etc/systemd removed 'etc/systemd/system/opensearch-dashboards.service' removed directory: 'etc/systemd/system' removed directory: 'etc/systemd' + cp -a etc usr var /tmp/tmpi4vmv5ll/BUILDROOT/opensearch-dashboards-1.3.0-1.x86_64 + chmod 0755 /tmp/tmpi4vmv5ll/BUILDROOT/opensearch-dashboards-1.3.0-1.x86_64/usr/share/opensearch-dashboards/bin/opensearch-dashboards /tmp/tmpi4vmv5ll/BUILDROOT/opensearch-dashboards-1.3.0-1.x86_64/usr/share/opensearch-dashboards/bin/opensearch-dashboards-keystore /tmp/tmpi4vmv5ll/BUILDROOT/opensearch-dashboards-1.3.0-1.x86_64/usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin + ln -s /var/lib/opensearch-dashboards /tmp/tmpi4vmv5ll/BUILDROOT/opensearch-dashboards-1.3.0-1.x86_64/usr/share/opensearch-dashboards/data + ln -s /var/log/opensearch-dashboards /tmp/tmpi4vmv5ll/BUILDROOT/opensearch-dashboards-1.3.0-1.x86_64/usr/share/opensearch-dashboards/logs + chmod -Rf a+rX,u+w,g-w,o-w /tmp/tmpi4vmv5ll/BUILDROOT/opensearch-dashboards-1.3.0-1.x86_64/etc /tmp/tmpi4vmv5ll/BUILDROOT/opensearch-dashboards-1.3.0-1.x86_64/usr /tmp/tmpi4vmv5ll/BUILDROOT/opensearch-dashboards-1.3.0-1.x86_64/var + exit 0 Processing files: opensearch-dashboards-1.3.0-1.x86_64 Provides: opensearch-dashboards = 1.3.0-1 opensearch-dashboards(x86-64) = 1.3.0-1 Requires(interp): /bin/sh /bin/sh /bin/sh Requires(rpmlib): rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(CompressedFileNames) <= 3.0.4-1 Requires(pre): /bin/sh Requires(post): /bin/sh Requires(preun): /bin/sh Checking for unpackaged file(s): /usr/lib/rpm/check-files /tmp/tmpi4vmv5ll/BUILDROOT/opensearch-dashboards-1.3.0-1.x86_64 Wrote: /tmp/tmpi4vmv5ll/RPMS/x86_64/opensearch-dashboards-1.3.0-1.x86_64.rpm Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.d2I1hH + umask 022 + cd /tmp/tmpi4vmv5ll/BUILD + /usr/bin/rm -rf /tmp/tmpi4vmv5ll/BUILDROOT/opensearch-dashboards-1.3.0-1.x86_64 + exit 0 2022-03-29 22:46:39 INFO Published <>/opensearch-build-peterzhuamazon/dist/opensearch-dashboards/opensearch-dashboards-1.3.0-linux-x64.rpm. 2022-03-29 22:46:40 INFO Done. ```
peterzhuamazon commented 2 years ago

20220329:

peterzhuamazon commented 2 years ago

20220406:

20220414:

peterzhuamazon commented 2 years ago

New issue 20220407:

In 2.0 core branch we have this change from main, and it is not backported to 1.3 branch: https://github.com/opensearch-project/OpenSearch/pull/875

In 1.3 branch, due to above change to check source condition does not exist, we are forced to add these temp fix so that 1.3.x can install plugin zips correctly: https://github.com/opensearch-project/opensearch-build/blob/main/src/assemble_workflow/bundle_rpm.py#L66-L76 https://github.com/opensearch-project/opensearch-build/blob/main/src/assemble_workflow/bundle_rpm.py#L91-L94

In 2.0 because of this change, the above fix in 1.3.x is actually causing 2.0.x to not able to install plugins. The if block needs to have actual code inside, and since our temp fix comment out source, it cause syntax error in if block:

2022-04-08 02:10:11 INFO     Executing "/tmp/tmpmwqeotv_/opensearch-2.0.0-alpha1/bin/opensearch-plugin install --batch file:/tmp/tmpmwqeotv_/opensearch-job-scheduler-2.0.0.0-alpha1.zip" in /tmp/tmpmwqeotv_/opensearch-2.0.0-alpha1
/tmp/tmpmwqeotv_/opensearch-2.0.0-alpha1/bin/opensearch-env: line 98: syntax error near unexpected token `fi'
Traceback (most recent call last):
  File "./src/run_assemble.py", line 58, in <module>
    sys.exit(main())
  File "./src/run_assemble.py", line 44, in main
    bundle.install_components()
  File "/var/jenkins/workspace/distribution-build-opensearch@4/src/assemble_workflow/bundle.py", line 77, in install_components
    self.install_plugin(c)
  File "/var/jenkins/workspace/distribution-build-opensearch@4/src/assemble_workflow/bundle_opensearch.py", line 22, in install_plugin
    self._execute(f"{cli_path} install --batch file:{tmp_path}")
  File "/var/jenkins/workspace/distribution-build-opensearch@4/src/assemble_workflow/bundle.py", line 106, in _execute
    subprocess.check_call(command, cwd=self.min_dist.archive_path, shell=True)
  File "/usr/local/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '/tmp/tmpmwqeotv_/opensearch-2.0.0-alpha1/bin/opensearch-plugin install --batch file:/tmp/tmpmwqeotv_/opensearch-job-scheduler-2.0.0.0-alpha1.zip' returned non-zero exit status 2.

Proposed fix: Backport this change from 2.0 into 1.3 branch of OpenSearch core. When we release 1.3.x rpm we will be able to consume this change. And since this change is only for rpm not for tar in opensearch core, it wont affect the released tar/docker artifacts.

We can then remove the temp fix code in build repo for both 1.3.0 and 2.0.0 to success.

Thanks.

bbarani commented 2 years ago

@dblock @CEHENKLE @saratvemulapalli Do you have any suggestions for the above issue?

saratvemulapalli commented 2 years ago

@peterzhuamazon @bbarani I did take a look at the PR. From what I understand, it make the standard path configurable to run multiple OpenSearch instances on the same host.

I don't see any harm to have this backported to 1.3 line. As it was called out in the PR, lets make sure this is documented for 1.3 line (more specifically 1.3.2 where this change will be going into).

peterzhuamazon commented 2 years ago

@bbarani @tianleh @saratvemulapalli Backport this commit to 1.3 branch, means 1.3.2 now has this change. We have three options right now:

  1. If we still want to release 1.3.1 then we have to hardcode different options in src/assemble_workflow/bundle_rpm.py for 1.3.1 and non-1.3.1 versions. If 1.3.1, comment out source; if not, skip backup/comment.

  2. If we still want to release 1.3.1, but ok to create a temp branch such as 1.3.1-temp or 1.3.1-rpm in core repo, we can cherry pick the 1.3.1 tag and only backport this commit. We will also create a temp 1.3.1-temp or 1.3.1-rpm manifest file only for this branch. This is a one-time setup and wont happen later. We just need to remove the backup/comment code in src/assemble_workflow/bundle_rpm.py.

  3. If we are ok to release 2.0.0 first, then 1.3.2, completely ignore 1.3.1 rpm release then we can just backport the change to 1.3 core branch, and remove the backup/comment code in src/assemble_workflow/bundle_rpm.py.

Thanks.

bbarani commented 2 years ago

@tianleh @saratvemulapalli @dblock @CEHENKLE @anasalkouz Which option do you suggest to proceed further?

dblock commented 2 years ago

I vote for (3) because there are no hacks. Generally I am against trying to do an RPM release for something that has already shipped (e.g. 1.3.1). We should only be moving forward.

saratvemulapalli commented 2 years ago

+1 to @dblock. I like 3, as it looks clean. In terms of the product, did we commit to release 1.3.1 RPM?

bbarani commented 2 years ago

We have committed to releasing 1.3.x RPM but we didn't commit to a specific version. @CEHENKLE @elfisher @setiah Are you ok with delaying the RPM for 1.3.x release and publish it after 2.0 (RC1) release?

CEHENKLE commented 2 years ago

@bbarani can you comment on the forum thread point people here, asking for their input?

CEHENKLE commented 2 years ago

I vote for (3) because there are no hacks. Generally I am against trying to do an RPM release for something that has already shipped (e.g. 1.3.1). We should only be moving forward.

I'm leaning towards 1.3.1-temp branch. If we go this route, we should be able to release RPM in mid-April. If we wait for 1.3.2, then we can't release until 1.3.2 is released (which is ~30 days). This gives us a short lived branch which can be deleted after 1.3.2 goes out, and sits a lot better with me than hardcoding something in our build script.

(BTW, another option is to release 1.3.2 early, but depending on if they plugins have changes, I'd hate to add that to the work for 2.0)

elfisher commented 2 years ago

I'm leaning with 1.3.1-temp. Since this is the first release of RPM on OpenSearch, I think it is okay with it to be out of band from the regular process and waiting for 1.3.2 will be too long. Also To @CEHENKLE's point, if there was a way to do 1.3.2 early, that would also help with getting this out sooner, but I don't know how feasible that will be.

dblock commented 2 years ago

I'm leaning with 1.3.1-temp. Since this is the first release of RPM on OpenSearch, I think it is okay with it to be out of band from the regular process and waiting for 1.3.2 will be too long. Also To @CEHENKLE's point, if there was a way to do 1.3.2 early, that would also help with getting this out sooner, but I don't know how feasible that will be.

The issue is that 1.3.whatever is not going to be 1.3.1 binaries + RPM distribution, it will be built from source. Effectively creating 2 different 1.3.1s which I think is worse.

marcosfsch commented 2 years ago

Would it be possible to rename the WIP 1.3.2 to 1.3.3, and re-release 1.3.1 + RPM as 1.3.2?

justchris1 commented 2 years ago

I guess this is a first heard from me that there would even be a 1.3.X (where X is defined) release number of when an RPM would be released. I had seen in the 1.3.0 release notes that RPM was removed from 1.3.0 and no update was given as to the release of the feature. I also saw in the forums that there was no definitive release where RPMs would be available and no estimated date, but that when it was it determined, it would be updated. I did see some type of a commitment that there would be some 1.X release with RPM, and since there doesn't seem to be a 1.4 planned, I guess it makes some sense it would be 1.3.X. I can't find the release date for the 1.3.X series (other than the release 1.3.0) on https://opensearch.org/blog/partners/2022/02/roadmap-proposal/ or https://github.com/orgs/opensearch-project/projects/1 or in the announcements in the forums https://discuss.opendistrocommunity.dev/c/announcements/5, so I apologize if there is some post somewhere where I should be able to find this but can't. I surmise from the discussion that 1.3.1 is planned before 2.0.0RC1 (2022 April 26, per the github roadmap) and 1.3.2 some time after that, but no clear indication on when that might be. Per @elfisher, I further surmise it will be a while for 1.3.2.

While the software engineer in me yearns for clean and (3) appears to be the cleanest of the solutions, we live in an imperfect world. If we were still pre 1.0 GA and were trying to select which implementation to launch with, I think (3) fits the bill. However, while (3) gives the cleanest release code, it also perpetuates the problem of the missing RPMs for the entire duration we get to hold the moral high-ground of living without any less than perfect solutions. Of course, the irony is we do so by giving no solution to the end-users for the feature. I would urge an expedient as prudent release of the feature. Like most things in life, we learn by doing, and deferring this will likely lead to deferring the identification and resolution of other issues that will no doubt need to be addressed after this one. I +1 @elfisher's vote for (2), understanding it is less than ideal.

peterzhuamazon commented 2 years ago

@justchris1 https://github.com/orgs/opensearch-project/projects/13 This is the panel we use to track distributions.

Thanks.

dblock commented 2 years ago

Please don’t make 1.3.1-temp branches. By the time you’ve finished (2) nobody will remember what happened and it will be impossible to reproduce. Why are we making our lives harder with hacks upon hacks? Just release 1.3.2 which is 1.3.1 + rpm distribution, a new feature. There are code changes, it’s not 1.3.1. What’s the big deal to make that 1.3.2 on the same schedule and make 1.3.3 what we planned for 1.3.2?

If we had a security vulnerability we’d just make a 1.3.2 release now in automated orderly fashion, we do this all the time.

bbarani commented 2 years ago

@dblock Fast tracking the patch release schedule to add RPM is a hack to patch release cadence as well. We do schedule emergency patch releases for critical security vulnerabilities and high severity bug fixes. I understand that we are fixing bugs related to RPM generation here but does it warrant a new release? We are talking about releasing new version for both OpenSearch and OpenSearch Dashboards to support this change even though this code change is specific to OpenSearch only. Releases are expensive (From developer and also from consumer perspective) and I’m ok with fast tracking the patch release if the situation warrants it ( and there are no feasible workarounds)

justchris1 commented 2 years ago

@peterzhuamazon Thanks - I have been loosely tracking that. Do you happen to know if there is somewhere I can find information about planned release dates for 1.3.1, 1.3.2, or any 1.3.X releases? It seems like some people on the thread know of some of these dates (either notionally or with real plans), but I can't find them anywhere.

dblock commented 2 years ago

@dblock I understand that we are fixing bugs related to RPM generation here but does it warrant a new release?

Yes it does because it's not the same code.

justchris1 commented 2 years ago

What’s the big deal to make that 1.3.2 on the same schedule and make 1.3.3 what we planned for 1.3.2?

Are you suggesting that we have a simultaneous 1.3.1 and 1.3.2 release? I guess I could see that alleviating the need for some different paths like (1) or (2), but I was assuming that was not workable per some earlier comments. I wonder what the use of 1.3.1 would be if 1.3.2 was released simultaneously. It seemed that if we did (3), we would be further punting RPM support to an indeterminate date to the right again. (Not that RPM has any determinate release after being punted from 1.0, 1.1, 1.2, and 1.3 releases - not to mention any release plan per the release notes of 1.3).

dblock commented 2 years ago

I think the question illustrates the kind of confusion we have if we release RPM 1.3.1, 1.3.1 has already been released - https://opensearch.org/docs/latest/version-history/

I don't think we should punt anything, all I am saying is that we should release 1.3.2 TAR and RPM as soon as that's ready. That's the only logical path in my brain, I don't know how to grok "1.3.1 was released March 30th, then 1.3.1 RPM is released April 15th at the same time as taxes were due except when in 2020 there was an extra 15 days grace period" :)

justchris1 commented 2 years ago

I think the question illustrates the kind of confusion we have if we release RPM 1.3.1, 1.3.1 has already been released - https://opensearch.org/docs/latest/version-history/

Oh, I see. I can't use this project until there is real support from RPMs so I didn't even notice the 1.3.1 release.

I don't think we should punt anything, all I am saying is that we should release 1.3.2 TAR and RPM as soon as that's ready.

Yeah, I guess I can't argue with 'as soon as that's ready' except to note that the community desperately needs this feature, it is the largest missing element/feature that never made it over from Elasticsearch, and it is long promised but seems to be continuously deprioritized. As Kyle stated in the forums in November 2021 "RPM and DEB are super vital to serious use for a lot of folks", which I agree completely.

CEHENKLE commented 2 years ago

So just to frame the question here. There are three dimensions we can optimize against:

1) RPM is a critical project. The absence of RPM causes real pain. So we want to get it out as soon as possible. 2) We don't want to slam a hack into production that will cause us more pain down the line. 3) We firmly believe in progress, not perfection. If there's something that solves our needs, but could be better, let's get it out and improve it.

Reading the comments, @dblock it seems like you think the second option (a temp branch) violates number 2 above so much that it outweighs the benefits to 1 and 3. Other voices (mine included) seem to think the second option balances the three concerns the best. Do you have another option that we haven't considered that balances the three concerns better? Or do you still think waiting for 1.3.2 (which optimizes for 2, but is less effective on 1 and 3) is the best option?

CEHENKLE commented 2 years ago

(and just a side note: Making 1.3.1 available means making the current version of the 1.x line available in RPM, rather than "re-releasing". Will they always be available on the same day going forward when we release? yes. Should we wait to make that true starting now? I don't think so, but that's how my brain squares that circle, dB).

stockholmux commented 2 years ago

Just catching up here.

Making a 1.3.2 because it has RPM is violation of semver as you're treating it like a feature. If you don't rev the version, then it's not a feature, just like if we added a new build to support RISC-V or something. Patches can only fix bugs - this isn't a bug.

CEHENKLE commented 2 years ago

Hey folks. -- I've slept on it, and we'll make the first RPM 2.0-RC1, and then follow up with a release on 1.3.2. I hate the delay viscerally, but I can't justify the hackery/extra churn on the EE team for the time saved.

Thanks. /C

justchris1 commented 2 years ago

Hey folks. -- I've slept on it, and we'll make the first RPM 2.0-RC1, and then follow up with a release on 1.3.2. I hate the delay viscerally, but I can't justify the hackery/extra churn on the EE team for the time saved.

Thanks. /C

Don't worry. No one will be too surprised.

dblock commented 2 years ago

Just catching up here.

Making a 1.3.2 because it has RPM is violation of semver as you're treating it like a feature. If you don't rev the version, then it's not a feature, just like if we added a new build to support RISC-V or something. Patches can only fix bugs - this isn't a bug.

So should we do a 1.4.0 then? I don't mean to throw us in a loop, ... but.

Isn't adding RPM in 1.3.1 the same problem as in 1.3.2? So should it be 1.4.0?

peterzhuamazon commented 2 years ago

Two backports:

Remove Hacks:

setiah commented 2 years ago

Two backports:

Remove Hacks:

Looks like we finally backported the PR that was the source of contention to 1.3. Does this imply the plan is to release 1.3.1 version rpm along with the 2.0.0 rc1 rpm?

peterzhuamazon commented 2 years ago

Hi @setiah option 3 is what we are going with: https://github.com/opensearch-project/opensearch-build/issues/1545#issuecomment-1099632988

Thanks.

stockholmux commented 2 years ago

@dblock So, IIRC, the original intention of having the distributions tracked away from numbered versions in the roadmap is that they would not have an effect on the actual logic/features of OpenSearch and could be released out of phase with any particular version. That's fine. No problem with SemVer. Use build metadata that is outside the scope of major.minor.patch.

As you're suggesting revving the patch to accommodate this fix doesn't fit the word nor the spirit of SemVer (PATCH version when you make backwards compatible bug fixes ... not this situation at all). Revving the minor doesn't really fit either (MINOR version when you add functionality in a backwards compatible manner ... not really either). I'm suggesting that the project use the build metadata appropriately as per spec. For someone who doesn't care about RPM this version is meaningless. Organizations have policies they have to follow as far as adopting the latest patch, so this is creating waste heat for those users.

I'm uncomfortable with the idea of bringing builds back into the numbered versioning scheme yet tracking them in a different place - it's really violating the trust users have in our roadmap and our versioning. All the (rightful) hubbub about RPM being in 1.3.0 roadmap is also meaningless since it was made clear that the distributions shouldn't have been in the numbered version roadmap, and now the project is attaching RPM to a patch version?

Thoughts @elfisher @CEHENKLE ?

dblock commented 2 years ago

Oh, I see. I think 1.3.2 should also get some low hanging CVEs.

setiah commented 2 years ago

@dblock So, IIRC, the original intention of having the distributions tracked away from numbered versions in the roadmap is that they would not have an effect on the actual logic/features of OpenSearch and could be released out of phase with any particular version. That's fine. No problem with SemVer. Use build metadata that is outside the scope of major.minor.patch.

As you're suggesting revving the patch to accommodate this fix doesn't fit the word nor the spirit of SemVer (PATCH version when you make backwards compatible bug fixes ... not this situation at all). Revving the minor doesn't really fit either (MINOR version when you add functionality in a backwards compatible manner ... not really either). I'm suggesting that the project use the build metadata appropriately as per spec. For someone who doesn't care about RPM this version is meaningless. Organizations have policies they have to follow as far as adopting the latest patch, so this is creating waste heat for those users.

I'm uncomfortable with the idea of bringing builds back into the numbered versioning scheme yet tracking them in a different place - it's really violating the trust users have in our roadmap and our versioning. All the (rightful) hubbub about RPM being in 1.3.0 roadmap is also meaningless since it was made clear that the distributions shouldn't have been in the numbered version roadmap, and now the project is attaching RPM to a patch version?

Thoughts @elfisher @CEHENKLE ?

+1, I agree the distribution release process should remain independent of the version release process. This helps with not just the RPM but other distributions such as Windows, Debian as well in future. Else, we could be falling prey to such issues down the line again while releasing other distributions.

dblock commented 2 years ago

@setiah except it's not separate in practice, the rpm process is implemented in a tightly coupled way to the tar release process

peterzhuamazon commented 2 years ago

Python code can now generate RPMs for both OS and OSD on every version pass 1.3.0.

peterzhuamazon commented 2 years ago

20220429: Remove PA not properly stopped before install/uninstall