search

opensearch-project / opensearch-build

🧰 OpenSearch / OpenSearch-Dashboards Build Systems
Apache License 2.0
141 stars 275 forks source link

k-NN library x64 build should use generic march #481

Closed jmazanec15 closed 3 years ago

jmazanec15 commented 3 years ago

In ODFE, we had run into issues with library incompatibilities due to the k-NN library being compiled on a machine with support for SIMD instruction extensions and being run on a machine without support for SIMD instruction extensions. We added a small fix in ODFE CD to resolve this.

ARM is not impacted.

We should have a similar fix in OpenSearch.

peterzhuamazon commented 3 years ago

Jack has provided the libs we can manually replace it. We can also try to run it on centos docker, if this PR merged very soon #588

peterzhuamazon commented 3 years ago

Issues coming as CentOS7 does not like -march=native during the build: https://gcc.gnu.org/onlinedocs/gcc/AArch64-Options.html

Native Error Log on CentOS7 make

``` + cmake . -- The C compiler identification is GNU 4.8.5 -- The CXX compiler identification is GNU 4.8.5 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Build type: Release -- Header files: /tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/cmd_options.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/distcomp.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/distcomp_edist.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/eval_metrics.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/eval_results.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/experimentconf.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/experiments.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/falconn_heap_mod.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/flexbuff.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/floatdiff.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/global.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/gold_standard.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/idtype.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/incremental_quick_select.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/index.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/init.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/inmem_inv_index.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/knnquery.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/knnqueue.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/logging.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/memory.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/meta_analysis.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/methodfactory.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/my_isnan_isinf.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/object.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/params.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/params_cmdline.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/params_def.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/permutation_type.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/permutation_utils.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/portable_align.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/portable_intrinsics.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/portable_popcount.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/portable_prefetch.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/portable_simd.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/ported_boost_progress.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/pow.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/query.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/query_creator.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/rangequery.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/read_data.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/report.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/report_intr_dim.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/searchoracle.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/simddebug.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/sort_arr_bi.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/spacefactory.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/thread_pool.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/tune.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/utils.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/ztimer.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/dummy.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/hnsw.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/hnsw_distfunc_opt_impl_inline.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/permutation_prefix_index.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/pivot_neighb_common.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/pivot_neighb_invindx.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/seqsearch.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/simple_inverted_index.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/small_world_rand.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/vptree.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/vptree_utils.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_ab_diverg.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_bit_hamming.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_bit_jaccard.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_bit_vector.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_bregman.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_dummy.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_js.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_l2sqr_sift.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_leven.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_lp.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_rank_correl.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_renyi_diverg.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_scalar.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_bin_common.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_dense_fusion.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_jaccard.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_lp.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_scalar.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_scalar_bin_fast.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_scalar_fast.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_vector.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_vector_inter.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_sqfd.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_string.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_vector.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_vector_gen.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_word_embed.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/factory/init_methods.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/factory/init_spaces.h -- Source files: /tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_bregman.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_diverg.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_edist.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_js.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_l2sqr_sift.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_lp.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_overlap.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_rankcorr.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_scalar.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_sparse_scalar_fast.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/experimentconf.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/global.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/init.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/knnquery.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/logging.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/memory.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/params.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/params_cmdline.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/query.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/rangequery.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/searchoracle.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/utils.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_ab_diverg.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_bregman.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_dummy.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_js.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_l2sqr_sift.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_lp.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_renyi_diverg.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_scalar.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_sparse_dense_fusion.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_sparse_jaccard.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_sparse_lp.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_sparse_scalar_bin_fast.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_sparse_scalar_fast.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_sparse_vector.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_sparse_vector_inter.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_string.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_vector.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_word_embed.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/method/dummy.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/method/hnsw.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/method/hnsw_distfunc_opt.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/method/pivot_neighb_invindx.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/method/seqsearch.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/method/simple_inverted_index.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/method/small_world_rand.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/method/vptree.cc -- Looking for include file pthread.h -- Looking for include file pthread.h - found -- Looking for pthread_create -- Looking for pthread_create - not found -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE -- Found Threads. -- OS: Linux -- Compiler: GNU 4.8.5 -- CXX flags release: -Wall -Wunreachable-code -Wcast-align -Ofast -lm -DNDEBUG -std=c++11 -pthread -DHAVE_CXX0X -march=native -Wl,--no-as-needed -fpic -- CXX flags debug: -Wall -Wunreachable-code -Wcast-align -ggdb -lm -DNDEBUG -std=c++11 -pthread -DHAVE_CXX0X -march=native -Wl,--no-as-needed -fpic -- Configuring done -- Generating done -- Build files have been written to: /tmp/tmp_8_16mqn/k-NN/jni + make Scanning dependencies of target NonMetricSpaceLib [ 2%] Building CXX object external/nmslib/similarity_search/src/CMakeFiles/NonMetricSpaceLib.dir/distcomp_bregman.cc.o /tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_bregman.cc:1:0: error: unknown value 'native' for -march /** ^ make[2]: *** [external/nmslib/similarity_search/src/CMakeFiles/NonMetricSpaceLib.dir/distcomp_bregman.cc.o] Error 1 make[1]: *** [external/nmslib/similarity_search/src/CMakeFiles/NonMetricSpaceLib.dir/all] Error 2 make: *** [all] Error 2 ```

We have resolved this by changing native to armv8-a as native is only introduced in CentOS8: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/conside[…]lers-and-development-tools_considerations-in-adopting-rhel-8

After changing to armv8-a the build success: knnlib centos7 armv8-a.txt

@jmazanec15 will raise a PR to fix this in the build workflows.

Thanks.

peterzhuamazon commented 3 years ago

Close this as it is working now in build workflow.