Running bleve extensions outside docker on Mac and windows

[ghost]

Is anyone working on this aspect ?

I want to run blast as a standard system service rather than inside docker. And I need to run on Mac and Windows. So that brings up the bleve cgo based extensions.

[mosuka]

Hi @gedw99

Unfortunately, I don't have a Windows platform, but I can describe you how to build Blast with enabling Bleve Extensions on macOS. Please try the following steps:

1. Build and install dependencies

$ brew install \
    icu4c \
    leveldb \
    rocksdb \
    zstd

$ CGO_LDFLAGS="-L/usr/local/opt/icu4c/lib" \
    CGO_CFLAGS="-I/usr/local/opt/icu4c/include" \
    go get -u -v github.com/blevesearch/cld2
$ cd ${GOPATH}/src/github.com/blevesearch/cld2
$ git clone https://github.com/CLD2Owners/cld2.git
$ cd cld2/internal
$ perl -p -i -e 's/soname=/install_name,/' compile_libs.sh
$ ./compile_libs.sh
$ sudo cp *.so /usr/local/lib

2. Build Blast with enabling Bleve Extensions on macOS

$ git clone git@github.com:mosuka/blast.git
$ cd blast
$ GOOS=darwin \
    CGO_LDFLAGS="-L/usr/local/opt/icu4c/lib -L/usr/local/opt/rocksdb/lib -lrocksdb -lstdc++ -lm -lz -lbz2 -lsnappy -llz4 -lzstd" \
    CGO_CFLAGS="-I/usr/local/opt/icu4c/include -I/usr/local/opt/rocksdb/include" \
    CGO_ENABLED=1 \
    BUILD_TAGS="full" \
    make build

[ghost]

thanks @mosuka

wow that perl in there. Have not seen that in a while :)

Ok so the only way to pull this off is goin to be to make amage script so that all this stuff can be scripted in golang and then run on Windows, Mac and linux locally and in CI.

are you ok with that in your repo though ? It solves a ton is issue with cross platform stuff.

here is a great example.

github.com/elastic/beats

Basically make and make.bat both check for mage and then its all golang from there on in. its very powerful. the exact same script can be called in CI too.

anyway let me know your thoughts as getting stuff like this working and handling regressions will be hell without a proper setup like this.

[mosuka]

Hi @gedw99, Thank you for your proposal. It is a great idea.

Above install steps is based on following document: https://blevesearch.com/docs/Building/

I will consider to using mage for cross platform building.

[ghost]

Ok. I will work on a mage script. I can see with does is not on the blevex building web page you mentioned, so will try my best to get it working.

I am pretty busy with things so leave this open as it's going to take a while. Also it's best done in stages

[ghost]

hey @mosuka Sorry about late late reply.

I think that its not possible to run on a desktop on Windows. Maybe it will require just doing a version without any none golang dependencies. SO that brings the question of doing some tagging for part of the code tagged as "nodeps" maybe. Is this something just way too had because there is CGO stuff in the code everywhere ? I know it will mean reduced functionality. Maybe not handle different written languages properly ? I see libstemmer is needed for this ? https://github.com/mosuka/blast/commit/35bb08909f929a8c44ff6c12176e3616ea81e103

Maybe other things too ?

If its just NUTS, just let me know :)

[ghost]

Did a compromise.

export CGO_ENABLED=0 export GOOS=darwin make build

works, but i guess i will not have certain functionality.

[mosuka]

I think Windows platform is difficult to use plugins required CGO_ENABLED=1. I agree with you.

Build for Windows platform will be like following: go build -tags="kagome cznicb badger"