search

ibmdb / go_ibm_db

GoLang Driver for Db2 family of database servers
BSD 3-Clause "New" or "Revised" License
107 stars 37 forks source link

How to use this in a makefile for CI/CD (sqlcli1.h: No such file or directory) #238

Open ZacZelner opened 4 months ago

ZacZelner commented 4 months ago

Hi, we are using this for years, and recently got a issue when we build a auto CI/CD pipeline. We are using a makefile to do the auto CI/CD, and code are:

ifeq (,$(shell go env GOBIN)) GOBIN=$(shell go env GOPATH)/bin else GOBIN=$(shell go env GOBIN) endif CGO_CFLAGS=-I$DB2HOME/include CGO_LDFLAGS=-L$DB2HOME/lib LD_LIBRARY_PATH=$DB2HOME/lib

install-runtime: GO111MODULE=on go install github.com/tools/godep@latest GO111MODULE=on go install github.com/ibmdb/go_ibm_db/installer@v0.4.1

local_setup:

sh ./deployment/local/setup.sh

go run /go/pkg/mod/github.com/ibmdb/go_ibm_db@v0.4.1/installer/setup.go

CIO CI CD

.PHONY: unit-test unit-test: go test tests/wc_test.go

.PHONY: lint lint: @echo '==> Linting...' go fmt go vet

However, when it runs to go vet, it pop up the error: vendor/github.com/ibmdb/go_ibm_db/api/api_unix.go:12:11: fatal error: sqlcli1.h: No such file or directory // #include

From our experience, it means the env setting is not correct.

We have already claim the env in the beginning, however it does not work.

Anyone know how to build this go_ibm_db in a makefile?

Thanks!

vmathur12 commented 4 months ago

@ZacZelner This error occurs when environment variables are not set properly or when some files are missing. Could you please share the output of the following commands: echo $IBM_DB_HOME go env ls $IBM_DB_HOME/include ls $IBM_DB_HOME/lib uname -a Thanks

bimalkjha commented 4 months ago

@ZacZelner Any update? Thanks.

ZacZelner commented 4 months ago

Thanks. @bimalkjha @vmathur12 .

It seems there is no $IBM_DB_HOME env in the container when the makefile build.

Part of our makefile:

Development environments

-----------------------------------------------------------------------------

install-runtime: GO111MODULE=on go install github.com/tools/godep@latest GO111MODULE=on go install github.com/ibmdb/go_ibm_db/installer@v0.4.1

local_setup:

sh ./deployment/local/setup.sh

go run /go/pkg/mod/github.com/ibmdb/go_ibm_db@v0.4.1/installer/setup.go

.PHONY: unit-test unit-test: go test tests/wc_test.go

.PHONY: lint lint: @echo '==> IBM_DB_HOME env...' @echo IBM_DB_HOME go env @env @echo '==> Linting...' go fmt go vet

I print all the env: go env GO111MODULE="" GOARCH="amd64" GOBIN="" GOCACHE="/opt/app-root/src/.cache/go-build" GOENV="/opt/app-root/src/.config/go/env" GOEXE="" GOEXPERIMENT="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOINSECURE="" GOMODCACHE="/workspace/go-mod-cache" GONOPROXY="" GONOSUMDB="" GOOS="linux" GOPATH="/opt/app-root/src/.go" GOPRIVATE="" GOPROXY="https://proxy.golang.org,direct" GOROOT="/usr/lib/golang" GOSUMDB="sum.golang.org" GOTMPDIR="" GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64" GOVCS="" GOVERSION="go1.20.12" GCCGO="gccgo" GOAMD64="v1" AR="ar" CC="gcc" CXX="g++" CGO_ENABLED="1" GOMOD="/workspace/cloned-git-repository/go.mod" GOWORK="" CGO_CFLAGS="-O2 -g" CGO_CPPFLAGS="" CGO_CXXFLAGS="-O2 -g" CGO_FFLAGS="-O2 -g" CGO_LDFLAGS="-O2 -g" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2109435798=/tmp/go-build -gno-record-gcc-switches" KUBERNETES_PORT_443_TCP_PORT=443 MAKEFILESCRIPT=lint SUMMARY=Platform for building and running Go Applications =/usr/bin/make VERSION=1.20.12 CONTAINER_NAME=rhel8/go-toolset NSS_SDB_USE_CACHE=no PATH=/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/app-root/src/.go/bin STI_SCRIPTS_PATH=/usr/libexec/s2i STI_SCRIPTS_URL=image:///usr/libexec/s2i GOPATH=/opt/app-root/src/.go KUBERNETES_PORT_443_TCP=tcp://172.21.0.1:443 container=oci PWD=/workspace/cloned-git-repository KUBERNETES_PORT=tcp://172.21.0.1:443 HOME=/opt/app-root/src HOSTNAME=cdm-acq-ocv-dxmjc-lint-pod GO_MAJOR_VERSION=1 SHLVL=1 NAME=golang MAKEFLAGS= MFLAGS= KUBERNETES_SERVICE_PORT_HTTPS=443 GO_PATCH_VERSION=12 PLATFORM=el8 KUBERNETES_SERVICE_HOST=172.21.0.1 APP_ROOT=/opt/app-root KUBERNETES_SERVICE_PORT=443 BUILD_VERSION= SSL_CERT_DIR=/tekton-custom-certs:/etc/ssl/certs:/etc/pki/tls/certs:/system/etc/security/cacerts KUBERNETES_PORT_443_TCP_ADDR=172.21.0.1 NODEJS_VER=20 GO_MINOR_VERSION=20 GO_MOD_CACHE_PATH=/workspace/go-mod-cache TERM=xterm DESCRIPTION=Go Toolset available as a container is a base platform for building and running various Go applications and frameworks. Go is an easy to learn, powerful, statically typed language in the C/C++ tradition with garbage collection, concurrent programming support, and memory safety features. KUBERNETES_PORT_443_TCP_PROTO=tcp MAKELEVEL=1

It works well in our dockerfile before. But not work well in Makefile.

Our CI/CD developer said: I understand that to use Db2 in a Go project you need to perform some extra steps like this. Like setting environment variables and running a setup script. Each "step" in the pipeline is executed in its own container. Meaning that you lose the file and environment variables contents except for locations that are explicitly specified in the pipeline. Such as the location where dependencies are installed, source code is, and executable is output. And not all volumes are carried over to every container task. For example, the build (compile) step needs the source code available, but the docker image build step does not. So the volume where the source code is (from the git clone step) is made available to the compile/build task, but not the docker image build task. All of that is to say, any environment variables that you are created in one task, are likely getting lost in the next task. And almost certainly not carried over to the application runtime. Same goes for whatever "setup.go" does.

ZacZelner commented 3 months ago

@bimalkjha @vmathur12 any update for this?

ZacZelner commented 3 months ago

Or how can we skip downloading go install github.com/ibmdb/go_ibm_db/installer, but packed all related folder (like github.com/ibmdb/go_ibm_db/installer/clidriver) directly into image and make your tool work well? @bimalkjha @vmathur12

bimalkjha commented 3 months ago

@ZacZelner Please check install instructions for go_ibm_db in this doc: https://github.com/ibmdb/go_ibm_db?tab=readme-ov-file#how-to-install-in-linuxmac

If you already have a db2client like dsdriver or dsclient or db2server installed in the system, you can set IBM_DB_HOME env var pointing to the install directory of db2client and then install go driver. Otherwise, you need to run setup.go command so that setup.go can pull clidriver and install in the system for use by go. sqlcli1.h file comes with db2clients under include directory. The clidriver that get pulled by setup.go also as include directory with sqlcli1.h file. Then you need to set the CGO flags like in the install doc.

I would suggest to update your CI/CD pipeline script to pull https://public.dhe.ibm.com/ibmdl/export/pub/software/data/db2/drivers/odbc_cli/linuxx64_odbc_cli.tar.gz , untar it and get full path of clidriver directory. Then set IBM_DB_HOME=full path of untared clidriver. Then set below env vars: CGO_CFLAGS=-I$IBM_DB_HOME/include CGO_LDFLAGS=-L$IBM_DB_HOME/lib LD_LIBRARY_PATH=$IBM_DB_HOME/lib

Replace DB2HOME with IBM_DB_HOME in your CI/CD pipeline script. Now, run your go program, it should work. Please try and let us know all steps that you tried and location of clidriver. Share output of below commands from CI/CD pipeline if go script fails:

db2level
echo $IBM_DB_HOME
ls $IBM_DB_HOME
echo $CGO_CFLAGS
echo $CGO_LDFLAGS
echo $LD_LIBRARY_PATH
echo $PATH

Thanks.

ZacZelner commented 2 months ago

@bimalkjha

Thanks! We tried some solution like you said. However, due to our user access, we just can put all these clidriver into vendor folder, and set: CGO_CFLAGS="-I/workspace/cloned-git-repository/vendor/github.com/ibmdb/go_ibm_db/installer/clidriver//include" etc.

Our application source code folder is /workspace/cloned-git-repository.

When we build the go application, it still has this error:

Building go application using version: go version go1.20.12 linux/amd64 Setting custom GOMODCACHE to /workspace/go-mod-cache

github.com/ibmdb/go_ibm_db/api

vendor/github.com/ibmdb/go_ibm_db/api/api_unix.go:12:11: fatal error: sqlcli1.h: No such file or directory // #include ^~~ compilation terminated.

the $PATH is:/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/app-root/src/.go/bin

Our question is: can the support clidriver folder be put in vendor folder and work well? If not, should it must be installed in the parent root folder?