search

goccy / bigquery-emulator

BigQuery emulator server implemented in Go
MIT License
743 stars 86 forks source link

How to install and run this project locally on my computer? #285

Closed j-antunes closed 1 month ago

j-antunes commented 2 months ago

What happened?

I found this project and was interested in trying it out for another project. I was able to get the docker container up and running and doing GET and POST request to the docker container works without any issue.

However, I found a bug with uploading csv files, and wanted to run the project locally to fix it. However, when I followed the README and ran this cmd: go install ./cmd/bigquery-emulator I got the following error:

❯ go install ./cmd/bigquery-emulator

# github.com/goccy/go-zetasql/internal/ccall/go-absl/time/go_internal/cctz/time_zone
In file included from bind.cc:97:
In file included from ../../go/pkg/mod/github.com/goccy/go-zetasql@v0.5.4/internal/ccall/absl/time/internal/cctz/include/cctz/time_zone.h:23:
In file included from /usr/lib/llvm-14/bin/../include/c++/v1/chrono:697:
In file included from /usr/lib/llvm-14/bin/../include/c++/v1/__chrono/calendar.h:13:
In file included from /usr/lib/llvm-14/bin/../include/c++/v1/__chrono/duration.h:14:
In file included from /usr/lib/llvm-14/bin/../include/c++/v1/limits:105:
In file included from /usr/lib/llvm-14/bin/../include/c++/v1/type_traits:420:
/usr/lib/llvm-14/bin/../include/c++/v1/cstddef:44:15: fatal error: 'stddef.h' file not found

What are the necessary libraries or steps that I need to do, to be able to successfully install and run the project locally? Thanks!

OS: Ubuntu 22.04

What did you expect to happen?

Be able to install the project and run it on my laptop, without needing a docker container

How can we reproduce it (as minimally and precisely as possible)?

Get a fresh install of Ubuntu 22.04, and try to install this project following the readme

Anything else we need to know?

No response

totem3 commented 2 months ago

Thank you for your interest in the project.

In the "What happened?" section, the OS version is mentioned as 22.04, but in the "steps to reproduce" section, it says 24.04. 22.04 is correct?

I usually build the bigquery-emulator on a VM (ubuntu 22.04) in GCP and have never encountered any issues. The build process I follow is as follows:

Given the error mentions a missing header file, it seems there might be an issue with the clang installation?

FYI: this is install steps and log. https://gist.github.com/totem3/fab8b764cafd443b9244f18513a9b566

j-antunes commented 2 months ago

@totem3 - Thank you for taking the time to answer my question, and for compiling such detailed instructions! To answer your question, everything was run on a 22.04 image. Sorry for the typo I'll edit it.

I tried to use a VM on GCP, and I think it was working, the VM that I had was just really slow, so I think it was just taking to long to compile the zeta library. I noticed that the default image in GCP is debian-12-bookworm. I'm using Ubuntu 22.04, I wonder if there is a different between them.

I followed your instructions and I was not able to get it running on my personal machine. This could just be that my machine does not have the necessary packages to be able to compile correctly. What I did next was to boot a VM on my machine. For this VM I used the ISO Ubuntu 22.04.3 - Server Image. Downloaded from here https://ubuntu.com/download/server

With that installed, then I followed your steps here and from the gist: Update image sudo apt update; sudo apt upgrade Install clang using the "Automatic installation script" from https://apt.llvm.org/ 

sudo bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"

Install packages

sudo apt-get install clang-format clang-tidy clang-tools clang clangd libc++-dev libc++1 libc++abi-dev libc++abi1 libclang-dev libclang1 liblldb-dev libllvm-ocaml-dev libomp-dev libomp5 lld lldb llvm-dev llvm-runtime llvm python3-clang

Install go 1.20.5

cd /tmp/
wget https://go.dev/dl/go1.21.5.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.21.5.linux-amd64.tar.gz

Clone bigquery repo

cd
git clone https://github.com/goccy/bigquery-emulator
cd bigquery-emulator/

Compile project

export PATH=$PATH:/usr/local/go/bin
CGO_ENABLED=1 CC=clang CXX=clang++ go install ./cmd/bigquery-emulator

I was not able to get it running. For me the project just stopped at this line in your gist https://gist.github.com/totem3/fab8b764cafd443b9244f18513a9b566#file-log-L709 I can see from the computer resources that it is doing something, but I'm not sure what. Then it just ends there. If I try to run ./bigquery-emulator --project test it just says "No such file or directory"

Here is the log when I try to run go install

go install ./cmd/bigquery-emulator/
go: downloading github.com/jessevdk/go-flags v1.5.0
go: downloading cloud.google.com/go/bigquery v1.53.0
go: downloading cloud.google.com/go/storage v1.30.1
go: downloading github.com/apache/arrow/go/v10 v10.0.1
go: downloading cloud.google.com/go v0.110.7
go: downloading github.com/go-playground/validator/v10 v10.11.0
go: downloading github.com/goccy/go-json v0.10.0
go: downloading github.com/goccy/go-yaml v1.9.5
go: downloading github.com/goccy/go-zetasqlite v0.19.1
go: downloading github.com/gorilla/mux v1.8.0
go: downloading github.com/linkedin/goavro/v2 v2.12.0
go: downloading github.com/segmentio/parquet-go v0.0.0-20221020201645-63215c8128ff
go: downloading go.uber.org/zap v1.21.0
go: downloading golang.org/x/sync v0.3.0
go: downloading google.golang.org/api v0.126.0
go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20230822172742-b8732ec3820d
go: downloading google.golang.org/genproto v0.0.0-20230822172742-b8732ec3820d
go: downloading google.golang.org/grpc v1.59.0
go: downloading google.golang.org/protobuf v1.31.0
go: downloading github.com/goccy/go-zetasql v0.5.5
go: downloading golang.org/x/sys v0.18.0
go: downloading cloud.google.com/go/compute/metadata v0.2.3
go: downloading cloud.google.com/go/iam v1.1.1
go: downloading cloud.google.com/go/compute v1.23.0
go: downloading github.com/google/uuid v1.3.1
go: downloading github.com/googleapis/gax-go/v2 v2.11.0
go: downloading golang.org/x/oauth2 v0.11.0
go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20230822172742-b8732ec3820d
go: downloading golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2
go: downloading github.com/google/flatbuffers v2.0.8+incompatible
go: downloading github.com/klauspost/compress v1.15.9
go: downloading github.com/pierrec/lz4/v4 v4.1.15
go: downloading github.com/mattn/go-sqlite3 v1.14.16
go: downloading github.com/go-playground/universal-translator v0.18.0
go: downloading github.com/leodido/go-urn v1.2.1
go: downloading golang.org/x/crypto v0.21.0
go: downloading golang.org/x/text v0.14.0
go: downloading github.com/golang/snappy v0.0.4
go: downloading go.uber.org/atomic v1.7.0
go: downloading go.uber.org/multierr v1.6.0
go: downloading github.com/olekukonko/tablewriter v0.0.5
go: downloading github.com/segmentio/encoding v0.3.5
go: downloading golang.org/x/net v0.21.0
go: downloading github.com/apache/arrow/go/v12 v12.0.0
go: downloading go.opencensus.io v0.24.0
go: downloading github.com/zeebo/xxh3 v1.0.2
go: downloading github.com/fatih/color v1.10.0
go: downloading github.com/DataDog/go-hll v1.0.2
go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13
go: downloading github.com/dop251/goja v0.0.0-20221118162653-d4bf6fde1b86
go: downloading github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72
go: downloading gonum.org/v1/gonum v0.11.0
go: downloading github.com/go-playground/locales v0.14.0
go: downloading github.com/mattn/go-runewidth v0.0.9
go: downloading github.com/andybalholm/brotli v1.0.4
go: downloading github.com/golang/protobuf v1.5.3
go: downloading github.com/google/s2a-go v0.1.4
go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da
go: downloading github.com/googleapis/enterprise-certificate-proxy v0.2.3
go: downloading github.com/klauspost/cpuid/v2 v2.0.9
go: downloading github.com/mattn/go-colorable v0.1.12
go: downloading github.com/mattn/go-isatty v0.0.17
go: downloading github.com/pkg/errors v0.9.1
go: downloading github.com/dlclark/regexp2 v1.7.0
go: downloading github.com/apache/thrift v0.16.0
go: downloading github.com/go-sourcemap/sourcemap v2.1.3+incompatible
go: downloading google.golang.org/appengine v1.6.7
jantunes@bq:~/bigquery-emulator$
totem3 commented 1 month ago

@j-antunes ah... sorry that was my mistake. After go install, bigquery-emulator binary normally will be installed in the $GOPATH/bin (or $GOBIN) directory, which is likely to be $HOME/go/bin

so you may find the binary in ~/go/bin/bigquery-emulator

j-antunes commented 1 month ago

@totem3 You are correct on my local enviroment, I have a folder called go in my $HOME Inside the go folder I have 

❯ ls ~/go/bin
dlv  gopls  staticcheck

Should I have other stuff inside this folder?

When i try to run it I now get this error:

❯ CGO_ENABLED=1 CC=clang CXX=clang++ go install ./cmd/bigquery-emulator
# github.com/goccy/go-zetasql/internal/ccall/go-absl/time/go_internal/cctz/time_zone
In file included from bind.cc:97:
../../go/pkg/mod/github.com/goccy/go-zetasql@v0.5.5/internal/ccall/absl/time/internal/cctz/include/cctz/time_zone.h:23:10: fatal error: 'chrono' file not found
j-antunes commented 1 month ago

@totem3 nvm I figure out what you meant. The file is located in ~/go/bin/

What I did was cd ~/go/bin ./bigquery-emulator --project test

And it works!

Thanks for all your help!

Do you think this gist should be added to the docs or readme? I think it would be valuable to others.

totem3 commented 1 month ago

I'm glad to hear it worked out. I have fixed the gist. Thank you for getting in touch.

And thank you for the suggestion. I agree that this gist could be useful. But since this kind of instruction might need updates due to different environments such as OS, distributions, versions, etc.., I concerned that it increases maintainers' burden, I'm a bit reluctant to add such instructions directly to the README.

To help others, I will transfer this issue to a discussion and keep it there for reference.