search

cybergarage / puzzledb-go

PuzzleDB is a high-performance, distributed, cloud-native, multi-API, multi-model database.
Apache License 2.0
3 stars 2 forks source link

ARM64 Dockerfile cannot use the client and server debian packages #2

Open al-sabr opened 1 month ago

al-sabr commented 1 month ago

The following lines are meant to download debian packages on amd64 arch, but unfortunately those files are not available for aarch64.

https://github.com/cybergarage/puzzledb-go/blob/3366bc205979dc0736b29f5ed4e3482b2b0a1659/Dockerfile#L12

https://github.com/cybergarage/puzzledb-go/blob/3366bc205979dc0736b29f5ed4e3482b2b0a1659/Dockerfile#L16

How can I run puzzledb-go in a Raspberry Pi 4?

Can you make a cross-platform Dockerfile please?

cybergarage commented 4 weeks ago

@al-sabr Thank you for your report.

I'll create a new Dockerfile for Raspberry Pi to switch the default storage from FoundationDB to go-memdb.

al-sabr commented 4 weeks ago

I've asked FDB team to push arm64 Debian package let's see what they will say.

al-sabr commented 3 weeks ago

@al-sabr Thank you for your report.

I'll create a new Dockerfile for Raspberry Pi to switch the default storage from FoundationDB to go-memdb.

Could you give me an approximate on when this will be ready because I need puzzledb for my project ASAP.

Kind Regards

al-sabr commented 3 weeks ago

Any thought about the target time?

cybergarage commented 3 weeks ago

@al-sabr

I am about to set up a RasberryPI, what operating system are you planning to use?

al-sabr commented 3 weeks ago

@al-sabr

I am about to set up a RasberryPI, what operating system are you planning to use?

I am using DietPi which is ultimate lightweight distro, I really recommend it!

al-sabr commented 2 weeks ago

@al-sabr

I am about to set up a RasberryPI, what operating system are you planning to use?

You can just use QEMU for aarch64 emulation you really don't need any Raspberry Pi if you don't have one.

al-sabr commented 2 weeks ago

Do you know how long it is going to take to change the underlying dependency rergarding FoundationDB?

al-sabr commented 2 weeks ago

Do you know how long it is going to take to change the underlying dependency rergarding FoundationDB?

Any thoughts?

cybergarage commented 2 weeks ago

@al-sabr

If you want it to work on RasberryPI, delete all the following FoundationDB-based plugins directories

It would be easy to modify Server::reloadEmbeddedPlugins() to not load that deleted plugins. The standard config file has the FoundationDB plugins disabled, so this procedure should work.

al-sabr commented 2 weeks ago

@al-sabr

If you want it to work on RasberryPI, delete all the following FoundationDB-based plugins directories

* puzzledb/plugins

  * coordinator/fdb
  * store/kv/fdb

It would be easy to modify Server::reloadEmbeddedPlugins() to not load that deleted plugins. The standard config file has the FoundationDB plugins disabled, so this procedure should work. It's pretty interesting I am Confuscious!

So you mean that FoundationDB is never used? Why is it in the Dockerfile then? I have another question ... how functional is puzzledb does anyone use it in production and does it fully replace mongo,mysql, postgres & redis?

That line might pose a problem https://github.com/cybergarage/puzzledb-go/blob/2e05dd364c8943145b532d84cf8167549b4926ea/docker/entrypoint.sh#L2

al-sabr commented 2 weeks ago

Is it correct ? https://github.com/cybergarage/puzzledb-go/pull/3

It seems that those settings also are to be changed what are the other values since we removed FDB?

https://github.com/cybergarage/puzzledb-go/blob/2e05dd364c8943145b532d84cf8167549b4926ea/Dockerfile#L29

Are those correct ? https://github.com/al-sabr/puzzledb-go/commit/7e36c2b1bc0406118a29a40f23286c8c1ea65d62

al-sabr commented 2 weeks ago

Getting this error once I did the clean-up

go build -o /puzzledb-server doc/cmd/server/main.go
# github.com/apple/foundationdb/bindings/go/src/fdb
C:\Users\DUMMY\go\pkg\mod\github.com\apple\foundationdb\bindings\go@v0.0.0-20230317231317-e330030d07af\src\fdb\database.go:26:11: fatal error: foundationdb/fdb_c.h: No such file or directory
   26 | // #include <foundationdb/fdb_c.h>
      |           ^~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
# vitess.io/vitess/go/vt/servenv

https://github.com/al-sabr/puzzledb-go/commits/raspberrypi-patch/

al-sabr commented 1 week ago

Can you help me to change the remaining code to get rid of the last dependencies to FoundationDB?

I installed FDB 6.3.12 on my windows machine from this link :

https://apple.github.io/foundationdb/downloads.html

and CHANGED the code as suggested on this site:

https://forums.foundationdb.org/t/compilation-golang-binding-on-windows-10/1705/5

I retried the build with this result:

go build -o /puzzledb-server doc/cmd/server/main.go
# vitess.io/vitess/go/vt/servenv
C:\Users\DUMMY\go\pkg\mod\vitess.io\vitess@v0.12.6\go\vt\servenv\pprof.go:310:30: undefined: syscall.SIGUSR1        
C:\Users\DUMMY\go\pkg\mod\vitess.io\vitess@v0.12.6\go\vt\servenv\servenv.go:114:22: undefined: syscall.Rlimit       
C:\Users\DUMMY\go\pkg\mod\vitess.io\vitess@v0.12.6\go\vt\servenv\servenv.go:115:20: undefined: syscall.Getrlimit    
C:\Users\DUMMY\go\pkg\mod\vitess.io\vitess@v0.12.6\go\vt\servenv\servenv.go:115:38: undefined: syscall.RLIMIT_NOFILE
# github.com/apple/foundationdb/bindings/go/src/fdb
C:\Users\DUMMY\go\pkg\mod\github.com\apple\foundationdb\bindings\go@v0.0.0-20230317231317-e330030d07af\src\fdb\database.go:96:21: could not determine kind of name for C.fdb_database_reboot_worker

image

al-sabr commented 1 week ago

I was able to get rid of that error by replacing the include folder from FDB 6.3.12 with the latest folder from : bindings/c/foundationdb

image

I also edited the CMake template file manually because I didn't want to deal with CMake compilation and generation

image

Now that error is gone but there's still the others left.

go build -o /puzzledb-server doc/cmd/server/main.go
# vitess.io/vitess/go/vt/servenv
C:\Users\DUMMY\go\pkg\mod\vitess.io\vitess@v0.12.6\go\vt\servenv\pprof.go:310:30: undefined: syscall.SIGUSR1        
C:\Users\DUMMY\go\pkg\mod\vitess.io\vitess@v0.12.6\go\vt\servenv\servenv.go:114:22: undefined: syscall.Rlimit       
C:\Users\DUMMY\go\pkg\mod\vitess.io\vitess@v0.12.6\go\vt\servenv\servenv.go:115:20: undefined: syscall.Getrlimit    
C:\Users\DUMMY\go\pkg\mod\vitess.io\vitess@v0.12.6\go\vt\servenv\servenv.go:115:38: undefined: syscall.RLIMIT_NOFILE
al-sabr commented 1 week ago

I have bumped the version of of vitess to latest one and the compilations goes through on Windows but other errors are appering now:

go build -o /puzzledb-server doc/cmd/server/main.go
# github.com/cybergarage/go-mysql/mysql/query
C:\Users\DUMMY\go\pkg\mod\github.com\cybergarage\go-mysql@v1.0.6\mysql\query\condition.go:40:27: undefined: vitesssp.RangeCond
C:\Users\DUMMY\go\pkg\mod\github.com\cybergarage\go-mysql@v1.0.6\mysql\query\statement.go:63:32: undefined: vitesssp.SetTransaction
C:\Users\DUMMY\go\pkg\mod\github.com\cybergarage\go-mysql@v1.0.6\mysql\query\statement.go:66:32: undefined: vitesssp.Characteristic
C:\Users\DUMMY\go\pkg\mod\github.com\cybergarage\go-mysql@v1.0.6\mysql\query\statement.go:69:32: undefined: vitesssp.IsolationLevel
C:\Users\DUMMY\go\pkg\mod\github.com\cybergarage\go-mysql@v1.0.6\mysql\query\statement.go:72:28: undefined: vitesssp.AccessMode
C:\Users\DUMMY\go\pkg\mod\github.com\cybergarage\go-mysql@v1.0.6\mysql\query\value.go:32:22: undefined: vitesssp.BitVal
C:\Users\DUMMY\go\pkg\mod\github.com\cybergarage\go-mysql@v1.0.6\mysql\query\insert.go:35:27: stmt.Insert.Table.Name undefined (type *sqlparser.AliasedTableExpr has no field or method Name)
C:\Users\DUMMY\go\pkg\mod\github.com\cybergarage\go-mysql@v1.0.6\mysql\query\parser.go:32:24: undefined: vitesssp.Parse
C:\Users\DUMMY\go\pkg\mod\github.com\cybergarage\go-mysql@v1.0.6\mysql\query\schema.go:43:44: undefined: vitesssp.NewTableIdent
# go.opentelemetry.io/otel/sdk/trace
C:\Users\DUMMY\go\pkg\mod\go.opentelemetry.io\otel\sdk@v1.15.1\trace\provider.go:90:30: cannot use &TracerProvider{} (value of type *TracerProvider) as "go.opentelemetry.io/otel/trace".TracerProvider value in variable declaration: *TracerProvider does not implement "go.opentelemetry.io/otel/trace".TracerProvider (missing method tracerProvider)
C:\Users\DUMMY\go\pkg\mod\go.opentelemetry.io\otel\sdk@v1.15.1\trace\provider.go:170:10: cannot use t (variable of type *tracer) as "go.opentelemetry.io/otel/trace".Tracer value in return statement: *tracer does not implement "go.opentelemetry.io/otel/trace".Tracer (missing method tracer)
C:\Users\DUMMY\go\pkg\mod\go.opentelemetry.io\otel\sdk@v1.15.1\trace\span.go:161:23: cannot use (*recordingSpan)(nil) (value of type *recordingSpan) as ReadWriteSpan value in variable declaration: *recordingSpan does not implement ReadWriteSpan (missing method AddLink)
                have addLink("go.opentelemetry.io/otel/trace".Link)
                want AddLink("go.opentelemetry.io/otel/trace".Link)
C:\Users\DUMMY\go\pkg\mod\go.opentelemetry.io\otel\sdk@v1.15.1\trace\span.go:692:9: cannot use s.tracer.provider (variable of type *TracerProvider) as "go.opentelemetry.io/otel/trace".TracerProvider value in return statement: *TracerProvider does not implement "go.opentelemetry.io/otel/trace".TracerProvider (missing method tracerProvider)
C:\Users\DUMMY\go\pkg\mod\go.opentelemetry.io\otel\sdk@v1.15.1\trace\span.go:780:20: cannot use nonRecordingSpan{} (value of type nonRecordingSpan) as "go.opentelemetry.io/otel/trace".Span value in variable declaration: nonRecordingSpan does not implement "go.opentelemetry.io/otel/trace".Span (missing method AddLink)
C:\Users\DUMMY\go\pkg\mod\go.opentelemetry.io\otel\sdk@v1.15.1\trace\span.go:811:74: cannot use s.tracer.provider (variable of type *TracerProvider) as "go.opentelemetry.io/otel/trace".TracerProvider value in return statement: *TracerProvider does not implement "go.opentelemetry.io/otel/trace".TracerProvider (missing method tracerProvider)
C:\Users\DUMMY\go\pkg\mod\go.opentelemetry.io\otel\sdk@v1.15.1\trace\tracer.go:30:22: cannot use &tracer{} (value of type *tracer) as "go.opentelemetry.io/otel/trace".Tracer value in variable declaration: *tracer does not implement "go.opentelemetry.io/otel/trace".Tracer (missing method tracer)
C:\Users\DUMMY\go\pkg\mod\go.opentelemetry.io\otel\sdk@v1.15.1\trace\tracer.go:47:21: impossible type assertion: p.(*recordingSpan)
        *recordingSpan does not implement "go.opentelemetry.io/otel/trace".Span (missing method AddLink)
                have addLink("go.opentelemetry.io/otel/trace".Link)
                want AddLink("go.opentelemetry.io/otel/trace".Link)
C:\Users\DUMMY\go\pkg\mod\go.opentelemetry.io\otel\sdk@v1.15.1\trace\tracer.go:117:10: cannot use tr.newNonRecordingSpan(sc) (value of type nonRecordingSpan) as "go.opentelemetry.io/otel/trace".Span value in return statement: nonRecordingSpan does not implement "go.opentelemetry.io/otel/trace".Span (missing method AddLink)
C:\Users\DUMMY\go\pkg\mod\go.opentelemetry.io\otel\sdk@v1.15.1\trace\tracer.go:119:9: cannot use tr.newRecordingSpan(psc, sc, name, samplingResult, config) (value of type *recordingSpan) as "go.opentelemetry.io/otel/trace".Span value in return statement: *recordingSpan does not implement "go.opentelemetry.io/otel/trace".Span (missing method AddLink)
                have addLink("go.opentelemetry.io/otel/trace".Link)
                want AddLink("go.opentelemetry.io/otel/trace".Link)
C:\Users\DUMMY\go\pkg\mod\go.opentelemetry.io\otel\sdk@v1.15.1\trace\tracer.go:119:9: too many errors

image

al-sabr commented 1 week ago

It seems that vitess is not even needed anymore in your code and also there's still foundation db dependencies.

al-sabr commented 1 week ago

Hi are too busy or you forgot about reading the logs?

cybergarage commented 1 week ago

@al-sabr

go-mysql has been redesigned and replaced with its own implementation in the latest branch and is no longer dependent on Vittess.

This new go-mysql version will be released as v2.0 and we are currently modifying puzzledb-go to the latest go-mysql.

al-sabr commented 1 week ago

@al-sabr

go-mysql has been redesigned and replaced with its own implementation in the latest branch and is no longer dependent on Vittess.

This new go-mysql version will be released as v2.0 and we are currently modifying puzzledb-go to the latest go-mysql.

I would like to integrate puzzledb-go in my project for 4 db in 1. Is it ready to be used as embedded db and how in terms of percentage is it working for mysql, postgres, redis & mongo?