[Bug] Build fails collab-entity

[bytersproblem]

Describe the bug A clear and concise description of what the bug is.

Build fails with no config modification and on running /script/run_local_server.sh

Error reported
error: failed to run custom build command for `collab-entity v0.2.0 (https://github.com/AppFlowy-IO/AppFlowy-Collab?rev=6f98f054a8306a8791b65bf1825091ab3fe60166#6f98f054)`

Caused by:
  process didn't exit successfully: `/home/borivoje/AppFlowy-Cloud/target/debug/build/collab-entity-139c7bb98eb6684a/build-script-build` (exit status: 1)
  --- stderr
  Error: Os { code: 2, kind: NotFound, message: "No such file or directory" }
warning: build failed, waiting for other jobs to finish...

To Reproduce Steps to reproduce the behavior:

System is PopOS 22.04. installed the latest docker-engine via apt procedure installed the postgresql from popsos repositories cloned the repo copied the dev.env to .env ran the script /script/run_local_server.sh

Expected behavior

Screenshots

Desktop (please complete the following information):

• OS: PopOS 22.04
• Browser : didn't reach the step
• Version [e.g. 22] rustc 1.80.1 (3f5fd8dd4 2024-08-06)

[bytersproblem]

The same error on Ubuntu Server 24.04. Installed the latest docker-ce, cargo from the repositories and postgres, since the bash script requires psql.

[khorshuheng]

Can you install protobuf compiler on the machine? While we have set up a fallback using the vendored protobuf compiler, this fallback may fail in some systems.

I will submit a fix that use vendored protobuf compiler by default so that users don't have to install protobuf compiler on their machine.

[bytersproblem]

I managed to pass the build with installing the latest rustc from https://www.rust-lang.org/tools/install. I also managed to complete the migration processes. Now, with cargo run I get the message that the database server name cannot be resolved temporarilty

AppFlowy Cloud with RUST_LOG=info
  2024-08-24T11:47:44.980173Z  INFO  Preparing to run database migrations...
    at src/application.rs:202

  2024-08-24T11:47:44.980299Z  INFO  Connecting to postgres database with setting: DatabaseSetting { pg_conn_opts: PgConnectOptions { host: "appflowy-cloud-postgres", port: 5432, socket: None, username: "postgres", password: Some("password"), database: Some("postgres"), ssl_mode: Prefer, ssl_root_cert: None, ssl_client_cert: None, ssl_client_key: None, statement_cache_capacity: 100, application_name: None, log_settings: LogSettings { statements_level: Debug, slow_statements_level: Warn, slow_statements_duration: 1s }, extra_float_digits: Some("2"), options: None }, require_ssl: false, max_connections: 40 }
    at src/application.rs:472

Error: Failed to initialize application state: Failed to connect to postgres database: error communicating with database: failed to lookup address information: Temporary failure in name resolution

I installed manually with deploying the "dev" version of the docker-compose. The port of the database is exposed to the localhost. Where should I look now, having the error presented?

[bytersproblem]

I must have changed the database hostname variable to something else. I also removed the AI part of the docker-compose, since it kept restarting.

Now it cannot send confirmation mail.

AppFlowy Cloud with RUST_LOG=info
  2024-08-24T12:17:14.826911Z  INFO  Preparing to run database migrations...
    at src/application.rs:202

  2024-08-24T12:17:14.827248Z  INFO  Connecting to postgres database with setting: DatabaseSetting { pg_conn_opts: PgConnectOptions { host: "localhost", port: 5432, socket: None, username: "postgres", password: Some("password"), database: Some("postgres"), ssl_mode: Prefer, ssl_root_cert: None, ssl_client_cert: None, ssl_client_key: None, statement_cache_capacity: 100, application_name: None, log_settings: LogSettings { statements_level: Debug, slow_statements_level: Warn, slow_statements_duration: 1s }, extra_float_digits: Some("2"), options: None }, require_ssl: false, max_connections: 40 }
    at src/application.rs:472

  2024-08-24T12:17:14.996065Z  INFO  Setting up S3 bucket...
    at src/application.rs:207

  2024-08-24T12:17:15.050840Z  INFO  Bucket already exists
    at src/application.rs:455

  2024-08-24T12:17:15.050935Z  INFO  Connecting to GoTrue...
    at src/application.rs:218

  2024-08-24T12:17:15.050968Z  INFO  Connecting to GoTrue with setting: GoTrueSetting { base_url: "http://localhost:9999", ext_url: "http://localhost:9999", jwt_secret: Secret([REDACTED alloc::string::String]), admin_email: "admin@example.com", admin_password: Secret([REDACTED alloc::string::String]) }
    at src/application.rs:492

  2024-08-24T12:17:15.972224Z ERROR  error: code: 500, msg:Error sending confirmation mail, error_id: Some("0a692691-98fa-48a3-a047-8660c1ee6e10")
    at libs/gotrue/src/api.rs:57
    in sign_up

Error: Failed to initialize application state: code: 500, msg:Error sending confirmation mail, error_id: Some("0a692691-98fa-48a3-a047-8660c1ee6e10")

[khorshuheng]

Glad to know that installing newer tooling fixed the issue. As for the other issues:

1. While it is indeed possible that you accidentally change the postgres hostname in dev.env, another possibility is that you have copied deploy.env as opposed to dev.env - which would have explained why the db hostname can not be resolved. So do verify that you env file is indeed copied from dev.env.

2. I am assuming, that you are running the docker compose dev directly, via docker compose up. In the script, you will find this: https://github.com/AppFlowy-IO/AppFlowy-Cloud/blob/d111282f7ec38bdcacca87dcddf77854ff599210/script/run_local_server.sh#L16

By the way, the docker compose dev is mainly used for AppFlowy cloud development. It is actually not possible to use this version of the docker compose as the self host deployment in the AppFlowy flutter app.

[bytersproblem]

So are you saying that I will not be able to sync with the dev setup android and linux apps?

[khorshuheng]

Yes. For proper sync up, you will need to use the docker-compose file as opposed to docker-compose.dev . The main difference between the two, is that gotrue service and all other Appflowy services are accessible via nginx. The frontend app assumes that all these services are accessible via a single endpoint with different path, which will not be fulfil by the docker compose dev file

[bytersproblem]

The problem is that my nginx is on the different machine, and I would like to redirect them to that nginx since it serves my domain. But I do not see any ports exposed in the docker-compose.yml, only for the nginx part. How shoud I make this forwarded to my other machine where I have a nginx domain?

Thanks for your exhaustive help

[bytersproblem]

Shoud I still continue with the dev docker-compose and forward the ports to virtual locations in nginx?

[khorshuheng]

You can refer to https://github.com/AppFlowy-IO/AppFlowy-Cloud/issues/622 and the related issue as a guideline on how to start.

[bytersproblem]

Thanks. I followed #622, made a docker-compose.yml, .env and brought up containers as usual. Some containers report errors. I thing that the main error is in the appflowy-cloud-appflowy_cloud-1. It reports that a relation does not exist.

Error: Failed to initialize application state: Failed to run migrations: while executing migration 20231130140000: error returned from database: relation "auth.users" does not exist

I did not succeed in doing initial migrations with sqlx becase it reports that the hostname is empty. Do I need to do the migrations, and from what variable does sqlx read the hostname of postgres?

[bytersproblem]

I cheked the previous contitions of the error. It seems that the database info is obtained from the .env. I think that the main problem is that sqlx is not run inside the container, and there are no ports exposed from docker to connect to postrges hostname.

[khorshuheng]

Sqlx migrations are applied during Appflowy cloud service start up, rather than via cargo. However, "auth.users" is actually created via an SQL migration script that was mounted to the postgres pod during start up (you can check the volume section of the docker compose file, for postgres).

The reason why this is not managed automatically by Sqlx, is that this table is actually required by the gotrue service, rather than Appflowy cloud.

[bytersproblem]

I see. I am checking the gotrue container for errors it shows this

goroutine 1 [running]:
net/url.(*URL).Query(0xc00004c00d?)
    /usr/local/go/src/net/url/url.go:1159 +0xe
github.com/supabase/gotrue/cmd.migrate(0xc00040fb80?, {0x0?, 0x0?, 0x0?})
    /go/src/supabase/gotrue/cmd/migrate_cmd.go:55 +0x227
github.com/supabase/gotrue/cmd.init.func3(0x19e8e00, {0x1a26ae0?, 0x4?, 0x10b8b3b?})
    /go/src/supabase/gotrue/cmd/root_cmd.go:17 +0x1d
github.com/spf13/cobra.(*Command).execute(0x19e8e00, {0xc00003e2b0, 0x0, 0x0})
    /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:920 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0x19e8e00)
    /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3a5
github.com/spf13/cobra.(*Command).Execute(...)
    /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968
github.com/spf13/cobra.(*Command).ExecuteContext(0x12630e8?, {0x1263270?, 0xc0004bc500?})
    /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:961 +0x47
main.main()
    /go/src/supabase/gotrue/main.go:31 +0xf9
{"level":"info","msg":"Go runtime metrics collection started","time":"2024-08-30T12:24:27Z"}
{"level":"info","msg":"received graceful shutdown signal","time":"2024-08-30T12:24:27Z"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x619cee]

[bytersproblem]

I tried to set th FQDN env variable as my local IP. Since I installed traefic on that maching, my plan is to host the trafic on local address only, and to transfer it with nginx to my domain hosting server.