Invalid database url

[David-Lor]

Hello there,

I'm trying to run a syncstorage server in Docker, but I'm receiving this error when the server starts:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ApiError { kind: Db(DbError { kind: Common(SyncstorageDbError { kind: Internal("Invalid database url: mysql://syncserver:****@mozillasync_syncstorage_db:3306/syncstorage"), status: 500, backtrace:    0: <syncstorage_db_common::error::SyncstorageDbError as core::convert::From<syncstorage_db_common::error::SyncstorageDbErrorKind>>::from

Where **** is my user password, and it only contains upper/lower letters and numbers, no special characters or symbols. I've tried with and without specifying the port.

The database credentials and connection details are correct, as I can connect to it from other container:

bash-4.4# mysql -u syncserver -p'****' -h mozillasync_syncstorage_db -P 3306 -D syncstorage
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 10
Server version: 8.2.0 MySQL Community Server - GPL

Copyright (c) 2000, 2023, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use syncstorage;
Database changed

mysql> create table foo(id int primary key);
Query OK, 0 rows affected (0.43 sec)

mysql> drop table foo;
Query OK, 0 rows affected (1.79 sec)

Judging by the error Invalid database url, it would seem my URI is incorrect, but I don't see structural differences with the example provided here.

This is my dockercompose:

version: "3.4"

services:
  syncstorage:
    image: mozilla/syncstorage-rs:0.14.4
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    environment:
      SYNC_HOST: 0.0.0.0
      SYNC_MASTER_SECRET: ***
      SYNC_SYNCSTORAGE__DATABASE_URL: mysql://syncserver:***@syncstorage_db:3306/syncstorage
      SYNC_TOKENSERVER__DATABASE_URL: mysql://syncserver:***@syncstorage_db:3306/tokenserver
      SYNC_TOKENSERVER__RUN_MIGRATIONS: "true"
    depends_on:
      - syncstorage_db
    networks:
      - backend

  syncstorage_db:
    image: mysql:8.2.0
    volumes:
      - /home/docker/volumes/mozillasync_syncstorage_mysql_data:/var/lib/mysql
      - ./initdb.sql:/docker-entrypoint-initdb.d/1-syncserver-initdb.sql:ro
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    environment:
      MYSQL_ROOT_PASSWORD: ***
      MYSQL_DATABASE: root
      MYSQL_USER: syncserver
      MYSQL_PASSWORD: ***
    networks:
      - backend

networks:
  backend:
    external:
      name: backend-overlay

┆Issue is synchronized with this Jira Task

[jackyzy823]

See https://github.com/mozilla-services/syncstorage-rs/issues/1495 and https://github.com/mozilla-services/syncstorage-rs/issues/1482

[WAdama]

At the moment you have to use version 0.13.7 for use with mysql/mariadb. This still works...

[azoller1]

I still cant get past this invalid URL error even with version 0.13.7. Using similar compose file as referenced earlier in this thread

[jrconlin]

The error [is generated] in syncstorage-spanner, which indicates that sync believes you are using spanner for this. syncstorage-settings presumes you're using spanner if the database URL begins with spanner:// The default for this is set to mysql://root..., so you may have either an environment variable like SYNC_STORAGE__DATABASE_URL="spanner://..." (or if you're using a --config file database_url = "spanner://...")

[Unfaehig]

Is there a reason why syncstorage is going to use spanner? I mean don't get me wrong but i think a lot of Firefox users are not using Google Chrome to get rid of Google dependencies. When Mozilla is now building Google dependencies into backend functionalities like this here sync private information like bookmarks, extensions etc. doesn't really make sense for me!

[jrconlin]

There are a few reasons.

1) MySQL isn't sufficient. Put bluntly, we have lots of experience running MySQL nodes, and at the scale we need to run at, MySQL isn't the answer. Nodes fail, often, which meant that we had a HUGE data loss problem. This is invisible for anyone who has multiple devices, but many folk considered "Sync" to be a Backup service and were sad and upset when they needed to reinstall and all their data was gone because the node failed. (Postgres is a bit better, but not significantly so.)

2) Cost. Put simply, we save a LOT of money by running this on a hosted, distributed database. We don't have to pay for the servers, co-lo space, someone to go push buttons or swap disks, etc. It's the reason that a lot of companies use distributed databases and shared services. As to which service we use, that decision is outside of my pay-grade. For now, we're on GCP. We may move in the future to some other database provided we get a better deal somewhere else. That said, most of the big, distributed databases aren't exactly SQL. They get very close, but they're different. There are a lot of reasons for that, but it's the hand we have to work with, so we do.

Our problems are not your problems (and frankly, I really hope that you never have to deal with our problems).

All that said, Spanner is just a big SQL-like database, so there's no difference in what gets stored. It's still completely end-to-end encrypted, so all anyone outside of your devices can tell it's just blobs of random crap. The syncstorage server basically does some auth, and then just stores and retrieves those blobs of "stuff" based on the API. (Honestly? Anyone could write their own syncserver if they wanted to.) syncstorage is basically just a bunch of buckets with a slightly smart way of storing and retrieving them.

[Unfaehig]

@jrconlin Thanks for taking the time and going into details really appreciate it! I totally get the point that MySQL is may be not the best for huge amount of data with a lot of requests. You guys need to think in difference scales. I just really hope this will not mean MySQL will be deprecated in this project. It has a huge potential using Rust and give the ability to self host your services which is quite unique in this area!

[jrconlin]

@Unfaehig Honestly? I hope not either. That said, the decision isn't mine to make. I can definitely argue that we need to have the level of data abstraction that lets us migrate to a different storage engine system, so that part will definitely stick around, but as for the MySQL work, that's a bit less certain. We really need the community who uses that to support it because it's impossible to argue for the time (and money) to do it internally. There are already some wonderful soul that is stepping up to the challenge, and I'd encourage anyone to help her out. Even if you don't know how to write rust code, you could help with writing documents or cleaning up comments. If you know python, help with the testing code. If you know docker configs, you can help create something better than the confusing test specific configurations we have. I'm happy to provide pointers and tips when I can.

There's the Firefox Sync matrix board on chat.mozilla.org that you can use to coordinate if you want.

Frankly, please put us to shame and make something better than what we've would have come up with.