search

mozilla / ichnaea

Mozilla Ichnaea
http://location.services.mozilla.com
Apache License 2.0
574 stars 139 forks source link

Documentation for running local instance #1642

Closed HJJac closed 3 years ago

HJJac commented 3 years ago

I have been following https://ichnaea.readthedocs.io/en/latest/deploy.html for implementing a local instance. It does mention that you can run it outside of AWS but does not mention how or what dependencies you need. I'm not a complete noob but the instructions are hard to follow. The following is what I've come up with so far. Please have a look at it and maybe update the documentation to explain how to run a local version.

//INSTALL LATEST VERSION OF UBUNTU SERVER (21.04 at the time)
//GUIDE FOLLOWED: https://ichnaea.readthedocs.io/en/latest/deploy.html

//UPDATE TO LATEST VERSIONS
sudo apt-get update
sudo apt-get upgrade

//INSTALL MYSQL
sudo apt-get install -y mariadb-server redis-server

//SETUP DB
sudo mysql                  //if password mysql -u root -p
CREATE DATABASE location;
GRANT ALL PRIVILEGES ON location.* TO 'ichnaea'@'localhost' IDENTIFIED BY '********';
quit

//INSTALL DOCKER
sudo apt  install docker.io

//DOWNLOAD LATEST DOCKER (https://github.com/mozilla/ichnaea/releases)
sudo docker pull mozilla/location:2021.07.12

//TEST IF IMAGE WAS DOWNLOADED SUCCESSFULLY
sudo docker run -it --rm mozilla/location:2021.07.12 shell
exit

//SETUP DOCKER ENVIROMENT FILE
sudo nano /etc/ichnaeaenv.txt
//EDITED CONFIG
//******************************************************************************
DB_HOST=localhost
DB_USER=jay
DB_PASSWORD=********
GEOIP_PATH=/app/geoip/GeoLite2-City.mmdb
REDIS_HOST=127.0.0.1:6379
SECRET_KEY=*************
//******************************************************************************

//RUN THE INITIAL DB SETUP
sudo docker run -it --rm --env-file /etc/ichnaeaenv.txt mozilla/location:2021.07.12 alembic stamp base

//UPDATE THE DATABASE WITH THE LATEST SCHEMA
sudo docker run -it --rm --env-file /etc/ichnaeaenv.txt mozilla/location:2021.07.12 alembic upgrade head

//DOWNLOAD THE "GeoLite2-City.mmbd" FROM https://www.maxmind.com/en/accounts/500894/geoip/downloads
//COPPY AND UNZIP IN "/opt/geoip"
sudo mv GeoLite2-City_20210720.tar.gz /opt/geoip/
cd /opt/geoip/
sudo tar xvzf GeoLite2-City_20210720.tar.gz
sudo rm -r GeoLite2-City_20210720.tar.gz
sudo mv GeoLite2-City_20210720 GeoLite2-City.mmdb

//START SCHEDULER
sudo docker run -d --env-file /etc/ichnaeaenv.txt --volume /opt/geoip:/app/geoip mozilla/location:2021.07.12 scheduler

//START WORKER
sudo docker run -d --env-file /etc/ichnaeaenv.txt --volume /opt/geoip:/app/geoip mozilla/location:2021.07.12 worker

//START WEB
sudo docker run -d --env-file /etc/ichnaeaenv.txt --volume /opt/geoip:/app/geoip -p 8000:8000/tcp mozilla/location:2021.07.12 web

//RUNTIME CHECKS
//CHECK WEB ROLE
curl -i http://localhost:8000/__heartbeat__

//TEST HTTP API ENDPOINTS
curl -H "X-Forwarded-For: 81.2.69.192" http://localhost:8000/v1/geolocate?key=test
jwhitlock commented 3 years ago

What issues are you seeing?

There's an issue for updating the deployment docs at https://github.com/mozilla/ichnaea/issues/1385, but no changes have been made since that was filed.

HJJac commented 3 years ago

I think my main issue is that the docker commands return :"Usage: /app/docker/app_entrypoint.sh {scheduler|web|worker|shell}" which I do not think is right. (I'm a little new to docker). This results in my DB not net being setup and so on.

jwhitlock commented 3 years ago

Thanks. The instructions are way out of date, and it will take some work to figure out the new ones. I mostly use the local development environment, and we use other technologies to deploy to Mozilla's production servers that don't fit well in general documentation.

I'm guessing that this is the command giving that error:

//RUN THE INITIAL DB SETUP
sudo docker run -it --rm --env-file /etc/ichnaeaenv.txt mozilla/location:2021.07.12 alembic stamp base

sudo docker shouldn't be needed, but it requires additional steps to allow regular users to use the commands.

There should be a latest tag, so that you install with mozilla/location instead of mozilla/location:2021.07.12. We'll need to update the build commands to ship that. I could also use an arm64 variant.

The docker image runs app_entrypoint.sh, which is giving that error. The shell command then runs shell comands, so it should be shell alembic stamp base instead of just alembic stamp base. In the development environment, the make setup command runs shell ./docker/run_setup.sh, and run_setup.sh drops and recreates the database. So, you are the pioneer here.

We've changed the format of the environment variables, so instead of this in /etc/ichnaeaenv.txt:

DB_HOST=localhost
DB_USER=jay
DB_PASSWORD=********

you'll want database URLs, that look something like:

DB_READONLY_URI=mysql+pymysql://ichnaea:****@localhost:3306/location
DB_READWRITE_URI=mysql+pymysql://ichnaea:****@localhost:3306/location

I switched jay for ichnaea, to match your SQL creating the ichnaea user.

See docker/config/local_dev.env for other possible environment variables you will need.

One other thing you may need is to set ICHNAEA_UID and ICHNAEA_GID to your account's uid and gid. See the local development environment docs, and search for "Set UID and GID for Docker container user".

To get the docker container to talk to your non-Docker MySQL, you'll need to add some networking configuration to your command. The Stackoverflow question How to connect to local MySQL server through Docker? has answers from several years. I think --network="host" is a good place to start.

With all that, after updating ichnaeaenv, this may be the new command:

//RUN THE INITIAL DB SETUP
sudo docker run -it --rm --env-file /etc/ichnaeaenv.txt --network="host" mozilla/location:2021.07.12 shell alembic stamp base
HJJac commented 3 years ago

@jwhitlock Thank you for the information, think we are making some progress here. I'll post the complete setup once I've\we figured out how to do a local deployment in the hope that it will help someone else.

I've done as @jwhitlock have suggested with the exception of ICHNAEA_UID and ICHNAEA_GID. The link provided does not explain clearly how to do this or it has to do with the development environment?

I've added a test to connect to the DB once in the docker and I'm sure it can see the Local DB. Adding --network="host" did do the trick here, but I had to use 127.0.0.1 instead of localhost for some reason.

Running docker run -it --rm --env-file /etc/ichnaeaenv.txt --network="host" mozilla/location shell alembic stamp base produces the following output:

Traceback (most recent call last):
  File "/usr/local/bin/alembic", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/alembic/config.py", line 559, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/usr/local/lib/python3.9/site-packages/alembic/config.py", line 553, in main
    self.run_cmd(cfg, options)
  File "/usr/local/lib/python3.9/site-packages/alembic/config.py", line 530, in run_cmd
    fn(
  File "/usr/local/lib/python3.9/site-packages/alembic/command.py", line 570, in stamp
    script.run_env()
  File "/usr/local/lib/python3.9/site-packages/alembic/script/base.py", line 490, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.9/site-packages/alembic/util/compat.py", line 184, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "ichnaea/alembic/env.py", line 24, in <module>
    run_migrations_online()
  File "ichnaea/alembic/env.py", line 14, in run_migrations_online
    db_url = get_sqlalchemy_url()
  File "/app/ichnaea/db.py", line 45, in get_sqlalchemy_url
    raise SqlAlchemyUrlNotSpecified()
ichnaea.db.SqlAlchemyUrlNotSpecified: SQLALCHEMY_URL is not specified in the environment

I'd presume that this worked as there is no error. SQLALCHEMY_URL is not specified, but I'm not migrating so not sure if this is an issue?

Afterwards I ran docker run -it --rm --env-file /etc/ichnaeaenv.txt --network="host" mozilla/location shell alembic upgrade head I added the shell as it did not like the command without it. This produced:

Traceback (most recent call last):
  File "/usr/local/bin/alembic", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/alembic/config.py", line 559, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/usr/local/lib/python3.9/site-packages/alembic/config.py", line 553, in main
    self.run_cmd(cfg, options)
  File "/usr/local/lib/python3.9/site-packages/alembic/config.py", line 530, in run_cmd
    fn(
  File "/usr/local/lib/python3.9/site-packages/alembic/command.py", line 294, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.9/site-packages/alembic/script/base.py", line 490, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.9/site-packages/alembic/util/compat.py", line 184, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "ichnaea/alembic/env.py", line 24, in <module>
    run_migrations_online()
  File "ichnaea/alembic/env.py", line 14, in run_migrations_online
    db_url = get_sqlalchemy_url()
  File "/app/ichnaea/db.py", line 45, in get_sqlalchemy_url
    raise SqlAlchemyUrlNotSpecified()
ichnaea.db.SqlAlchemyUrlNotSpecified: SQLALCHEMY_URL is not specified in the environment

I logged back into the DB and saw that no new tables have been created. Please can you check the result of my last two commands and see if this is correct? For reference I've added all the steps I did to get to this point.

//UPDATE TO LATEST VERSIONS
sudo apt-get update
sudo apt-get upgrade

//INSTALL MYSQL
sudo apt-get install -y mysql-server redis-server

//INSTALL DOCKER
sudo apt install docker.io
//MANAGE DOCKER AS NON-ROOT USER
sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker

//SETUP DB
sudo mysql                  //if password mysql -u root -p
CREATE DATABASE location;
GRANT ALL PRIVILEGES ON location.* TO 'ichnaea'@'localhost' IDENTIFIED BY '***************';
quit

//DOWNLOAD LATEST DOCKER (https://github.com/mozilla/ichnaea/releases)(2021.07.12 at the time)
docker pull mozilla/location

//TEST IF IMAGE WAS DOWNLOADED SUCCESSFULLY AND IF DOCKER CAN SEE DB
docker run -it --rm --network="host" mozilla/location shell
mysql -h 127.0.0.1 -u ichnaea -p ***************
exit

//SETUP DOCKER ENVIROMENT FILE
sudo nano /etc/ichnaeaenv.txt
//EDITED CONFIG
//******************************************************************************
DB_READONLY_URI=mysql+pymysql://ichnaea:***************@127.0.0.1:3306/location
DB_READWRITE_URI=mysql+pymysql://ichnaea:***************@127.0.0.1:3306/location
REDIS_URI=redis://redis:6379/0
GEOIP_PATH=/app/geoip/GeoLite2-City.mmdb
SECRET_KEY=***************
//******************************************************************************

//RUN THE INITIAL DB SETUP
docker run -it --rm --env-file /etc/ichnaeaenv.txt --network="host" mozilla/location shell alembic stamp base

//UPDATE THE DATABASE WITH THE LATEST SCHEMA
docker run -it --rm --env-file /etc/ichnaeaenv.txt --network="host" mozilla/location shell alembic upgrade head
jwhitlock commented 3 years ago

The SQLALCHEMY_URL is required. Add a line to ichnaeaenv.txt that matches the other two URLs:

DB_READONLY_URI=mysql+pymysql://ichnaea:***************@127.0.0.1:3306/location
DB_READWRITE_URI=mysql+pymysql://ichnaea:***************@127.0.0.1:3306/location
SQLALCHEMY_URL=mysql+pymysql://ichnaea:***************@127.0.0.1:3306/location

You'll then need to re-run those alembic commands, to create the tables (or find the next issue...)

HJJac commented 3 years ago

Thank you @jwhitlock. I did same and added all variables as suggested here: docker/config/local_dev.env. Ran the 2 setup setup Lines: docker run -it --rm --env-file /etc/ichnaeaenv.txt --network="host" mozilla/location shell alembic stamp base and docker run -it --rm --env-file /etc/ichnaeaenv.txt --network="host" mozilla/location shell alembic upgrade head. The tables looks to have been created, so that's good. I started the 3 different Roles with the following return:

jay@ichnaea:~$ docker run -d --env-file /etc/ichnaeaenv.txt --volume /opt/geoip:/app/geoip mozilla/location scheduler
9b7ec5aaed55ba699f11f294a1e99955afe3fdb00cd7d451639ca4a782f5c00e
jay@ichnaea:~$ docker run -d --env-file /etc/ichnaeaenv.txt --volume /opt/geoip:/app/geoip mozilla/location worker
7a64b74dfbdd1b18132115682486d2698086b8d248b2293e69608da294d3c0e2
jay@ichnaea:~$ docker run -d --env-file /etc/ichnaeaenv.txt --volume /opt/geoip:/app/geoip -p 8000:8000/tcp mozilla/location web
c6c5b32cb36f3fb5586ac9a0dc1813215bc5eb0b0953a9ba898e087a22331dbd

I then ran the run time check curl -i http://localhost:8000/__heartbeat__ and response:

HTTP/1.1 503 Service Unavailable
Content-Length: 172
Content-Type: application/json
Date: Wed, 28 Jul 2021 17:01:25 GMT
Server: waitress
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff

{"database":{"up":false,"time":0,"alembic_version":"unknown"},"geoip":{"up":false,"time":0,"age_in_days":-1,"version":"2020-02-26T12:17:00Z"},"redis":{"up":false,"time":0}}

By the looks of it, the Database, GeoIP and redis was not running. Logging into the docker I can connect to the DB:

jay@ichnaea:~$ docker run -it --rm --network="host" mozilla/location shell
Opening shell
app@ichnaea:/app$ mysql -h 127.0.0.1 -u ichnaea -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 33
Server version: 10.5.10-MariaDB-0ubuntu0.21.04.1 Ubuntu 21.04
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| location           |
+--------------------+
2 rows in set (0.000 sec)
MariaDB [(none)]>

I tried testing redis from the docker as well:

app@ichnaea:/app$ redis-cli
127.0.0.1:6379>ping
PONG

I'm not sure how to test the geoip.

Looks like the setup works now, but there is something missing when starting the roles.

jwhitlock commented 3 years ago

I think you need the --network=host bit when running the services as well, such as:

docker run -d --env-file /etc/ichnaeaenv.txt --volume /opt/geoip:/app/geoip --network=host mozilla/location scheduler

That will allow those containers to see the database and Redis servers.

The geoip service will be harder to test. First, you'll need an API key. To allocate an API key, something like this might work:

docker run -it --rm --env-file /etc/ichnaeaenv.txt --network="host" mozilla/location /app/ichnaea/scripts/apikey.py create test

That creates a key called test in the database, which you can use against your web service:

curl -v http://localhost:8000/v1/country?key=test

This uses your IP address to find your location. Since you'll be making a local request instead of your external IP address, you won't get a useful response unfortunately.

HJJac commented 3 years ago

@jwhitlock thank you so much for all your help and my apologies for my last issue. I was really tired. OK, think we nailed this. I could not get your last command to generate the key working, I figured it was a python script and just ran that inside the docker with the with the local sources arguments and that worked. Herewith the full trail in the hopes that it will help someone else. Again thank you for your time.

//INSTALL LATEST VERSION OF UBUNTU SERVER (21.04 at the time)

//UPDATE TO LATEST VERSIONS
sudo apt-get update
sudo apt-get upgrade

//INSTALL MYSQL AND REDIS
sudo apt-get install -y mysql-server redis-server

//INSTALL DOCKER
sudo apt install docker.io
//MANAGE DOCKER AS NON-ROOT USER
sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker

//SETUP DB
sudo mysql                  //if password mysql -u root -p
CREATE DATABASE location;
GRANT ALL PRIVILEGES ON location.* TO 'ichnaea'@'localhost' IDENTIFIED BY '*******************';
quit

//DOWNLOAD LATEST DOCKER (https://github.com/mozilla/ichnaea/releases)(2021.07.12 at the time)
docker pull mozilla/location

//TEST IF IMAGE WAS DOWNLOADED SUCCESSFULLY AND IF DOCKER CAN SEE DB
docker run -it --rm --network="host" mozilla/location shell
mysql -h 127.0.0.1 -u ichnaea -p
exit

//SETUP DOCKER ENVIROMENT FILE
sudo nano /etc/ichnaeaenv.txt
//ADD CONFIG TO FILE
//******************************************************************************
LOCAL_DEV_ENV=True
LOGGING_LEVEL=INFO
REDIS_URI=redis://127.0.0.1:6379/0
DB_READONLY_URI=mysql+pymysql://ichnaea:*******************@127.0.0.1:3306/location
DB_READWRITE_URI=mysql+pymysql://ichnaea:*******************@127.0.0.1:3306/location
SQLALCHEMY_URL=mysql+pymysql://ichnaea:*******************@127.0.0.1:3306/location
GEOIP_PATH=/app/geoip/GeoLite2-City.mmdb
SECRET_KEY=*******************
CELERY_WORKER_CONCURRENCY=1
//******************************************************************************

//RUN THE INITIAL DB SETUP
docker run -it --rm --env-file /etc/ichnaeaenv.txt --network="host" mozilla/location shell alembic stamp base

//UPDATE THE DATABASE WITH THE LATEST SCHEMA
docker run -it --rm --env-file /etc/ichnaeaenv.txt --network="host" mozilla/location shell alembic upgrade head

//CHECK IF DB WAS CRATED CORRECTLY
sudo mysql
SHOW DATABASES;
USE location;
SHOW TABLES;

//DOWNLOAD THE "GeoLite2-City.mmbd" FROM https://www.maxmind.com/en/accounts/500894/geoip/downloads
//COPPY AND UNZIP IN "/opt/geoip"
sudo mv GeoLite2-City_20210727.tar.gz /opt/geoip/
cd /opt/geoip/
sudo tar xvzf GeoLite2-City_20210727.tar.gz
sudo rm -r GeoLite2-City_20210727.tar.gz
sudo mv GeoLite2-City_20210727 GeoLite2-City.mmdb

//START SCHEDULER
docker run -d --env-file /etc/ichnaeaenv.txt --volume /opt/geoip:/app/geoip --network=host mozilla/location scheduler

//START WORKER
docker run -d --env-file /etc/ichnaeaenv.txt --volume /opt/geoip:/app/geoip --network=host mozilla/location worker

//START WEB
docker run -d --env-file /etc/ichnaeaenv.txt --volume /opt/geoip:/app/geoip --network=host -p 8000:8000/tcp mozilla/location web

//RUNTIME CHECKS
//CHECK WEB ROLE
curl -i http://localhost:8000/__heartbeat__

//TEST HTTP API ENDPOINTS
docker run -it --rm --env-file /etc/ichnaeaenv.txt --network="host" mozilla/location shell
/app/ichnaea/scripts/apikey.py create test
exit
curl -H "X-Forwarded-For: 81.2.69.192" http://localhost:8000/v1/geolocate?key=test