Move robotoff to a separate VM

[stephanegigandet]

This is a tracking bug for the move of robotoff and associated services (postgres, elasticsearch, graphana) from off2 to the new VM #20 on the new OVH machines.

robotoff was turned off on off2 on Saturday Feb 6th 2021 as the postgres database was hitting the disk too much, causing the mongodb database to be barely responsive. (see discussion in Slack: https://openfoodfacts.slack.com/archives/C1FPYCWM7/p1612607952024000 )

[stephanegigandet]

Doc for robotoff: https://github.com/openfoodfacts/robotoff/wiki

https://github.com/openfoodfacts/robotoff/wiki/Architecture

https://github.com/openfoodfacts/robotoff/wiki/Maintenance

[stephanegigandet]

One thing to note is that robotoff sends queries directly to mongodb on off2. But mongodb on off2 is only reachable from off1 and off2, so we will need to make it available from the new robotoff VM.

[stephanegigandet]

I'm going to create CNAMEs for robotoff.openfoodfacts.org and monitoring.openfoodfacts.org to point them to the new VM.

[stephanegigandet]

Current nginx config on off2 for robotoff + monitoring:

stephane@off2:/etc/nginx/sites-enabled$ cat robotoff-proxy 
server {
    listen 80;
    listen [::]:80;
    server_name off2.free.org;
    client_body_timeout 120s;
    client_header_timeout 120s;

    access_log /srv/off/logs/robotoff.nginx.access2.log;
    error_log /srv/off/logs/robotoff.nginx.error2.log;

    gzip on;
    gzip_min_length 1000;

    location / {
        proxy_pass http://127.0.0.1:5500$request_uri;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_read_timeout 90;
        client_max_body_size 15M;
    }

    location /api/v1/ann {
        proxy_pass http://49.12.34.189:5501$request_uri;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_read_timeout 90;
        client_max_body_size 1M;
    }
}

stephane@off2:/etc/nginx/sites-enabled$ more monitoring 
server {
    if ($host = monitoring.openfoodfacts.org) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen 80;
    listen [::]:80;
    server_name monitoring.openfoodfacts.org;

    location ~ /^/(.well-known)/ {
                try_files $uri/ =404;
        }
    return 301 https://monitoring.openfoodfacts.org;

}

server {
    client_body_timeout 120s;
    client_header_timeout 120s;

    server_name monitoring.openfoodfacts.org;

    listen 443 ssl http2;

    #include snippets/ssl.monitoring.openfoodfacts.org;
    #include snippets/ssl-params.conf;

    access_log /srv/off/logs/monitoring.nginx.access2.log;
    error_log /srv/off/logs/monitoring.nginx.error2.log;

    gzip on;
    gzip_min_length 1000;

   location / {
    proxy_pass http://127.0.0.1:3000$request_uri;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_read_timeout 90;
    client_max_body_size 512M;
    }

    ssl_certificate /etc/letsencrypt/live/monitoring.openfoodfacts.org/fullchain.pem; # managed by 
Certbot
    ssl_certificate_key /etc/letsencrypt/live/monitoring.openfoodfacts.org/privkey.pem; # managed b
y Certbot
}

[stephanegigandet]

CNAMEs created:

robotoff IN CNAME ovh1.openfoodfacts.org. monitoring IN CNAME ovh1.openfoodfacts.org.

[stephanegigandet]

I'm installing mongodb on robotoff, so that we can set up the connection to the mongodb on off2

Instructions from https://docs.mongodb.com/v4.4/tutorial/install-mongodb-on-debian/

wget -qO - https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add -

echo "deb http://repo.mongodb.org/apt/debian stretch/mongodb-org/4.4 main" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list

sudo apt-get update

Some issues:

root@robotoff:/home/stephanegigandet# sudo apt-get install -y mongodb-org Reading package lists... Done Building dependency tree
Reading state information... Done Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation:

The following packages have unmet dependencies: mongodb-org : Depends: mongodb-org-shell but it is not going to be installed Depends: mongodb-org-server but it is not going to be installed Depends: mongodb-org-mongos but it is not going to be installed E: Unable to correct problems, you have held broken packages. root@robotoff:/home/stephanegigandet# apt-get install mongodb-org-mongos mongodb-org-server mongodb-org-shell mongodb-org-tools Reading package lists... Done Building dependency tree
Reading state information... Done Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation:

The following packages have unmet dependencies: mongodb-org-mongos : Depends: libcurl3 (>= 7.16.2) but it is not installable mongodb-org-server : Depends: libcurl3 (>= 7.16.2) but it is not installable mongodb-org-shell : Depends: libcurl3 (>= 7.16.2) but it is not installable E: Unable to correct problems, you have held broken packages.

root@robotoff:/home/stephanegigandet# apt-get install libcurl3 Reading package lists... Done Building dependency tree
Reading state information... Done Package libcurl3 is not available, but is referred to by another package. This may mean that the package is missing, has been obsoleted, or is only available from another source However the following packages replace it: libcurl4

[stephanegigandet]

41 echo "deb http://deb.debian.org/debian/ stretch main" | sudo tee /etc/apt/sources.list.d/debian-stretch.list 42 apt-get update 43 apt-get install libcurl3

[stephanegigandet]

apt-get install -y mongodb-org-mongos mongodb-org-server mongodb-org-shell mongodb-org-tools

root@robotoff:/home/stephanegigandet# mongo --version MongoDB shell version v4.4.3 Build Info: { "version": "4.4.3", "gitVersion": "913d6b62acfbb344dde1b116f4161360acd8fd13", "openSSLVersion": "OpenSSL 1.1.1d 10 Sep 2019", "modules": [], "allocator": "tcmalloc", "environment": { "distmod": "debian92", "distarch": "x86_64", "target_arch": "x86_64" } }

[stephanegigandet]

mongo is installed on the robotoff VM, and I opened the port on off2 so that it can connect to the mongodb database on off2

[alexgarel]

Robotoff is now in a docker.