ooniprobe 1.5.1-1 (debian) daily cronjob fails

[anadahz]

Debian stretch ooniprobe 1.5.1-1

---

It seems that the ooniprobe cronjob fails:

/var/log/ooni/ooniprobe.log output:

[-] Looking up your IP address via torproject
[-] Found your IP via a GeoIP service
[-] Fetching required net test inputs...
[-] Looking up collector and test helpers
[-] [!] Lookup failed. Retrying.
[-] [!] Lookup failed. Retrying.
[-] [!] Lookup failed. Retrying.
[-] [!] Failed. Giving up.
[-] Traceback (most recent call last):
[-]   File "/usr/lib/python2.7/dist-packages/ooni/backend_client.py", line 173, in lookupTestCollector
[-]     query={'net-tests': net_tests})
[-]   File "/usr/lib/python2.7/dist-packages/ometa/protocol.py", line 53, in dataReceived
[-]     self._parser.receive(data)
[-]   File "/usr/lib/python2.7/dist-packages/ometa/tube.py", line 41, in receive
[-]     status = self._interp.receive(data)
[-]   File "/usr/lib/python2.7/dist-packages/ometa/interp.py", line 48, in receive
[-]     for x in self.next:
[-]   File "/usr/lib/python2.7/dist-packages/ometa/interp.py", line 166, in apply
[-]     for x in self._apply(f, ruleName, argvals):
[-]   File "/usr/lib/python2.7/dist-packages/ometa/interp.py", line 104, in _apply
[-]     for x in rule():
[-]   File "/usr/lib/python2.7/dist-packages/ometa/interp.py", line 245, in parse_Or
[-]     for x in self._eval(subexpr):
[-]   File "/usr/lib/python2.7/dist-packages/ometa/interp.py", line 230, in parse_And
[-]     for x in self._eval(subexpr):
[-]   File "/usr/lib/python2.7/dist-packages/ometa/interp.py", line 423, in parse_Action
[-]     val = eval(expr.data, self.globals, self._localsStack[-1])
[-]   File "<string>", line 1, in <module>
[-]   File "/usr/lib/python2.7/dist-packages/txsocksx/client.py", line 153, in serverResponse
[-]     raise e.socks5ErrorMap.get(status)()
[-] HostUnreachable
[-] [!] Could not start the director
[-] [!] Could not find a valid collector.
[-] Try with a different bouncer, specify a collector with -c or disable reporting to a collector with -n.
[-] Main loop terminated.

Contents of: /etc/cron.daily/ooniprobe

#!/bin/sh

set -e

sh -n /etc/default/ooniprobe 2>/dev/null || exit 0

. /etc/default/ooniprobe
test "$OONIPROBE_RUN_DAILY_TEST" = 'true' || exit 0

test -f /var/lib/ooni/decks/debian-default/user.deck || exit 0

fix_permissions() {
        local user group mode path

        read user group mode path
        chown $user:$group $path
        chmod $mode $path
}

if ! [ -d /var/log/ooni ]; then
        mkdir /var/log/ooni
        { dpkg-statoverride --list /var/log/ooni ||
          echo Debian-ooni adm 750 /var/log/ooni
        } | fix_permissions
fi

# Remove incomplete reports from former runs
su Debian-ooni -c 'oonireport -f /etc/ooniprobe.conf status' |
sed -n -e '/^Incomplete reports/,$s/^\* //p' | while read path; do
        rm -f "$path"
done

# Try to upload unsent reports
su Debian-ooni -c 'oonireport -f /etc/ooniprobe.conf upload' >> /var/log/ooni/oonireport.log

# Run daily test
flock --nonblock /run/oonprobe.daily.lock su Debian-ooni -c "cd /var/lib/ooni/reports && ooniprobe -f /etc/ooniprobe.conf -i /var/lib/ooni/decks/debian-default/user.deck" > /dev/null

Contents of: /var/lib/ooni/decks/debian-default/user.deck

- options:
    annotations: null
    bouncer: null
    collector: null
    no-collector: 0
    no-geoip: 0
    no-yamloo: 0
    reportfile: null
    subargs: &id001 []
    test_file: manipulation/http_invalid_request_line
    verbose: 0
- options:
    annotations: null
    bouncer: null
    collector: null
    no-collector: 0
    no-geoip: 0
    no-yamloo: 0
    reportfile: null
    subargs: *id001
    test_file: manipulation/http_header_field_manipulation
    verbose: 0
- options:
    annotations: null
    bouncer: null
    collector: null
    no-collector: 0
    no-geoip: 0
    no-yamloo: 0
    reportfile: null
    subargs: [-f, /var/lib/ooni/decks/debian-default/citizenlab-urls-global.txt]
    test_file: blocking/web_connectivity
    verbose: 0

[hellais]

The issue is in looking up the Hidden Service address. Usually this will happen if your system clock is not accurate. Is that the case?

[anadahz]

This is not the case. this system uses the tor client for other processes without any issues.

[hellais]

@anadahz hum, that's weird, are you able to reach the bouncer with torsocks curl http://nkvphnp3p6agi5qq.onion?

Note: the tor client doesn't require such an up to date system clock to work properly when not connecting to hidden services.

[anadahz]

@hellais yes i'm able to access the bouncer via torsocks

The following daily cronjobs seems to work as expected, after this day without any change. Adding an option in ooniprobe to retry bringing up tor again rather than failing after the first attempt could have been useful for such cases.

[hellais]

@anadahz I am not sure that restarting tor would fix the issue in this case. If the cause of the problem was in fact a clock with the wrong time restarting tor would have no effect.

We also make the assumption at this point that we do have a functioning Tor and from the looks of it that is indeed the case since the error message returned is returned by the SOCKS port when a hidden service is not reachable.

Adding logic for restarting tor would mean that we would also restart tor in the cases where the bouncer is really offline or tor is not configured properly so I don't think that is a satisfactory solution.

[kaerumy]

I'm getting different reason why it fails for same version of ooniprobe on Ubuntu/Debian:

test -f /var/lib/ooni/decks/debian-default/user.deck || exit 0

But the deck generated on installation for .my users is:

/var/lib/ooni/decks/deck-my/default-user.deck

[hellais]

@kaerumy do you have some error log output or provide more information on what the failure is?

[kaerumy]

This is the output when selecting daily configuration for debian package. Will need to look at the .deb file I guess why oonideckgen fails, I don't see any log or other output that shows what failed.

Subsequent runs of oonideckgen doesn't show any errors and generates and recommends usage of the deck-my config.

$ sudo dpkg-reconfigure ooniprobe
WARNING: Passing command line arguments is deprecated
Downloading https://github.com/TheTorProject/ooni-probe/releases/download/v1.5.1/ooni-resources.tar.gz to /var/lib/ooni
Written GeoIP files to /var/lib/ooni/GeoIP
Written resources files to /var/lib/ooni/resources
oonideckgen failed.
WARNING: ooniprobe will not be able to run daily.
Looking up your IP address via ubuntu
Found your IP via a GeoIP service
Deck written to /var/lib/ooni/decks/deck-my/default-user.deck
Run ooniprobe like so:
ooniprobe -i /var/lib/ooni/decks/deck-my/default-user.deck

A quick guess is that the failure is somewhere at the cron.daily/ooniprobe generation part, where it tries to generate it before the country default user deck is created.

[hellais]

Can we close this issue?

[hellais]

I am closing this as done.