Fresh install: no authenticated files under /var/rcynic/data/

[fendemann]

I followed the installation instruction to get a new xenial 16.04 RPKI Relying Party Cache.

While installing the rpki-rp package I get the following output:

root@test-rpki:~# apt install rpki-rp … Adding user postgres to group ssl-cert

Creating config file /etc/postgresql-common/createcluster.conf with new version

Creating config file /etc/logrotate.d/postgresql-common with new version Building PostgreSQL dictionaries from installed myspell/hunspell packages... Removing obsolete dictionary files: postgresql-9.5 (9.5.5-0ubuntu0.16.04) wird eingerichtet ... Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78. Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78. Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78. Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78. Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78. Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78. Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78. Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78. Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78. Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78. Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78. Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78. Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78. Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78. Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78. Creating new cluster 9.5/main ... config /etc/postgresql/9.5/main data /var/lib/postgresql/9.5/main locale de_DE.UTF-8 socket /var/run/postgresql port 5432

…

Creating config file /etc/default/sysstat with new version update-alternatives: /usr/bin/sar.sysstat wird verwendet, um /usr/bin/sar (sar) im automatischen Modus bereitzustellen update-inetd (4.43) wird eingerichtet ... python-tz (2014.10~dfsg1-0ubuntu2) wird eingerichtet ... rrdtool (1.5.5-4) wird eingerichtet ... xinetd (1:2.3.15-6) wird eingerichtet ... rpki-rp (1.0.1476127742~xenial) wird eingerichtet ... Initialized PostgreSQL driver, pw pwd.struct_passwd(pw_name='postgres', pw_passwd='x', pw_uid=111, pw_gid=118, pw_gecos='PostgreSQL administrator,,,', pw_dir='/var/lib/postgresql', pw_shell='/bin/bash') PostgreSQL driver changing EUID from 0 to 1001 PostgreSQL driver changing EUID from 1001 to 0 PostgreSQL driver opening connection to database postgres PostgreSQL driver changing EUID from 0 to 111 PostgreSQL driver changing EUID from 111 to 0 PostgreSQL driver executing "SELECT COUNT() FROM pg_database\nJOIN pg_roles ON pg_database.datdba = pg_roles.oid\nWHERE pg_roles.rolname = 'rpki'\n" None PostgreSQL driver executing "CREATE ROLE rpki LOGIN PASSWORD ‚xxx deleted xxx" None PostgreSQL driver executing "SELECT COUNT() FROM pg_database WHERE datname = 'rpki'" None PostgreSQL driver executing 'CREATE DATABASE rpki OWNER rpki' None Operations to perform: Apply all migrations: rcynicdb Running migrations: Rendering model states... DONE Applying rcynicdb.0001_initial... OK Applying rcynicdb.0002_auto_20160227_2003... OK Applying rcynicdb.0003_auto_20160301_0333... OK Trigger für libc-bin (2.23-0ubuntu5) werden verarbeitet ... Trigger für systemd (229-4ubuntu13) werden verarbeitet ... Trigger für ureadahead (0.100.0-19) werden verarbeitet ... Trigger für ufw (0.35-0ubuntu2) werden verarbeitet ...

The installation seems ok.

root@test-rpki:~# crontab -l -u rpki MAILTO=root 33 exec /usr/bin/rcynic-cron

root@test-rpki:~# date Mo 2. Jan 10:23:34 CET 2017 root@test-rpki:~# ll /var/rcynic/data/ total 8 drwxr-xr-x 2 rpki rpki 4096 Okt 10 22:09 ./ drwxr-xr-x 4 root root 4096 Jan 2 10:18 ../

After waiting and monitoring for the rcynic-cron job I only get the unauthenticated directory, the authenticated one is missing als after the next cron job run.

root@test-rpki:~# tail -f /var/log/syslog Jan 2 10:33:01 test-rpki CRON[6468]: (rpki) CMD (exec /usr/bin/rcynic-cron)

root@test-rpki:~# less /var/log/rpki/rcynic.log 2017-01-02 09:33:09 rcynic[6469]: Network I/O error for https://gogl-rpki.rarc.net/rrdp/notify.xml: [Errno 111] Connection refused

root@test-rpki:~# ll /var/rcynic/data/ total 80 drwxr-xr-x 3 rpki rpki 4096 Jan 2 10:38 ./ drwxr-xr-x 4 root root 4096 Jan 2 10:18 ../ -rw-rw-r-- 1 rpki rpki 0 Jan 2 10:33 lock -rw-rw-r-- 1 rpki rpki 69115 Jan 2 11:08 rcynic.xml drwxrwxr-x 11 rpki rpki 4096 Jan 2 10:33 unauthenticated/

root@test-rpki:~# ll /var/rcynic/data/unauthenticated/ total 44 drwxrwxr-x 11 rpki rpki 4096 Jan 2 10:33 ./ drwxr-xr-x 3 rpki rpki 4096 Jan 2 10:38 ../ drwxrwxr-x 3 rpki rpki 4096 Jan 2 10:33 ca0.rpki.net/ drwxrwxr-x 4 rpki rpki 4096 Jan 2 10:33 localcert.ripe.net/ drwxrwxr-x 3 rpki rpki 4096 Jan 2 10:33 msft-rpki.trafficmanager.net/ drwxrwxr-x 3 rpki rpki 4096 Jan 2 10:33 repository.lacnic.net/ drwxrwxr-x 3 rpki rpki 4096 Jan 2 10:33 rpki.afrinic.net/ drwxrwxr-x 4 rpki rpki 4096 Jan 2 10:33 rpki.apnic.net/ drwxrwxr-x 3 rpki rpki 4096 Jan 2 10:33 rpki-pilot.lab.dtag.de/ drwxrwxr-x 3 rpki rpki 4096 Jan 2 10:33 rpki-repository.nic.ad.jp/ drwxrwxr-x 3 rpki rpki 4096 Jan 2 10:33 rpki.ripe.net/

Any thoughts ...

[sraustein]

rcynic now stores most of its data in SQL rather than in plain files. The unauthentictated directory still exists because that's where rcynic downloads files retrieved via rsync, but everything else is in SQL. Try running a program like scan_roas which does something using the authenticated data.

[fendemann]

I think, scan_roas help will need an update. The help still describes the search on an authenticated tree.

fen@test-rpki:~$ scan_roas --help usage: scan_roas [-h] [-c CONFIG] [rcynic_dir]

Search an authenticated result tree from an rcynic run for ROAs, and prints out the signing time, ASN, and prefixes for each ROA, one ROA per line.

positional arguments: rcynic_dir rcynic authenticated output directory

optional arguments: -h, --help show this help message and exit -c CONFIG, --config CONFIG override default location of configuration file

I will try to look in the SQL directly tomorrow.

Thx

Von: Rob Austein [mailto:notifications@github.com] Gesendet: Montag, 2. Januar 2017 14:42 An: dragonresearch/rpki.net Cc: Endemann, Frank; Author Betreff: Re: [dragonresearch/rpki.net] Fresh install: no authenticated files under /var/rcynic/data/ (#848)

rcynic now stores most of its data in SQL rather than in plain files. The unauthentictated directory still exists because that's where rcynic downloads files retrieved via rsync, but everything else is in SQL. Try running a program like scan_roas which does something using the authenticated data.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/dragonresearch/rpki.net/issues/848#issuecomment-269973899, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AXuMY_AnWtgqCmLAIHmlRBgG73D084Hyks5rOP6PgaJpZM4LY9nl.

[sraustein]

scan_roas et al still allow one to specify an old-style authenticated directory, but it's optional (that's what the square brackets in the usage summary mean). If one doesn't supply a directory argument, those programs pull their information from SQL.

If you have scripts of your own that require the old format, you might want to look at the rcynic-dump script in the rp/rcynic/ source directory. We don't currently install that, because it's just a backwards compatibility hack, but we could promote it if users need it.

[fendemann]

Ok, I will try the tools. Maybe the error massages in the installation process

Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!)${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.

need some action, but that is an other problem. Thx