rcynic-cron still looking for /etc/rcynic.conf

[sraustein]

{{{ ca.rg.net:/root# /usr/bin/rcynic-cron 04:56:21: rcynic: Couldn't load config file /etc/rcynic.conf Program /usr/bin/rcynic -c /etc/rcynic.conf exited with status 1 }}}

Trac ticket #786 component testbed priority minor, owner None, created by randy on 2016-03-19T05:01:22Z, last modified 2016-04-22T01:58:20Z

[sraustein]

rcynic-cron updated to run rcynicng. Lightly tested, but I think everything works, with two exceptions:

• There is something wrong with the rpkigui-rcynic script, it's having trouble talking to its database, don't know why. So this script is disabled for the moment, and the GUI won't see RP status.
• rcynicng currently logs only to stdout/stderr, it doesn't know about syslog yet. Fixing this should be straightforward, but for the moment you might want to edit the crontab (crontab -u rpki -e) to redirect rcynic-cron's output to a log file instead of sending it as email every hour.

Trac comment by sra on 2016-03-20T03:21:31Z

[sraustein]

{{{ ca.rg.net:/root# rpkic version 0.6321 }}}

{{{ ca.rg.net:/root# /usr/bin/rcynic-cron Traceback (most recent call last): File "/usr/bin/rcynic", line 1457, in main() File "/usr/bin/rcynic", line 1427, in main django.core.management.call_command("migrate", verbosity = 0, interactive = False) File "/usr/lib/python2.7/dist-packages/django/core/management/init.py", line 120, in call_command return command.execute(_args, _defaults) File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 445, in execute output = self.handle(_args, _options) File "/usr/lib/python2.7/dist-packages/django/core/management/commands/migrate.py", line 93, in handle executor = MigrationExecutor(connection, self.migration_progress_callback) File "/usr/lib/python2.7/dist-packages/django/db/migrations/executor.py", line 19, in init self.loader = MigrationLoader(self.connection) File "/usr/lib/python2.7/dist-packages/django/db/migrations/loader.py", line 47, in init self.build_graph() File "/usr/lib/python2.7/dist-packages/django/db/migrations/loader.py", line 191, in build_graph self.applied_migrations = recorder.applied_migrations() File "/usr/lib/python2.7/dist-packages/django/db/migrations/recorder.py", line 59, in applied_migrations self.ensure_schema() File "/usr/lib/python2.7/dist-packages/django/db/migrations/recorder.py", line 49, in ensure_schema if self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor()): File "/usr/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 164, in cursor cursor = self.make_cursor(self._cursor()) File "/usr/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 135, in _cursor self.ensure_connection() File "/usr/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection self.connect() File "/usr/lib/python2.7/dist-packages/django/db/utils.py", line 98, in exit six.reraise(dj_exc_type, dj_exc_value, traceback) File "/usr/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection self.connect() File "/usr/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 119, in connect self.connection = self.get_new_connection(conn_params) File "/usr/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 176, in get_new_connection connection = Database.connect(conn_params) File "/usr/lib/python2.7/dist-packages/psycopg2/init**.py", line 179, in connect connection_factory=connection_factory, async=async) django.db.utils.OperationalError: FATAL: Peer authentication failed for user "rpki"

Program /usr/bin/rcynic exited with status 1 }}}

Trac comment by randy on 2016-03-20T05:40:55Z

[sraustein]

rcynic-cron has to run as user rpki, which is what the crontab does. If you want to run it manually while root, you need to do something like

{{{ sudo -u rpki rcynic-cron }}}

Trac comment by sra on 2016-03-20T06:14:25Z

[sraustein]

Oh, and just to make things more fun, RRDP client code which runs fine on MacOSX gets some kind of SSLv3 (sic, at least, that's how it's reported, but it might be a lie) error on trusty.

Trac comment by sra on 2016-03-20T07:14:29Z

[sraustein]

rcynic cron no longer looking for anything!

{{{ ca.rg.net:/root# rpkic version 0.6323 ca.rg.net:/root# crontab -l -u rpki MAILTO=root }}}

but i do not think this is a long-term solution

Trac comment by randy on 2016-03-21T05:46:50Z

[sraustein]

rcynic cron no longer looking for anything!

{{{ ca.rg.net:/root# crontab -l -u rpki MAILTO=root }}}

This is some kind of (one-time?) error updating an existing crontab. Not sure what caused it, will chase if reoccurs.

Repair for a single instance of this is simple, do:

{{{ crontab -u rpki -r }}}

before your next apt-get update and it should just reinstall a good crontab.

That said, I have not yet resolved the TLS mess talking to rrdp.ripe.net. Python/OpenSSL/Ubuntu version dependent Charlie Fox, not related to our code at all, we're just innocent victims. You can spot the affected systems by doing:

{{{ python -c 'from urllib import urlopen; urlopen("https://rrdp.ripe.net/notification.xml")' }}}

If this exits without comment, the TLS implementation used by this Python's ssl module is OK: if it throws a backtrace that whines about misnamed OpenSSL API errors (SSLv3, etc), it's borked.

I spent a few minutes looking at a tshark dump of success and failure cases, and am pretty sure this is a TLS algorithm negotiation error. Basically, rrdp.ripe.net is using a TLS configuration that's beyond the capabilities of the Python 2.7 ssl module on some systems.

Trusty is borked. Jessie, Mavericks, and FreeBSD are OK. Haven't tested others yet.

Problem doesn't appear to be quite as simple as just "it's an old version of SSL", there's something specific to how Python is using OpenSSL involved here, because w3m on the same systems works.

Options I can think of would be:

1) Use Debian Jessie or a newer version of Ubuntu;

2) Switch the RRDP client to using Tornado's alternate TLS client based on PyCurl instead of the standard ssl module;

3) Ask RIPE NCC to tune down the paranoia on this particular server for a few months while software catches up.

None of these are free. (3) is easiest but politically distasteful.

We want to get this code running on platforms other than trusty in any case, so if I have to put work into this that's where I'd prefer to put it.

Trac comment by sra on 2016-03-21T22:29:59Z

[sraustein]

crontab -u rpki -r

done. still waiting for propagation.

{{{ ca.rg.net:/root# rpkic version 0.6323

ca.rg.net:/root# python -c 'from urllib import urlopen; urlopen("https://rrdp.ripe.net/notification.xml")' Traceback (most recent call last): File "", line 1, in File "/usr/lib/python2.7/urllib.py", line 87, in urlopen return opener.open(url) File "/usr/lib/python2.7/urllib.py", line 208, in open return getattr(self, name)(url) File "/usr/lib/python2.7/urllib.py", line 437, in open_https h.endheaders(data) File "/usr/lib/python2.7/httplib.py", line 975, in endheaders self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line 835, in _send_output self.send(msg) File "/usr/lib/python2.7/httplib.py", line 797, in send self.connect() File "/usr/lib/python2.7/httplib.py", line 1182, in connect self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file) File "/usr/lib/python2.7/ssl.py", line 487, in wrap_socket ciphers=ciphers) File "/usr/lib/python2.7/ssl.py", line 243, in init self.do_handshake() File "/usr/lib/python2.7/ssl.py", line 405, in do_handshake self._sslobj.do_handshake() IOError: [Errno socket error] [Errno 1] _ssl.c:510: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure }}}

for the moment, asking tim to deparanoia the pilot may be the most expedient, if disgusting, path.

barring that, personally i would go to ubuntu 16 beta, as that is a path i will be following in the long run. though i was planning it for a month from now when 16 goes release.

Trac comment by randy on 2016-03-21T22:38:41Z

[sraustein]

{{{ ca.rg.net:/root# rpkic version 0.6324

ca.rg.net:/root# crontab -l -u rpki MAILTO=root 5 * * * * exec /usr/bin/rcynic-cron }}}

Trac comment by randy on 2016-03-21T22:41:22Z

[sraustein]

OK, I can now confirm that

• The RRDP client works as expected under (pre-release) Debian Stretch; and
• No backports are necessary on Stretch, and the new RPKI code pretty much just drops in.

Backporting modern versions of python-django and python-tornado to Wheezy is probably more trouble than it's worth.

Backporting those same packages to Jessie is a marginal case: I have not yet encountered anything in principal which would make this impossible, but the number of backports necessary is rather large (transitive closure of original requirements), and some of the packages themselves are rather large. We may have to do this anyway, but it's not going to be fun.

Yes, I know that Randy cares more about Ubuntu than about Debian, but some of the same issues may arise. The current port of the new code to Trusty uses backports from Xenial; in theory that should work for any Ubuntu release since Trusty, but this can spin out of control pretty quickly (see "transitive closure").

At any rate: I think we can consider this confirmation that the TLS problem is what we thought it was, not a subtle bug in our own code.

Remaining unresolved issue: rcynic (probably) should fail over when RRDP bombs out. I don't much care for the model in which the CA gets to list 27 URIs and the RP is require to check each one of them in turn, but we may need to support that, at least as a configurable option. Ick.

Trac comment by sra on 2016-03-22T23:44:06Z

[sraustein]

why should i not just run xenial beta?

Trac comment by randy on 2016-03-22T23:48:05Z

[sraustein]

why should i not just run xenial beta?

beta-1 out for a while, but i can only find desktop and 'alternate' images, whatever the heck they are.

beta-2, final beta, due out in 24-48 hours

Trac comment by randy on 2016-03-22T23:55:43Z

[sraustein]

why should i not just run xenial beta?

beta-1 out for a while, but i can only find desktop and 'alternate' images, whatever the heck they are.

beta-2, final beta, due out in 24-48 hours

OK, I'll plan to test b2.

Trac comment by sra on 2016-03-22T23:57:59Z

[sraustein]

Closed with resolution fixed