Open PGP support via RNP adapter

[skalee]

Extracted from #23.

Implement low-level methods in GPGME adapter which actually perform cryptographic work.

• [x] Support message encryption
• [x] Support calculation of detached signatures
• [x] Support message encryption and signing in one Open PGP message
• [x] Make required software versions available in Travis

[skalee]

@ronaldtse Should I care about ruby-rnp 0.2 at all, or only about the freshly rewritten version?

[ronaldtse]

It should be the latest version 1.0. Thanks!

[skalee]

Under some circumstances, ruby-rnp gem may crash Ruby interpreter. These ain't errors one can catch with begin-rescue-end block. And unfortunately, such crash may cause a data loss, e.g. in Sidekiq all currently processed jobs will be lost.

There is an alternative solution. We can spawn an RNP process (rnp --sign --detach), pass data to stdin, and read result from stdout. This is a common pattern. For example, https://github.com/minimagick/minimagick does it.

Good idea or bad idea?

cc @ronaldtse @dewyatt

[dewyatt]

I wouldn't really recommend it personally, but it may work for you. Be aware that RNP may still have some rough edges with stdin/stdout (https://github.com/riboseinc/rnp/issues/542 https://github.com/riboseinc/rnp/issues/572).

[ronaldtse]

Agree with @dewyatt, the whole point is to not spawn a shell. However, whatever works, works.

On he other hand, @skalee can you post the crash conditions to ruby-rnp so @dewyatt can try to fix them? Thanks!

[skalee]

@ronaldtse see riboseinc/ruby-rnp#9.

[ronaldtse]

@skalee RNP has had some changes recently that improves its stability. Could you help try it out again? Thanks!

[skalee]

Thanks for update, @ronaldtse! I'll try again.

[ronaldtse]

@skalee the RNP yum package is available here now:

https://github.com/riboseinc/yum

Thanks to @dewyatt 's latest changes, the RNP/ruby-rnp pair seems ready to use, it should be more straightforward than before now.

[skalee]

Thanks! It appears that ruby-rnp behaves much more stable now. I am able to load keys now.

FYI The public keyring created by GnuPG (KBX) is readable, the secret one (G10) is not, but nothing crashes. I understand that's just an incompatibility, and expected. Keyrings created by Rnp (both in GPG format) are both readable.

[ronaldtse]

Well RNP is supposed to be fully compatible, so let’s ping @dewyatt and @ni4 here to see how we can make This instance of G10 loadable.

[skalee]

Then FYI GnuPG keys have been generated in <project root>/tmp/gpgme by invoking bundle exec rake generate_gpg_keys. Rakefile was at following revision: bbdf6cb6cba788494dea93dcf5f342b62df2813b.

[ni4]

@skalee What is the algorithm of that key(s)? We doesn't seem to support ECC g10 keys yet.

[dewyatt]

@ronaldtse

Well RNP is supposed to be fully compatible

Remember there's a few incompatibilities, as mentioned in https://github.com/riboseinc/enmail/issues/36#issuecomment-385564861, including the one @ni4 mentioned above.

(I'm a bit busy refreshing my rust skills to prepare for the rust bindings, but can set that aside if needed)

[ronaldtse]

Indeed! I really meant "mostly compatible"; we will need to discuss what are the important things to be compatible with.

[skalee]

@ni4

<GnupgKeyParms format="internal">
%no-protection
Key-Type: DSA
Key-Length: 2048
Subkey-Type: ELG-E
Subkey-Length: 2048
Name-Real: Some Arbitrary Key
Name-Email: whatever@example.test
Name-Comment: Without passphrase
Expire-Date: 0
%commit

Key-Type: DSA
Key-Length: 2048
Subkey-Type: ELG-E
Subkey-Length: 2048
Name-Real: Cato Elder
Name-Email: cato.elder@example.test
Name-Comment: Without passphrase
Expire-Date: 0
%commit

Key-Type: DSA
Key-Length: 2048
Subkey-Type: ELG-E
Subkey-Length: 2048
Name-Real: Roman Senate
Name-Email: senate@example.test
Name-Comment: Without passphrase
Expire-Date: 0
</GnupgKeyParms>

These keys are meant for testing only, I don't mind using different algorithms if it solves the issue.

[ni4]

@skalee Thanks, I'll check. Actually these ones should work.

[skalee]

Also, I was using GnuPG 2.2.8 from Homebrew.

[ni4]

@skalee I tried to reproduce the problem with those keys but without luck. Can you please describe what do you mean under the 'g10 keyring is not readable'? Probably, there is some misunderstanding or I tried the wrong operations.

[dewyatt]

@skalee I took a look and was able to load the rake-generated keys with:

require 'rnp'

# Note the formats here
rnp = Rnp.new('KBX', 'G10')

# KBX
rnp.load_keys(
  input: Rnp::Input.from_path('/tmp/gpg/enmail/tmp/gpgme/pubring.kbx'),
  public_keys: true,
  format: 'KBX'
)

# G10
rnp.load_keys(
  input: Rnp::Input.from_path('/tmp/gpg/enmail/tmp/gpgme/private-keys-v1.d'),
  secret_keys: true,
  format: 'G10'
)

Was there some other operation failing (signing, etc)?

[skalee]

I have prepared a following script:

require "fileutils"
require "gpgme"
require "gpgme/version"
require "rnp"
require "shellwords"

#### SETUP

homedir = File.expand_path("./issue-gpghome", __dir__)

FileUtils.mkdir_p homedir

::GPGME::Engine.home_dir = homedir

puts "\n*** SOFTWARE VERSIONS ***"

puts <<-VERSIONS
Ruby:        #{RUBY_VERSION}
RNP gem:     #{::Rnp::VERSION}
RNP:         #{`rnp --version 2>&1 | head -n 1`.chomp}
GPGME gem:   #{::GPGME::VERSION}
GPG:         #{`gpg --version | head -n 1`.chomp}
VERSIONS

puts "\n*** CREATING HOMEDIR ***"
puts homedir
puts "\n*** GENERATING KEYS ***"

::GPGME::Ctx.new.genkey(<<~SCRIPT)
  <GnupgKeyParms format="internal">
  %no-protection
  Key-Type: DSA
  Key-Length: 2048
  Subkey-Type: ELG-E
  Subkey-Length: 2048
  Name-Real: Some Arbitrary Key
  Name-Email: whatever@example.test
  Name-Comment: Without passphrase
  Expire-Date: 0
  %commit

  Key-Type: DSA
  Key-Length: 2048
  Subkey-Type: ELG-E
  Subkey-Length: 2048
  Name-Real: Cato Elder
  Name-Email: cato.elder@example.test
  Name-Comment: Without passphrase
  Expire-Date: 0
  %commit

  Key-Type: DSA
  Key-Length: 2048
  Subkey-Type: ELG-E
  Subkey-Length: 2048
  Name-Real: Roman Senate
  Name-Email: senate@example.test
  Name-Comment: Without passphrase
  Expire-Date: 0
  </GnupgKeyParms>
SCRIPT

puts `gpg --list-keys --homedir #{Shellwords.escape homedir}`

#### PROBLEM

puts "\n*** HOMEDIR INFO ACCORDING TO RNP ***"
info = Rnp.homedir_info(homedir)
puts info.inspect
rnp = Rnp.new

%i[public secret].each do |keyring|
  puts "\n*** LOADING #{keyring.upcase} KEYRING ***"
  format = info[keyring][:format]
  path = info[keyring][:path]
  rnp.load_keys(format: format, input: Rnp::Input.from_path(path))
  puts rnp.keyids.inspect
end

On my computer, env LC_ALL=C ruby issue.rb gives following output:

*** SOFTWARE VERSIONS ***
Ruby:        2.5.1
RNP gem:     1.0.0
RNP:         rnp 0.0.0+git20180705.2a59b7c
GPGME gem:   2.0.16
GPG:         gpg (GnuPG) 2.2.8

*** CREATING HOMEDIR ***
/Users/skale/Projekty/ribose/enmail/issue-gpghome

*** GENERATING KEYS ***
gpg: WARNING: unsafe permissions on homedir '/Users/skale/Projekty/ribose/enmail/issue-gpghome'
gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   3  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 3u
/Users/skale/Projekty/ribose/enmail/issue-gpghome/pubring.kbx
-------------------------------------------------------------
pub   dsa2048 2018-07-05 [SCA]
      77C8C829865164B89FF730C91F7401F28E6B44C2
uid           [ultimate] Some Arbitrary Key (Without passphrase) <whatever@example.test>
sub   elg2048 2018-07-05 [E]

pub   dsa2048 2018-07-05 [SCA]
      C5F3C830D8CC2E64129BFECB6299ECBA9129D8E8
uid           [ultimate] Cato Elder (Without passphrase) <cato.elder@example.test>
sub   elg2048 2018-07-05 [E]

pub   dsa2048 2018-07-05 [SCA]
      76BF3160605AAA89E0EBA8507726FDFF621BA17C
uid           [ultimate] Roman Senate (Without passphrase) <senate@example.test>
sub   elg2048 2018-07-05 [E]

*** HOMEDIR INFO ACCORDING TO RNP ***
{:public=>{:format=>"KBX", :path=>"/Users/skale/Projekty/ribose/enmail/issue-gpghome/pubring.kbx"}, :secret=>{:format=>"G10", :path=>"/Users/skale/Projekty/ribose/enmail/issue-gpghome/private-keys-v1.d"}}

*** LOADING PUBLIC KEYRING ***
["1F7401F28E6B44C2", "472141A1A5F66D08", "6299ECBA9129D8E8", "EC52F09D62A99E22", "7726FDFF621BA17C", "D8F57820CDFB91FB"]

*** LOADING SECRET KEYRING ***
[do_load_keys() /tmp/rnp-20180705-22431-1ax8pkf/src/lib/rnp2.cpp:887] This key format conversion is not yet supported
Traceback (most recent call last):
    5: from issue.rb:72:in `<main>'
    4: from issue.rb:72:in `each'
    3: from issue.rb:76:in `block in <main>'
    2: from /Users/skale/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rnp-1.0.0/lib/rnp/rnp.rb:141:in `load_keys'
    1: from /Users/skale/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rnp-1.0.0/lib/rnp/utils.rb:21:in `call_ffi'
/Users/skale/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rnp-1.0.0/lib/rnp/error.rb:29:in `raise_error': rnp_load_keys failed - (rc: 0x10000003): Not implemented (Rnp::Error)

cc @dewyatt

[dewyatt]

Thanks @skalee. Instead of:

info = Rnp.homedir_info(homedir)
puts info.inspect
rnp = Rnp.new

Do:

info = Rnp.homedir_info(homedir)
puts info.inspect
rnp = Rnp.new(info[:public][:format], info[:secret][:format])

[ribose-jeffreylau]

Hi @skalee , we're going to fully migrate to RNP by next week. Would this be ready by then? Thanks!

[skalee]

@ribose-jeffreylau I really dislike idea of giving any promises or estimates about this gem. I was stumbling upon too many surprising issues so far.

AFAIR there are following major issues to resolve:

1. Actual Rnp gem integration — I hope it was only a constructor issue, I'll check that soon. In general, it is is halfway (or more) done on some branch, and should be relatively easy to finish, provided that there are no more surprises. However, it is also potentially difficult to test in Travis. I was going to rely on Docker for that, but I have little experience with this tool. I have started reading docs for that purpose already, but perhaps, I'll go with some Rnp install script for simplicity sake.
2. Correct transfer encoding for some mail parts — this is a general issue, related both to GPGME and RNP. The standard requires specific transfer encoding (otherwise, signature could be randomly invalidated), but Mail gem does not allow for that easily. I remember I had some ideas how to deal with that, but I need to recap on topic. This issue is only relevant to signed messages. See one of the checkboxes in #23.
3. Ability to select encryption & digesting algorithms. Right now it relies on default settings, whatever they are.
4. Support for password-protected keys, which is probably less important. For GPGME, it is generally done on some side branch.

Actually, this weekend I was going to spend much more time on this project than I recently did. But given the number of surprising issues I was dealing with, I don't want to give any promises.

Anyway, I am starting with 1). When it's done, it will be much easier to tell something.

[ronaldtse]

Thanks @skalee ! I know themail-gpg gem dealt with transfer encoding correctly, so feel free to check out how they do it.

For item 1, RNP can be installed via homebrew-rnp (https://github.com/riboseinc/homebrew-rnp) and our yum repo (CentOS/RHEL) https://github.com/riboseinc/yum (go here for instructions).

[skalee]

After constructor fixes according to @dewyatt, the output is as follows. The exception no longer happens, but the secret keyring still can't be loaded.

*** SOFTWARE VERSIONS ***
Ruby:        2.5.1
RNP gem:     1.0.0
RNP:         rnp 0.0.0+git20180705.2a59b7c
GPGME gem:   2.0.16
GPG:         gpg (GnuPG) 2.2.8

*** CREATING HOMEDIR ***
/Users/skale/Projekty/ribose/enmail/issue-gpghome

*** GENERATING KEYS ***
gpg: WARNING: unsafe permissions on homedir '/Users/skale/Projekty/ribose/enmail/issue-gpghome'
gpg: sprawdzanie bazy zaufania
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: poziom: 0 poprawnych:   3 podpisanych:   0 zaufanie: 0-,0q,0n,0m,0f,3u
/Users/skale/Projekty/ribose/enmail/issue-gpghome/pubring.kbx
-------------------------------------------------------------
pub   dsa2048 2018-07-08 [SCA]
      39721579A2A367B49AF6C37EA95CBD6E2214D433
uid           [    absolutne   ] Some Arbitrary Key (Without passphrase) <whatever@example.test>
sub   elg2048 2018-07-08 [E]

pub   dsa2048 2018-07-08 [SCA]
      B9048A7472CC4F2980E53EEB033FF4503ED02E33
uid           [    absolutne   ] Cato Elder (Without passphrase) <cato.elder@example.test>
sub   elg2048 2018-07-08 [E]

pub   dsa2048 2018-07-08 [SCA]
      973858851642B816F479625A344E6B598796B32B
uid           [    absolutne   ] Roman Senate (Without passphrase) <senate@example.test>
sub   elg2048 2018-07-08 [E]

*** HOMEDIR INFO ACCORDING TO RNP ***
{:public=>{:format=>"KBX", :path=>"/Users/skale/Projekty/ribose/enmail/issue-gpghome/pubring.kbx"}, :secret=>{:format=>"G10", :path=>"/Users/skale/Projekty/ribose/enmail/issue-gpghome/private-keys-v1.d"}}

*** LOADING PUBLIC KEYRING ***
["A95CBD6E2214D433", "F3757E85618524EF", "033FF4503ED02E33", "88860777FEF32DA2", "344E6B598796B32B", "8F5E4ED955898515"]

*** LOADING SECRET KEYRING ***
["A95CBD6E2214D433", "F3757E85618524EF", "033FF4503ED02E33", "88860777FEF32DA2", "344E6B598796B32B", "8F5E4ED955898515"]

---

Or maybe I'm misunderstanding something… It's been a while since I was struggling with original issue. I'll do some investigation and write more details.

[dewyatt]

@skalee

the secret keyring still can't be loaded

I can't see anything in the output that would indicate that. What am I missing?

[skalee]

@dewyatt Loading secret keyring does not add any keys. This puzzles me. However, maybe it's me who's missing something. Is it possible that secret keys are loaded from a public keyring?

---

Also, I've prepared another script, see this gist: https://gist.github.com/skalee/34372769daa1f1aea8dd0f93b1ba0993.

It relies on the presence of tmp/gpgme and tmp/rnp, two homedirs generated with GPG and RNP, respectively. Both are supposed to contain three keypairs. To be precise, these two directories were generated with generate_keys Rake task, following revision: https://github.com/riboseinc/bbdf6cb6cba788494dea93dcf5f342b62df2813b/blob/rnp/Rakefile (not on master).

See "Output 1", "Output 2", and "Output 3". In case of RNP-generated homedirs, it seems that:

• It does not matter if you only load public keyring, secret keyring, or both. The result is always the same: six keys with identical ids.

See "Output 4", "Output 5", and "Output 6". In case of GPGME-generated homedirs, it seems that:

• G10 secret keyring can't be read prior loading the public keyring
• Loading G10 secret keyring after loading the public keyring does not add any new keys.

[dewyatt]

@skalee The public and secret keys will always have the same identifiers. The differences you should be able to observe before and after loading the secret keys would be with Rnp#secret_key_count and Key#secret_key_present?.

[skalee]

@dewyatt FYI calling rnp#rnp.public_key_count raises following exception:

Traceback (most recent call last):
    6: from issue1.rb:55:in `<main>'
    5: from issue1.rb:55:in `each'
    4: from issue1.rb:69:in `block in <main>'
    3: from /Users/skale/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rnp-1.0.0/lib/rnp/rnp.rb:227:in `public_key_count'
    2: from /Users/skale/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rnp-1.0.0/lib/rnp/utils.rb:20:in `call_ffi'
    1: from /Users/skale/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rnp-1.0.0/lib/rnp/utils.rb:20:in `call'
/Users/skale/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rnp-1.0.0/lib/rnp/utils.rb:20:in `rnp_get_public_key_count': wrong number of arguments (1 for 2) (ArgumentError)

Rnp has been installed yesterday, rnp-ruby points to master.

I'll try with ?-method though.

---

Yes, it looks like loading the secret keyring changes secret_key_present? from false to true.

[dewyatt]

@skalee Thanks, I'll add a test for those methods and get that fixed.

[dewyatt]

@skalee Btw ruby-rnp 1.0.1 should fix the key count issue.

[skalee]

Thanks a lot! I've noticed the version bump. I confirm that it is fixed now.

[skalee]

@ni4 wrote:

@skalee Thanks, I'll check. Actually these ones should work.

I don't know whether it's an expected incompatibility or bug, but RNP fails to sign documents with these keys. See this gist: https://gist.github.com/skalee/81d90f3d8680cc191ea10170fbe16c7d.

I am able to sign documents with keys generated by RNP though.

cc @dewyatt

[dewyatt]

@skalee I dropped the ball and forgot about this, but I'm looking into it now.

[dewyatt]

I think I see at least one issue, I'll try to get a fix in tomorrow.

[dewyatt]

@skalee This is now fixed in rnp master. You'll still have failures in your script for elg keys (which are not supposed to sign).

[ronaldtse]

Thanks @dewyatt !

[ronaldtse]

@skalee FYI @ribose-jeffreylau has fully integrated RNP into our proprietary code and will be able to answer questions on Ruby-RNP, feel free to ping him. Thanks!

[skalee]

Thanks @ronaldtse! I don't think it will be much needed. After recent changes to rnp and rnp-ruby it works like a charm on my local computer. Though it doesn't in Travis yet.

[ronaldtse]

That's perfect, thanks @skalee !

[ronaldtse]

@skalee BTW what's the issue in Travis?

[skalee]

@ronaldtse Few of them, but neither was serious, and all resolved already.

BTW EnMail will not work with RNP 0.9.1 at all, but HEAD includes some fixes by @dewyatt and is fine.

[skalee]

This issue has been closed already. However few subtasks (checkboxes) were still in progress. They have been extracted to separate issues.

1. Fetching keys from the Internet to #55, and #56.
2. Password-protected keys support to #77.