fort does not accept publication point self signed certificate

[cli0]

I am using self signed certificates for my Publication Server and wanted to test Fort as RP. I added the self signed certs to /usr/lib/ssl/certs according to the website. Also ran openssl rehash. Fort was compiled and installed using the git repository. However Fort throws me this error:

Jan  6 11:24:38 INF: Starting validation.
Jan  6 11:24:38 INF [Validation]: /root/tal/ta.tal: HTTP GET: https://server.com/ta/ta.cer
Jan  6 11:24:38 ERR [Validation]: /root/tal/ta.tal: Error requesting URL: error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding. (HTTP code: 0)
Jan  6 11:24:38 WRN [Validation]: /root/tal/ta.tal: TAL URI 'https://server.com/ta/ta.cer' could not be downloaded.
Jan  6 11:24:38 WRN [Validation]: /root/tal/ta.tal: Looking for the TA certificate at the local files.
Jan  6 11:24:38 ERR [Validation]: /root/tal/ta.tal: stat(repository/server.com/ta/ta.cer) failed: No such file or directory
Jan  6 11:24:38 ERR [Validation]: /root/tal/ta.tal: stat(repository/server.com/ta/ta.cer) failed: No such file or directory
Jan  6 11:24:38 ERR: /root/tal/ta.tal: None of the URIs of the TAL '/root/tal/ta.tal' yielded a successful traversal.
Stack trace:
  fort(print_stack_trace+0x3b) [0x5627abbd78cb]
  fort(pr_op_err+0x9f) [0x5627abbd800f]
  fort(+0x35dda) [0x5627abbeadda]
  fort(+0x44aac) [0x5627abbf9aac]
  /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f8a486fe609]
  /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f8a48625293]
(End of stack trace)
Jan  6 11:24:38 WRN: Validation from TAL '/root/tal/ta.tal' yielded error, discarding any other validation results.
Jan  6 11:24:38 INF: Validation finished:
Jan  6 11:24:38 INF: - Valid ROAs: 0
Jan  6 11:24:38 INF: - Valid Router Keys: 0
Jan  6 11:24:38 INF: - No serial number.
Jan  6 11:24:38 INF: - Real execution time: 0 secs.
Jan  6 11:24:38 ERR: First validation wasn't successful.
Stack trace:
  fort(print_stack_trace+0x3b) [0x5627abbd78cb]
  fort(pr_op_err+0x9f) [0x5627abbd800f]
  fort(main+0x180) [0x5627abbd35e0]
  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f8a4852a0b3]
  fort(_start+0x2e) [0x5627abbd362e]
(End of stack trace)

I turned off CURL_SSL_VERIFYPEER to 0L and then CURL_SSL_VERIFYHOST to 0L in http.c and now the error (for either change) becomes this:

Jan  6 11:28:49 INF: Starting validation.
Jan  6 11:28:49 INF [Validation]: /root/tal/ta.tal: HTTP GET: https://server.com/ta/ta.cer
Jan  6 11:28:49 INF [Validation]: https://server.com/rrdp/notification.xml: HTTP GET: https://server.com/rrdp/notification.xml
Jan  6 11:28:49 ERR [Validation]: https://server.com/rrdp/notification.xml: Type positiveInteger doesn't allow value '0' (at line 1)
Jan  6 11:28:49 ERR [Validation]: https://server.com/rrdp/notification.xml: Element notification failed to validate attributes (at line 1)
Jan  6 11:28:49 ERR [Validation]: https://server.com/rrdp/notification.xml: Invalid attribute serial for element notification (at line 1)
Jan  6 11:28:49 ERR [Validation]: https://server.com/rrdp/notification.xml: XML document isn't valid.
Jan  6 11:28:49 INF [Validation]: https://server.com/ta/ta.cer: Couldn't fetch data from RRDP repository 'https://server.com/rrdp/notification.xml', trying to fetch data now from 'rsync://server.com/repo/ta/0'.
Jan  6 11:28:49 ERR [Validation]: https://server.com/ta/ta.cer: stat(repository/server.com/repo/ta/0/6346E2BA25F361C1B04124AAE7A3831FA54840C7.mft) failed: No such file or directory
Jan  6 11:28:49 WRN: Validation from TAL '/root/tal/ta.tal' yielded error, discarding any other validation results.
Jan  6 11:28:49 INF: Validation finished:
Jan  6 11:28:49 INF: - Valid ROAs: 0
Jan  6 11:28:49 INF: - Valid Router Keys: 0
Jan  6 11:28:49 INF: - No serial number.
Jan  6 11:28:49 INF: - Real execution time: 0 secs.
Jan  6 11:28:49 ERR: First validation wasn't successful.
Stack trace:
  fort(print_stack_trace+0x3b) [0x55b0aa4548cb]
  fort(pr_op_err+0x9f) [0x55b0aa45500f]
  fort(main+0x180) [0x55b0aa4505e0]
  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f775c3b40b3]
  fort(_start+0x2e) [0x55b0aa45062e]
(End of stack trace)

Both errors were not raised by any other RP (Routinator or Octorpki) connecting to the same PP with the exact same certificates, only Fort. I have restarted Krill but nothing changes with regards to how Fort interprets the data. Can anyone explain the error in the second log output and what it means? And is there an alternative way to make self signed certs as recognizable by Fort without throwing errors?

Update:

The second error message is apparently due to Krill starting its serial with 0 which is apparently not RFC compliant. After adding a ROA to the manifest the serial increased so the error does not appear anymore. However, for the same Publication Server I get now another type of error and the error stack is not transparent enough to debug. Fort is the only RP sensitive enough to pick up on whatever the error here is. How to debug whatever potential error Fort is picking up on?

Jan  7 16:11:42 INF [Validation]: https://server.com/rrdp/notification.xml: HTTP GET: https://server.com/rrdp/notification.xml
Jan  7 16:11:42 INF [Validation]: https://server.com/rrdp/52664a41-5b37-473e-9062-63b95df45f77/1/8f7b82f5afc5e86b/snapshot.xml: HTTP GET: https://server.com/rrdp/52664a41-5b37-473e-9062-63b95df45f77/1/8f7b82f5afc5e86b/snapshot.xml
Jan  7 16:11:42 ERR [Validation]: rsync://server.com/repo/ca2/0/9492043940A2E3E9CFA7912107996984F20674CD.mft: Error reading certificate
Jan  7 16:11:42 ERR [Validation]: rsync://server.com/repo/ca2/0/9492043940A2E3E9CFA7912107996984F20674CD.mft: libcrypto error stack:
Jan  7 16:11:42 ERR [Validation]: rsync://server.com/repo/ca2/0/9492043940A2E3E9CFA7912107996984F20674CD.mft: 
Jan  7 16:11:42 ERR [Validation]: rsync://server.com/repo/ca2/0/9492043940A2E3E9CFA7912107996984F20674CD.mft: 
Jan  7 16:11:42 ERR [Validation]: rsync://server.com/repo/ca2/0/9492043940A2E3E9CFA7912107996984F20674CD.mft: End of libcrypto stack.
Jan  7 16:11:42 INF: Validation finished:
Jan  7 16:11:42 INF: - Valid ROAs: 0
Jan  7 16:11:42 INF: - Valid Router Keys: 0
Jan  7 16:11:42 INF: - Serial: 1
Jan  7 16:11:42 INF: - Real execution time: 0 secs.
Jan  7 16:11:42 WRN: First validation cycle successfully ended, now you can connect your router(s)

[ydahhrk]

Hi. Sorry for the wait.

I turned off CURL_SSL_VERIFYPEER to 0L and then CURL_SSL_VERIFYHOST to 0L in http.c and now the error (for either change) becomes this:

Ok, but that defeats the whole purpose.

What happens if you do curl https://server.com/ta/ta.cer > ta.cer?

serial with 0 which is apparently not RFC compliant.

Agree.

I get now another type of error and the error stack is not transparent enough to debug. Fort is the only RP sensitive enough to pick up on whatever the error here is. How to debug whatever potential error Fort is picking up on?

I suppose Fort is using the wrong error printing function. libcrypto has shown to be inconsistent when reporting errors.

Any chance you could send me this manifest file? (rsync://server.com/repo/ca2/0/9492043940A2E3E9CFA7912107996984F20674CD.mft)

(My mail is in my Github profile)

[cli0]

Ok, but that defeats the whole purpose.

You are right but I just needed to test my setup with my self signed keys before going in production and I could not find any other way to temporarily make my self signed keys acceptable for Fort, at least for a short while. If there is any method without changing the source code (temporarily) that has not occurred to me yet or if I made an error in my previous approach I'm all ears :)

What happens if you do curl https://server.com/ta/ta.cer > ta.cer?

Because it is a self signed certificate curl doesn't recognize it as safe, but if I add -k as in curl https://server.com/ta/ta.cer > ta.cer -k the certificate ta.cer is downloaded fine.

Any chance you could send me this manifest file?

Sent! Thank you.

[ydahhrk]

Ok, Manifest issue first:

It seems like an incorrectly implemented validation. Unfortunately, there are two candidates, and I will not be able to tell which it is unless you raise validation logging to debug and send me the new output:

--validation-log.level debug

It's a bit of a shame that you didn't show up about 2 months later, because I've actually been working on both validations in a different branch. But I can't merge the patches yet, because the refactor was extremely intrusive, and still needs testing.

Alternatively, since you're running the git version, you can simply compile both validations out by replacing the following empty lines with "return 0;":

• src/object/certificate.c, line 314
• src/object/certificate.c, line 1920

Did you happen to disable rsync? I can see how the 1920 validation would be affected by that. And it would certainly confirm that it's incorrectly implemented.

As to why it's not printing a proper error message... Fort bug. I just patched it.

Need to do a few things before debugging the ca-path issue. BRB.

[cli0]

Hi there, thank you for looking into these issues!

>Unfortunately, there are two candidates, and I will not be able to tell which it is unless you raise validation logging to debug and send me the new output:

I cannot supply this info right now because I found a temporary work around: I issue new keys for my CA. When I do that Fort processes the mft normally although I've seen this remedy "expire" in a day or two so if it reappears I will re-run the validation with debug.

> Did you happen to disable rsync?

I did in fact disable rsync. Maybe that was the issue all along.

I came across yet another weird error message after I temporarily fixed the cryptostack issue with issuing new keys like I described earlier.

I re-run Krill (rsync enabled/disabled does not affect the result), I have re-issued all my CAs' keys and run Fort using the following command:

nohup fort --tal tal/ --local-repository repository/ --server.address localhost --server.port 8323 --log.enabled=true --log.level=info --validation-log.enabled=true --validation-log.level=info --output.roa roas.csv --output.bgpsec bgpsec.csv --rsync.enabled=false --server.interval.validation=120 > fortlog.txt &

The first validation run always goes smoothly, everything is fine. Second validation run and onwards I get a 304 redirect error. I am not using redirection on any of my web servers, i double checked. Also this issue is only being picked up on subsequent validation runs and never on the first run (?). Fort and Publication Server are running on different lxc containers. The following are the logs:

Jan 16 21:50:32 INF: Starting validation.
Jan 16 21:50:32 INF: - Serial before validation: 1
Jan 16 21:50:32 INF [Validation]: /root/tal/ta.tal: HTTP GET: https://server.com/ta/ta.cer
Jan 16 21:50:32 INF [Validation]: https://server.com/rrdp/notification.xml: HTTP GET: https://server.com/rrdp/notification.xml
Jan 16 21:50:32 ERR [Validation]: https://server.com/rrdp/notification.xml: HTTP result code: 304. I don't follow redirects; discarding file.
Jan 16 21:50:32 INF [Validation]: https://server.com/ta/ta.cer: Couldn't fetch data from RRDP repository 'https://server.com/rrdp/notification.xml', trying to fetch data now from 'rsync://server.com/repo/ta/0'.
Jan 16 21:50:32 INF [Validation]: rsync://server.com/repo/ta/0/5FBFB6C0C0B17334BD03B7957699A814A7AC3ACD.cer: RRDP repository 'https://server.com/rrdp/notification.xml' download/processing returned error previously, now I will try to fetch data from 'rsync://server.com/repo/testbed/0'.

[ydahhrk]

The first validation run always goes smoothly, everything is fine. Second validation run and onwards I get a 304 redirect error. I am not using redirection on any of my web servers, i double checked. Also this issue is only being picked up on subsequent validation runs and never on the first run (?). Fort and Publication Server are running on different lxc containers.

I just uploaded a patch for it.

[ydahhrk]

Jan 6 11:24:38 ERR [Validation]: /root/tal/ta.tal: Error requesting URL: error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding. (HTTP code: 0)

Hmmm... works fine for me.

For reference, this is what I did:

• Point Fort to https://self-signed.badssl.com/. (Did this by manually calling http_download_file() from main().)
• Download that site's certificate (Firefox: Enter https://self-signed.badssl.com/, click "Advanced..." > "View Certificate" > "PEM (cert)") into dedicated directory capath/.
• Run Fort normally. https://self-signed.badssl.com/ is refused because its certificate is self-signed.
• Run openssl rehash capath/.
• Run Fort with --http.ca-path capath/. https://self-signed.badssl.com/ is downloaded fine.

[ydahhrk]

What happens if you run

curl --capath /usr/lib/ssl/certs https://server.com/ta/ta.cer

?

[ydahhrk]

Erm... feel free to postpone this if you don't want to spend your Sunday further. My hands are largely free next week.

(But I'll still be around today.)

[cli0]

Your sunday started right when mine ended :P sorry for the late reply on your feedback :)

When I run a simple curl request it's all good, file is downloaded.

root@fort-testbed:~# curl --capath /usr/lib/ssl/certs https://server.com/ta/ta.cer --output ta.cer
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1157  100  1157    0     0  77133      0 --:--:-- --:--:-- --:--:-- 77133

Running Fort with the following command gives me the same rsa error: invalid padding as mentioned above:

nohup fort --tal tal/ --local-repository repository/ --server.address localhost --server.port 8323 --log.enabled=true --log.level=info --validation-log.enabled=true --validation-log.level=info --http.ca-path /usr/lib/ssl/certs/ --output.roa roas.csv --output.bgpsec bgpsec.csv --rsync.enabled=false --server.interval.validation=60 > fortlog.txt &

With or without rsync is the same outcome. I don't know why the http.ca-path doesn't behave the same.

Something else I noticed in the compiled version where I disabled all crypto verification, the first run of the validation works fine as always, on the second validation run onwards (ad infinitum) I get:

Jan 17 18:49:23 INF: Starting validation.
Jan 17 18:49:23 INF: - Serial before validation: 1
Jan 17 18:49:23 INF [Validation]: /root/tal/ta.tal: HTTP GET: https://server.com/ta/ta.cer
Jan 17 18:49:23 INF [Validation]: https://server.com/rrdp/notification.xml: HTTP GET: https://server.com/rrdp/notification.xml
Jan 17 18:49:23 ERR [Validation]: https://server.com/rrdp/notification.xml: Couldn't delete 'repository/D3D16E00/server.com/rrdp/notification.xml': No such file or directory

The path in question is there and available. Error pops up only after the first successful run.

[ydahhrk]

Something else I noticed in the compiled version where I disabled all crypto verification, the first run of the validation works fine as always, on the second validation run onwards (ad infinitum) I get:

Ok. This is a bug, of course, but it makes sense. But it's ignored; it shouldn't ruin the validation. I'll just silence it until the refactor.

Running Fort with the following command gives me the same rsa error: invalid padding as mentioned above:

If you point it to a different trust anchor (eg. one of the RIR's), do you get the same problem?

Alternatively, any chance you could give me access to https://server.com/ta/ta.cer (and its certificate)?