search

mozilla / tls-observatory

An observatory for TLS configurations, X509 certificates, and more.
Mozilla Public License 2.0
528 stars 88 forks source link

Evaluate certificates with zlint #331

Open adamdecaf opened 6 years ago

adamdecaf commented 6 years ago

Similar to https://github.com/mozilla/tls-observatory/issues/187, https://github.com/mozilla/tls-observatory/issues/257, and https://github.com/mozilla/tls-observatory/issues/258, but to run zmap/zlint over the certificates.

This tool is pretty complete and has most of the BR checks included. (List: https://docs.google.com/spreadsheets/d/1ywp0op9mkTaggigpdF2YMTubepowJ50KQBhc_b00e-Y/edit#gid=0)

zlint looks to be gaining steam from (at least one) CA over their own tools. https://cabforum.org/pipermail/public/2018-April/013233.html Either way it's a really complete validator.

adamdecaf commented 6 years ago

The response from zlint is pretty easy to parse. I've included a full response:

```json $ zlint -pretty github.com.crt | jq . { "e_basic_constraints_not_critical": { "result": "NA" }, "e_ca_common_name_missing": { "result": "NA" }, "e_ca_country_name_invalid": { "result": "NA" }, "e_ca_country_name_missing": { "result": "NA" }, "e_ca_crl_sign_not_set": { "result": "NA" }, "e_ca_is_ca": { "result": "NA" }, "e_ca_key_cert_sign_not_set": { "result": "NA" }, "e_ca_key_usage_missing": { "result": "NA" }, "e_ca_key_usage_not_critical": { "result": "NA" }, "e_ca_organization_name_missing": { "result": "NA" }, "e_ca_subject_field_empty": { "result": "NA" }, "e_cab_dv_conflicts_with_locality": { "result": "NA" }, "e_cab_dv_conflicts_with_org": { "result": "NA" }, "e_cab_dv_conflicts_with_postal": { "result": "NA" }, "e_cab_dv_conflicts_with_province": { "result": "NA" }, "e_cab_dv_conflicts_with_street": { "result": "NA" }, "e_cab_iv_requires_personal_name": { "result": "NA" }, "e_cab_ov_requires_org": { "result": "NA" }, "e_cert_contains_unique_identifier": { "result": "pass" }, "e_cert_extensions_version_not_3": { "result": "pass" }, "e_cert_policy_iv_requires_country": { "result": "NA" }, "e_cert_policy_iv_requires_province_or_locality": { "result": "NA" }, "e_cert_policy_ov_requires_country": { "result": "NA" }, "e_cert_policy_ov_requires_province_or_locality": { "result": "NA" }, "e_cert_unique_identifier_version_not_2_or_3": { "result": "NA" }, "e_distribution_point_incomplete": { "result": "pass" }, "e_dnsname_bad_character_in_label": { "result": "pass" }, "e_dnsname_contains_bare_iana_suffix": { "result": "pass" }, "e_dnsname_empty_label": { "result": "pass" }, "e_dnsname_hyphen_in_sld": { "result": "pass" }, "e_dnsname_label_too_long": { "result": "pass" }, "e_dnsname_left_label_wildcard_correct": { "result": "pass" }, "e_dnsname_not_valid_tld": { "result": "pass" }, "e_dnsname_underscore_in_sld": { "result": "pass" }, "e_dnsname_wildcard_only_in_left_label": { "result": "pass" }, "e_dsa_correct_order_in_subgroup": { "result": "NA" }, "e_dsa_improper_modulus_or_divisor_size": { "result": "NA" }, "e_dsa_params_missing": { "result": "NA" }, "e_dsa_shorter_than_2048_bits": { "result": "NA" }, "e_dsa_unique_correct_representation": { "result": "NA" }, "e_ec_improper_curves": { "result": "NA" }, "e_ev_business_category_missing": { "result": "pass" }, "e_ev_country_name_missing": { "result": "pass" }, "e_ev_locality_name_missing": { "result": "pass" }, "e_ev_organization_name_missing": { "result": "pass" }, "e_ev_serial_number_missing": { "result": "pass" }, "e_ev_valid_time_too_long": { "result": "pass" }, "e_ext_aia_marked_critical": { "result": "pass" }, "e_ext_authority_key_identifier_critical": { "result": "pass" }, "e_ext_authority_key_identifier_missing": { "result": "pass" }, "e_ext_authority_key_identifier_no_key_identifier": { "result": "pass" }, "e_ext_cert_policy_disallowed_any_policy_qualifier": { "result": "pass" }, "e_ext_cert_policy_duplicate": { "result": "pass" }, "e_ext_cert_policy_explicit_text_ia5_string": { "result": "NA" }, "e_ext_cert_policy_explicit_text_too_long": { "result": "NA" }, "e_ext_duplicate_extension": { "result": "pass" }, "e_ext_freshest_crl_marked_critical": { "result": "NA" }, "e_ext_ian_dns_not_ia5_string": { "result": "NA" }, "e_ext_ian_empty_name": { "result": "NA" }, "e_ext_ian_no_entries": { "result": "NA" }, "e_ext_ian_rfc822_format_invalid": { "result": "NA" }, "e_ext_ian_space_dns_name": { "result": "NA" }, "e_ext_ian_uri_format_invalid": { "result": "NA" }, "e_ext_ian_uri_host_not_fqdn_or_ip": { "result": "NA" }, "e_ext_ian_uri_not_ia5": { "result": "NA" }, "e_ext_ian_uri_relative": { "result": "NA" }, "e_ext_key_usage_cert_sign_without_ca": { "result": "pass" }, "e_ext_key_usage_without_bits": { "result": "pass" }, "e_ext_name_constraints_not_critical": { "result": "NA" }, "e_ext_name_constraints_not_in_ca": { "result": "NA" }, "e_ext_policy_constraints_empty": { "result": "NA" }, "e_ext_policy_constraints_not_critical": { "result": "NA" }, "e_ext_policy_map_any_policy": { "result": "NA" }, "e_ext_san_contains_reserved_ip": { "result": "pass" }, "e_ext_san_directory_name_present": { "result": "pass" }, "e_ext_san_dns_name_too_long": { "result": "pass" }, "e_ext_san_dns_not_ia5_string": { "result": "pass" }, "e_ext_san_edi_party_name_present": { "result": "pass" }, "e_ext_san_empty_name": { "result": "pass" }, "e_ext_san_missing": { "result": "pass" }, "e_ext_san_no_entries": { "result": "pass" }, "e_ext_san_not_critical_without_subject": { "result": "pass" }, "e_ext_san_other_name_present": { "result": "pass" }, "e_ext_san_registered_id_present": { "result": "pass" }, "e_ext_san_rfc822_format_invalid": { "result": "pass" }, "e_ext_san_rfc822_name_present": { "result": "pass" }, "e_ext_san_space_dns_name": { "result": "pass" }, "e_ext_san_uniform_resource_identifier_present": { "result": "pass" }, "e_ext_san_uri_format_invalid": { "result": "pass" }, "e_ext_san_uri_host_not_fqdn_or_ip": { "result": "pass" }, "e_ext_san_uri_not_ia5": { "result": "pass" }, "e_ext_san_uri_relative": { "result": "pass" }, "e_ext_subject_directory_attr_critical": { "result": "NA" }, "e_ext_subject_key_identifier_critical": { "result": "pass" }, "e_ext_subject_key_identifier_missing_ca": { "result": "NA" }, "e_generalized_time_does_not_include_seconds": { "result": "NA" }, "e_generalized_time_includes_fraction_seconds": { "result": "NA" }, "e_generalized_time_not_in_zulu": { "result": "NA" }, "e_ian_bare_wildcard": { "result": "NA" }, "e_ian_dns_name_includes_null_char": { "result": "NA" }, "e_ian_dns_name_starts_with_period": { "result": "NA" }, "e_ian_wildcard_not_first": { "result": "NA" }, "e_inhibit_any_policy_not_critical": { "result": "NA" }, "e_international_dns_name_not_nfkc": { "result": "pass" }, "e_international_dns_name_not_unicode": { "result": "pass" }, "e_invalid_certificate_version": { "result": "pass" }, "e_issuer_dn_country_not_printable_string": { "result": "pass" }, "e_issuer_field_empty": { "result": "pass" }, "e_name_constraint_empty": { "result": "NA" }, "e_name_constraint_maximum_not_absent": { "result": "NA" }, "e_name_constraint_minimum_non_zero": { "result": "NA" }, "e_old_root_ca_rsa_mod_less_than_2048_bits": { "result": "NA" }, "e_old_sub_ca_rsa_mod_less_than_1024_bits": { "result": "NA" }, "e_old_sub_cert_rsa_mod_less_than_1024_bits": { "result": "NA" }, "e_path_len_constraint_improperly_included": { "result": "pass" }, "e_path_len_constraint_zero_or_less": { "result": "pass" }, "e_public_key_type_not_allowed": { "result": "pass" }, "e_root_ca_extended_key_usage_present": { "result": "NA" }, "e_root_ca_key_usage_must_be_critical": { "result": "NA" }, "e_root_ca_key_usage_present": { "result": "NA" }, "e_rsa_exp_negative": { "result": "pass" }, "e_rsa_mod_less_than_2048_bits": { "result": "pass" }, "e_rsa_no_public_key": { "result": "pass" }, "e_rsa_public_exponent_not_odd": { "result": "pass" }, "e_rsa_public_exponent_too_small": { "result": "pass" }, "e_san_bare_wildcard": { "result": "pass" }, "e_san_dns_name_includes_null_char": { "result": "pass" }, "e_san_dns_name_starts_with_period": { "result": "pass" }, "e_san_wildcard_not_first": { "result": "pass" }, "e_serial_number_longer_than_20_octets": { "result": "pass" }, "e_serial_number_not_positive": { "result": "pass" }, "e_signature_algorithm_not_supported": { "result": "pass" }, "e_sub_ca_aia_does_not_contain_ocsp_url": { "result": "NA" }, "e_sub_ca_aia_marked_critical": { "result": "NA" }, "e_sub_ca_aia_missing": { "result": "NA" }, "e_sub_ca_certificate_policies_missing": { "result": "NA" }, "e_sub_ca_crl_distribution_points_does_not_contain_url": { "result": "NA" }, "e_sub_ca_crl_distribution_points_marked_critical": { "result": "NA" }, "e_sub_ca_crl_distribution_points_missing": { "result": "NA" }, "e_sub_cert_aia_does_not_contain_ocsp_url": { "result": "pass" }, "e_sub_cert_aia_marked_critical": { "result": "pass" }, "e_sub_cert_aia_missing": { "result": "pass" }, "e_sub_cert_cert_policy_empty": { "result": "pass" }, "e_sub_cert_certificate_policies_missing": { "result": "pass" }, "e_sub_cert_country_name_must_appear": { "result": "NE" }, "e_sub_cert_crl_distribution_points_does_not_contain_url": { "result": "pass" }, "e_sub_cert_crl_distribution_points_marked_critical": { "result": "pass" }, "e_sub_cert_eku_missing": { "result": "pass" }, "e_sub_cert_eku_server_auth_client_auth_missing": { "result": "pass" }, "e_sub_cert_given_name_surname_contains_correct_policy": { "result": "NA" }, "e_sub_cert_key_usage_cert_sign_bit_set": { "result": "pass" }, "e_sub_cert_key_usage_crl_sign_bit_set": { "result": "pass" }, "e_sub_cert_locality_name_must_appear": { "result": "NE" }, "e_sub_cert_locality_name_must_not_appear": { "result": "NE" }, "e_sub_cert_not_is_ca": { "result": "pass" }, "e_sub_cert_or_sub_ca_using_sha1": { "result": "pass" }, "e_sub_cert_postal_code_must_not_appear": { "result": "NE" }, "e_sub_cert_province_must_appear": { "result": "NE" }, "e_sub_cert_province_must_not_appear": { "result": "NE" }, "e_sub_cert_street_address_should_not_exist": { "result": "NE" }, "e_sub_cert_valid_time_longer_than_39_months": { "result": "NE" }, "e_sub_cert_valid_time_longer_than_825_days": { "result": "NE" }, "e_subject_common_name_max_length": { "result": "pass" }, "e_subject_common_name_not_from_san": { "result": "pass" }, "e_subject_contains_noninformational_value": { "result": "pass" }, "e_subject_contains_reserved_ip": { "result": "pass" }, "e_subject_country_not_iso": { "result": "pass" }, "e_subject_dn_country_not_printable_string": { "result": "pass" }, "e_subject_dn_serial_number_max_length": { "result": "pass" }, "e_subject_dn_serial_number_not_printable_string": { "result": "pass" }, "e_subject_email_max_length": { "result": "pass" }, "e_subject_empty_without_san": { "result": "pass" }, "e_subject_given_name_max_length": { "result": "pass" }, "e_subject_info_access_marked_critical": { "result": "NA" }, "e_subject_locality_name_max_length": { "result": "pass" }, "e_subject_not_dn": { "result": "pass" }, "e_subject_organization_name_max_length": { "result": "pass" }, "e_subject_organizational_unit_name_max_length": { "result": "pass" }, "e_subject_postal_code_max_length": { "result": "pass" }, "e_subject_state_name_max_length": { "result": "pass" }, "e_subject_street_address_max_length": { "result": "pass" }, "e_subject_surname_max_length": { "result": "pass" }, "e_utc_time_does_not_include_seconds": { "result": "pass" }, "e_utc_time_not_in_zulu": { "result": "pass" }, "e_validity_time_not_positive": { "result": "pass" }, "e_wrong_time_format_pre2050": { "result": "pass" }, "n_ca_digital_signature_not_set": { "result": "NA" }, "n_contains_redacted_dnsname": { "result": "pass" }, "n_sub_ca_eku_missing": { "result": "NA" }, "n_sub_ca_eku_not_technically_constrained": { "result": "NA" }, "n_subject_common_name_included": { "result": "info" }, "w_distribution_point_missing_ldap_or_uri": { "result": "pass" }, "w_dnsname_underscore_in_trd": { "result": "pass" }, "w_dnsname_wildcard_left_of_public_suffix": { "result": "pass" }, "w_eku_critical_improperly": { "result": "pass" }, "w_ext_aia_access_location_missing": { "result": "pass" }, "w_ext_cert_policy_contains_noticeref": { "result": "pass" }, "w_ext_cert_policy_explicit_text_includes_control": { "result": "NA" }, "w_ext_cert_policy_explicit_text_not_nfc": { "result": "NA" }, "w_ext_cert_policy_explicit_text_not_utf8": { "result": "NA" }, "w_ext_crl_distribution_marked_critical": { "result": "pass" }, "w_ext_ian_critical": { "result": "NA" }, "w_ext_key_usage_not_critical": { "result": "pass" }, "w_ext_policy_map_not_critical": { "result": "NA" }, "w_ext_policy_map_not_in_cert_policy": { "result": "NA" }, "w_ext_san_critical_with_subject_dn": { "result": "pass" }, "w_ext_subject_key_identifier_missing_sub_cert": { "result": "pass" }, "w_ian_iana_pub_suffix_empty": { "result": "NA" }, "w_issuer_dn_leading_whitespace": { "result": "pass" }, "w_issuer_dn_trailing_whitespace": { "result": "pass" }, "w_multiple_issuer_rdn": { "result": "pass" }, "w_multiple_subject_rdn": { "result": "pass" }, "w_name_constraint_on_edi_party_name": { "result": "NA" }, "w_name_constraint_on_registered_id": { "result": "NA" }, "w_name_constraint_on_x400": { "result": "NA" }, "w_root_ca_basic_constraints_path_len_constraint_field_present": { "result": "NA" }, "w_root_ca_contains_cert_policy": { "result": "NA" }, "w_rsa_mod_factors_smaller_than_752": { "result": "pass" }, "w_rsa_mod_not_odd": { "result": "pass" }, "w_rsa_public_exponent_not_in_range": { "result": "pass" }, "w_san_iana_pub_suffix_empty": { "result": "pass" }, "w_serial_number_low_entropy": { "result": "NE" }, "w_sub_ca_aia_does_not_contain_issuing_ca_url": { "result": "NA" }, "w_sub_ca_certificate_policies_marked_critical": { "result": "NA" }, "w_sub_ca_eku_critical": { "result": "NA" }, "w_sub_ca_name_constraints_not_critical": { "result": "NA" }, "w_sub_cert_aia_does_not_contain_issuing_ca_url": { "result": "pass" }, "w_sub_cert_certificate_policies_marked_critical": { "result": "pass" }, "w_sub_cert_eku_extra_values": { "result": "pass" }, "w_sub_cert_sha1_expiration_too_long": { "result": "NA" }, "w_subject_dn_leading_whitespace": { "result": "pass" }, "w_subject_dn_trailing_whitespace": { "result": "pass" } } ```

Here's an example parsing the output. https://play.golang.org/p/-6gCyT3xSWu

Would we want to store the entire response? The result field is limited to just a few values. We could probably get away with only storing info, warn, error, and fatal.

Note: NE refers to checks which are date related, but aren't applicable during evaluation. See: https://github.com/zmap/zlint/blob/88032a5e59f98690016d7dd312a3620cecb3e2e0/lints/base.go#L76-L79

Values: https://github.com/zmap/zlint/blob/88032a5e59f98690016d7dd312a3620cecb3e2e0/lints/result.go#L54

cc @jvehent thoughts?

jvehent commented 6 years ago

I agree we should avoid storing passing tests results in the database. I think a list of failing tests (or info, warn, etc.) is really what we want.

Also, I'd rather not update the worker every time zlint adds a new check, which would be needed right now because they use check names as json keys (a bad practice, IMO). Could the worker be written in a way that parses the json as map[string]result where string is the test name? That would allow us to update to upstream zlint without having to modify the worker's go code.

adamdecaf commented 6 years ago

Could the worker be written in a way that parses the json as map[string]result where string is the test name?

I believe that's exactly what my example parser does. I was thinking we could delete from that map any NA, NE, or pass checks and save the resulting map in our db.

There's no way I want to encode specific zlint checks into tls-observatory. That's just going to be a nightmare to maintain.

I can it as technically possible to try and parse out BR sections, human readable descriptions, etc from that spreadsheet though. I'd probably save that for a second PR as it's probably messy to do and might require google auth.