Can't able to genrate server and client certs for the old Root certificate authority certs.

[Swarna-dev]

For freshly created root CA we can able to generate the server and client certificate after some time maybe a day it was throwing the 500 INTERNAL SERVER ERROR. Also ROOT CA is not visible in the UI also https://:447/

GET API I used to fetch RCA details -> http://:8000/api/1/authorities?filter=name;

we have tried reacreating the lemur instance as well as recreating the lemur postgress DB - it works only for some time again we are getting the below error.

logs captured as below:

2024-01-18 09:19:47,017 INFO sqlalchemy.engine.base.Engine SELECT 1;
2024-01-18 09:19:47,018 INFO sqlalchemy.engine.base.Engine {}
2024-01-18 09:19:47,724 INFO sqlalchemy.engine.base.Engine BEGIN (implicit)
2024-01-18 09:19:47,724 INFO sqlalchemy.engine.base.Engine SELECT users.id AS users_id, users.password AS users_password, users.active AS users_active, users.confirmed_at AS users_confirmed_at, users.username AS users_username, users.email AS users_email, users.profile_picture AS users_profile_picture
FROM users
WHERE users.id = %(id_1)s
2024-01-18 09:19:47,725 INFO sqlalchemy.engine.base.Engine {'id_1': 1}
2024-01-18 09:19:47,727 INFO sqlalchemy.engine.base.Engine SELECT users.id AS users_id, users.password AS users_password, users.active AS users_active, users.confirmed_at AS users_confirmed_at, users.username AS users_username, users.email AS users_email, users.profile_picture AS users_profile_picture
FROM users
WHERE users.id = %(id_1)s
2024-01-18 09:19:47,727 INFO sqlalchemy.engine.base.Engine {'id_1': 1}
2024-01-18 09:19:47,729 INFO sqlalchemy.engine.base.Engine SELECT roles.id AS roles_id, roles.name AS roles_name, roles.username AS roles_username, roles.password AS roles_password, roles.description AS roles_description, roles.authority_id AS roles_authority_id, roles.user_id AS roles_user_id, roles.third_party AS roles_third_party
FROM roles, roles_users
WHERE %(param_1)s = roles_users.user_id AND roles.id = roles_users.role_id
2024-01-18 09:19:47,729 INFO sqlalchemy.engine.base.Engine {'param_1': 1}
2024-01-18 09:19:47,731 INFO sqlalchemy.engine.base.Engine SELECT authorities.id AS authorities_id, authorities.owner AS authorities_owner, authorities.name AS authorities_name, authorities.body AS authorities_body, authorities.chain AS authorities_chain, authorities.active AS authorities_active, authorities.plugin_name AS authorities_plugin_name, authorities.description AS authorities_description, authorities.options AS authorities_options, authorities.date_created AS authorities_date_created, authorities.user_id AS authorities_user_id
FROM authorities
WHERE %(param_1)s = authorities.user_id
2024-01-18 09:19:47,731 INFO sqlalchemy.engine.base.Engine {'param_1': 1}
2024-01-18 09:19:47,733 INFO sqlalchemy.engine.base.Engine SELECT roles.id AS roles_id, roles.name AS roles_name, roles.username AS roles_username, roles.password AS roles_password, roles.description AS roles_description, roles.authority_id AS roles_authority_id, roles.user_id AS roles_user_id, roles.third_party AS roles_third_party
FROM roles, roles_users
WHERE %(param_1)s = roles_users.user_id AND roles.id = roles_users.role_id
2024-01-18 09:19:47,733 INFO sqlalchemy.engine.base.Engine {'param_1': 1}
2024-01-18 09:19:47,735 INFO sqlalchemy.engine.base.Engine SELECT count(*) AS count_1
FROM authorities
WHERE authorities.name ILIKE %(name_1)s
2024-01-18 09:19:47,735 INFO sqlalchemy.engine.base.Engine {'name_1': '%Certificate-Authority-dev.mybuilding.leviton.com%'}
2024-01-18 09:19:47,738 INFO sqlalchemy.engine.base.Engine SELECT authorities.id AS authorities_id, authorities.owner AS authorities_owner, authorities.name AS authorities_name, authorities.body AS authorities_body, authorities.chain AS authorities_chain, authorities.active AS authorities_active, authorities.plugin_name AS authorities_plugin_name, authorities.description AS authorities_description, authorities.options AS authorities_options, authorities.date_created AS authorities_date_created, authorities.user_id AS authorities_user_id
FROM authorities
WHERE authorities.name ILIKE %(name_1)s
 LIMIT %(param_1)s OFFSET %(param_2)s
2024-01-18 09:19:47,738 INFO sqlalchemy.engine.base.Engine {'name_1': '%Certificate-Authority-dev.mybuilding.leviton.com%', 'param_1': 10, 'param_2': 0}
2024-01-18 09:19:47,742 INFO sqlalchemy.engine.base.Engine SELECT certificates.id AS certificates_id, certificates.external_id AS certificates_external_id, certificates.owner AS certificates_owner, certificates.name AS certificates_name, certificates.description AS certificates_description, certificates.notify AS certificates_notify, certificates.body AS certificates_body, certificates.chain AS certificates_chain, certificates.csr AS certificates_csr, certificates.private_key AS certificates_private_key, certificates.issuer AS certificates_issuer, certificates.serial AS certificates_serial, certificates.cn AS certificates_cn, certificates.deleted AS certificates_deleted, certificates.dns_provider_id AS certificates_dns_provider_id, certificates.not_before AS certificates_not_before, certificates.not_after AS certificates_not_after, certificates.date_created AS certificates_date_created, certificates.signing_algorithm AS certificates_signing_algorithm, certificates.status AS certificates_status, certificates.bits AS certificates_bits, certificates.san AS certificates_san, certificates.rotation AS certificates_rotation, certificates.user_id AS certificates_user_id, certificates.authority_id AS certificates_authority_id, certificates.root_authority_id AS certificates_root_authority_id, certificates.rotation_policy_id AS certificates_rotation_policy_id, certificates.key_type AS certificates_key_type
FROM certificates
WHERE %(param_1)s = certificates.root_authority_id
2024-01-18 09:19:47,742 INFO sqlalchemy.engine.base.Engine {'param_1': 6}
[2024-01-18 09:19:47,747] ERROR in app: Exception on /api/1/authorities [GET]
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/utils.py", line 349, in _get_value_for_key
    return obj[key]
TypeError: 'Authority' object is not subscriptable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/venv/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/venv/lib/python3.8/site-packages/flask_restful/__init__.py", line 467, in wrapper
    resp = resource(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/flask/views.py", line 89, in view
    return self.dispatch_request(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/flask_restful/__init__.py", line 582, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/opt/lemur/lemur/auth/service.py", line 141, in decorated_function
    return f(*args, **kwargs)
  File "/opt/lemur/lemur/common/schema.py", line 175, in decorated_function
    return unwrap_pagination(resp, output_schema_to_use), 200
  File "/opt/lemur/lemur/common/schema.py", line 128, in unwrap_pagination
    marshaled_data["items"] = output_schema.dump(data["items"], many=True).data
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/schema.py", line 513, in dump
    preresult = marshal(
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/marshalling.py", line 114, in serialize
    ret = [self.serialize(d, fields_dict, many=False,
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/marshalling.py", line 114, in <listcomp>
    ret = [self.serialize(d, fields_dict, many=False,
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/marshalling.py", line 135, in serialize
    value = self.call_and_store(
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/marshalling.py", line 63, in call_and_store
    value = getter_func(data)
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/marshalling.py", line 134, in <lambda>
    getter = lambda d: field_obj.serialize(attr_name, d, accessor=accessor)
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/fields.py", line 243, in serialize
    value = self.get_value(attr, obj, accessor=accessor)
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/fields.py", line 186, in get_value
    return accessor_func(check_key, obj, default)
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/schema.py", line 415, in get_attribute
    return utils.get_value(attr, obj, default)
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/utils.py", line 336, in get_value
    return _get_value_for_keys(key.split('.'), obj, default)
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/utils.py", line 341, in _get_value_for_keys
    return _get_value_for_key(keys[0], obj, default)
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/utils.py", line 352, in _get_value_for_key
    attr = getattr(obj, key)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 294, in __get__
    return self.impl.get(instance_state(instance), dict_)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 730, in get
    value = self.callable_(state, passive)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/strategies.py", line 759, in _load_for_state
    return self._emit_lazyload(
  File "<string>", line 1, in <lambda>
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/strategies.py", line 900, in _emit_lazyload
    q(session)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/ext/baked.py", line 544, in all
    return list(self)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 100, in instances
    cursor.close()
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.raise_(
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 80, in instances
    rows = [proc(row) for row in fetch]
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 80, in <listcomp>
    rows = [proc(row) for row in fetch]
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 579, in _instance
    _populate_full(
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 725, in _populate_full
    dict_[key] = getter(row)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/sql/type_api.py", line 1278, in process
    return process_value(impl_processor(value), dialect)
  File "/opt/lemur/lemur/utils.py", line 122, in process_result_value
    return MultiFernet(self.keys).decrypt(value).decode("utf8")
  File "/opt/venv/lib/python3.8/site-packages/cryptography/fernet.py", line 202, in decrypt
    raise InvalidToken
cryptography.fernet.InvalidToken
2024-01-18 09:19:47,752 INFO sqlalchemy.engine.base.Engine ROLLBACK
Exception on /api/1/authorities [GET]
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/utils.py", line 349, in _get_value_for_key
    return obj[key]
TypeError: 'Authority' object is not subscriptable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/venv/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/venv/lib/python3.8/site-packages/flask_restful/__init__.py", line 467, in wrapper
    resp = resource(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/flask/views.py", line 89, in view
    return self.dispatch_request(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/flask_restful/__init__.py", line 582, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/opt/lemur/lemur/auth/service.py", line 141, in decorated_function
    return f(*args, **kwargs)
  File "/opt/lemur/lemur/common/schema.py", line 175, in decorated_function
    return unwrap_pagination(resp, output_schema_to_use), 200
  File "/opt/lemur/lemur/common/schema.py", line 128, in unwrap_pagination
    marshaled_data["items"] = output_schema.dump(data["items"], many=True).data
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/schema.py", line 513, in dump
    preresult = marshal(
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/marshalling.py", line 114, in serialize
    ret = [self.serialize(d, fields_dict, many=False,
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/marshalling.py", line 114, in <listcomp>
    ret = [self.serialize(d, fields_dict, many=False,
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/marshalling.py", line 135, in serialize
    value = self.call_and_store(
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/marshalling.py", line 63, in call_and_store
    value = getter_func(data)
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/marshalling.py", line 134, in <lambda>
    getter = lambda d: field_obj.serialize(attr_name, d, accessor=accessor)
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/fields.py", line 243, in serialize
    value = self.get_value(attr, obj, accessor=accessor)
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/fields.py", line 186, in get_value
    return accessor_func(check_key, obj, default)
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/schema.py", line 415, in get_attribute
    return utils.get_value(attr, obj, default)
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/utils.py", line 336, in get_value
    return _get_value_for_keys(key.split('.'), obj, default)
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/utils.py", line 341, in _get_value_for_keys
    return _get_value_for_key(keys[0], obj, default)
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/utils.py", line 352, in _get_value_for_key
    attr = getattr(obj, key)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 294, in __get__
    return self.impl.get(instance_state(instance), dict_)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 730, in get
    value = self.callable_(state, passive)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/strategies.py", line 759, in _load_for_state
    return self._emit_lazyload(
  File "<string>", line 1, in <lambda>
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/strategies.py", line 900, in _emit_lazyload
    q(session)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/ext/baked.py", line 544, in all
    return list(self)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 100, in instances
    cursor.close()
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.raise_(
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 80, in instances
    rows = [proc(row) for row in fetch]
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 80, in <listcomp>
    rows = [proc(row) for row in fetch]
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 579, in _instance
    _populate_full(
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 725, in _populate_full
    dict_[key] = getter(row)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/sql/type_api.py", line 1278, in process
    return process_value(impl_processor(value), dialect)
  File "/opt/lemur/lemur/utils.py", line 122, in process_result_value
    return MultiFernet(self.keys).decrypt(value).decode("utf8")
  File "/opt/venv/lib/python3.8/site-packages/cryptography/fernet.py", line 202, in decrypt
    raise InvalidToken
cryptography.fernet.InvalidToken
Exception on /api/1/authorities [GET]
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/utils.py", line 349, in _get_value_for_key
    return obj[key]
TypeError: 'Authority' object is not subscriptable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/venv/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/venv/lib/python3.8/site-packages/flask_restful/__init__.py", line 467, in wrapper
    resp = resource(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/flask/views.py", line 89, in view
    return self.dispatch_request(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/flask_restful/__init__.py", line 582, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/opt/lemur/lemur/auth/service.py", line 141, in decorated_function
    return f(*args, **kwargs)
  File "/opt/lemur/lemur/common/schema.py", line 175, in decorated_function
    return unwrap_pagination(resp, output_schema_to_use), 200
  File "/opt/lemur/lemur/common/schema.py", line 128, in unwrap_pagination
    marshaled_data["items"] = output_schema.dump(data["items"], many=True).data
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/schema.py", line 513, in dump
    preresult = marshal(
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/marshalling.py", line 114, in serialize
    ret = [self.serialize(d, fields_dict, many=False,
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/marshalling.py", line 114, in <listcomp>
    ret = [self.serialize(d, fields_dict, many=False,
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/marshalling.py", line 135, in serialize
    value = self.call_and_store(
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/marshalling.py", line 63, in call_and_store
    value = getter_func(data)
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/marshalling.py", line 134, in <lambda>
    getter = lambda d: field_obj.serialize(attr_name, d, accessor=accessor)
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/fields.py", line 243, in serialize
    value = self.get_value(attr, obj, accessor=accessor)
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/fields.py", line 186, in get_value
    return accessor_func(check_key, obj, default)
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/schema.py", line 415, in get_attribute
    return utils.get_value(attr, obj, default)
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/utils.py", line 336, in get_value
    return _get_value_for_keys(key.split('.'), obj, default)
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/utils.py", line 341, in _get_value_for_keys
    return _get_value_for_key(keys[0], obj, default)
  File "/opt/venv/lib/python3.8/site-packages/marshmallow/utils.py", line 352, in _get_value_for_key
    attr = getattr(obj, key)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 294, in __get__
    return self.impl.get(instance_state(instance), dict_)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 730, in get
    value = self.callable_(state, passive)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/strategies.py", line 759, in _load_for_state
    return self._emit_lazyload(
  File "<string>", line 1, in <lambda>
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/strategies.py", line 900, in _emit_lazyload
    q(session)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/ext/baked.py", line 544, in all
    return list(self)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 100, in instances
    cursor.close()
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.raise_(
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 80, in instances
    rows = [proc(row) for row in fetch]
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 80, in <listcomp>
    rows = [proc(row) for row in fetch]
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 579, in _instance
    _populate_full(
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 725, in _populate_full
    dict_[key] = getter(row)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/sql/type_api.py", line 1278, in process
    return process_value(impl_processor(value), dialect)
  File "/opt/lemur/lemur/utils.py", line 122, in process_result_value
    return MultiFernet(self.keys).decrypt(value).decode("utf8")
  File "/opt/venv/lib/python3.8/site-packages/cryptography/fernet.py", line 202, in decrypt
    raise InvalidToken
cryptography.fernet.InvalidToken

[jtschladen]

Hi @Swarna-dev, your error message seems to be a decryption issue. Have you confirmed that you're setting a stable value for LEMUR_ENCRYPTION_KEYS (and not using random value every time the server restarts)? This would be similar to lemur#3643.

[Swarna-dev]

Hi @jtschladen, Thank you!

the above solution works after updating the stable value for LEMUR_ENCRYPTION_KEYS.