Open nikolkam opened 12 months ago
Since the problem was fixed after tpm2 flushcontext --saved-session
there seem to be to many active sessions. You can list them with: tpm2_getcap handles-saved-session
or determine the max number with:
tpm2_getcap properties-fixed| grep ACTIVE -A 1
But if this number is exceeded the error message should actually be:
ERROR:esys:src/tss2-esys/api/Esys_StartAuthSession.c:136:Esys_StartAuthSession() Esys Finish ErrorCode (0x00000905) ERROR: Esys_StartAuthSession(0x905) - tpm:warn(2.0): out of session handles
instead of an tcti error. If you could attach your example I could try to reproduce it.
You could also check Chapter "30.5 Session Context Management" in the Architecture part of the TPM specification on this topic.
@JuergenReppSIT
I don't think it's reaching a handle max limit as tpm2_getcap handles-saved-session
only shows two handles 0x2000000 and 0x2000001 and the active session max is 64.
Below is the code I use to reproduce the bug. Please initialize a token with some private key for signing and set the PKCS11_LIB_PATH and TOKEN_LABEL in the code. Then start one process with ./<program_name> one
and another with no argument. One will do one sign and wait indefinitely and another will perform sign loops until failure. It's quicker to reproduce on physical machines but also happens on my hyper-v ubuntu vm with tpm enabled after around 65k sign attempts. (10~20 min)
#include <cstdint>
#include <iostream>
#include <vector>
#include <cstring>
#include <dlfcn.h>
#include <string.h>
#include <future>
#include "pkcs11.h"
const char* PKCS11_LIB_PATH = "tpm2_pkcs11.so";
const char* TOKEN_LABEL = "token-name";
CK_FUNCTION_LIST_PTR p11;
// Initialize the PKCS11 library
bool init_pkcs11() {
CK_RV rv;
CK_C_GetFunctionList pGetFunctionList;
void* pkcs11lib = dlopen(PKCS11_LIB_PATH, RTLD_LAZY);
if (!pkcs11lib) {
std::cerr << "Failed to load library." << std::endl;
return false;
}
pGetFunctionList = (CK_C_GetFunctionList) dlsym(pkcs11lib, "C_GetFunctionList");
if (!pGetFunctionList) {
std::cerr << "Failed to get function list." << std::endl;
return false;
}
rv = pGetFunctionList(&p11);
if (rv != CKR_OK) {
std::cerr << "Failed to initialize PKCS11. Error: " << rv << std::endl;
return false;
}
rv = p11->C_Initialize(NULL);
if (rv != CKR_OK) {
std::cerr << "Error in C_Initialize: " << rv << std::endl;
return false;
}
return true;
}
bool get_key(CK_SESSION_HANDLE *session, CK_OBJECT_HANDLE *key) {
CK_RV rv;
CK_SLOT_ID slot;
CK_ULONG count;
rv = p11->C_GetSlotList(CK_TRUE, NULL, &count);
if (rv != CKR_OK || count <= 0) {
std::cerr << "Failed to get slot list." << std::endl;
return false;
}
std::vector<CK_SLOT_ID> slotList(count);
rv = p11->C_GetSlotList(CK_TRUE, slotList.data(), &count);
if (rv != CKR_OK) {
std::cerr << "Failed to get slot list." << std::endl;
return false;
}
bool foundToken = false;
for (CK_ULONG i = 0; i < count; i++) {
CK_TOKEN_INFO tokenInfo;
rv = p11->C_GetTokenInfo(slotList[i], &tokenInfo);
if (rv != CKR_OK) {
printf("Failed to get token info: %lu\n", rv);
return false;
}
if(strncmp(TOKEN_LABEL, reinterpret_cast<const char*>(tokenInfo.label), strlen(TOKEN_LABEL)) == 0) {
slot = slotList[i];
printf("found token: %lu\n", slot);
foundToken = true;
}
}
if (!foundToken) {
printf("token not found\n");
p11->C_CloseSession(*session);
return false;
}
rv = p11->C_OpenSession(slot, CKF_SERIAL_SESSION | CKF_RW_SESSION, NULL, NULL, session);
if (rv != CKR_OK) {
printf("Failed to open session: %lu\n", rv);
p11->C_CloseSession(*session);
return false;
}
CK_OBJECT_CLASS keyClass = CKO_PRIVATE_KEY;
CK_ATTRIBUTE keyTemplate[] = {
{CKA_CLASS, &keyClass, sizeof(keyClass)}
};
rv = p11->C_FindObjectsInit(*session, keyTemplate, 1);
if (rv != CKR_OK) {
std::cerr << "FindObjectsInit failed." << std::endl;
p11->C_CloseSession(*session);
return false;
}
rv = p11->C_FindObjects(*session, key, 1, &count);
if (rv != CKR_OK || count != 1) {
std::cerr << "FindObjects failed." << std::endl;
p11->C_CloseSession(*session);
return false;
}
rv = p11->C_FindObjectsFinal(*session);
if (rv != CKR_OK) {
std::cerr << "FindObjectsFinal failed." << std::endl;
p11->C_CloseSession(*session);
return false;
}
return true;
}
// Function to keep signing a string
void signLoop() {
CK_SESSION_HANDLE session;
CK_OBJECT_HANDLE key;
if (!get_key(&session, &key)) {
return;
}
unsigned char data[] = "Hello World";
unsigned char signature[256];
CK_ULONG sigLen = sizeof(signature);
printf("sign loop\n");
uint64_t success = 0;
CK_RV rv;
while (true) {
CK_MECHANISM mechanism = {CKM_RSA_PKCS, NULL_PTR, 0};
rv = p11->C_SignInit(session, &mechanism, key);
if (rv != CKR_OK) {
std::cerr << "SignInit failed: "<<rv<< std::endl;
break;
}
rv = p11->C_Sign(session, data, sizeof(data), signature, &sigLen);
if (rv != CKR_OK) {
std::cerr << "Sign operation failed: "<<rv<< std::endl;
break;
}
success++;
if (success % 1000 == 0) {
printf("success: %lu\n", success);
}
}
printf("sign operation failed after %lu success\n", success);
p11->C_CloseSession(session);
}
// performs one sign operation and sleeps forever
void one_sign() {
CK_SESSION_HANDLE session;
CK_OBJECT_HANDLE key;
if (!get_key(&session, &key)) {
return;
}
CK_MECHANISM mechanism = {CKM_RSA_PKCS, NULL_PTR, 0};
CK_RV rv = p11->C_SignInit(session, &mechanism, key);
if (rv != CKR_OK) {
std::cerr << "SignInit failed: "<<rv<< std::endl;
return;
}
unsigned char data[] = "Hello World";
unsigned char signature[256];
CK_ULONG sigLen = sizeof(signature);
rv = p11->C_Sign(session, data, sizeof(data), signature, &sigLen);
if (rv != CKR_OK) {
std::cerr << "Sign operation failed: "<<rv<< std::endl;
return;
}
printf("sign ok\n");
// sleep forever
std::promise<void>().get_future().wait();
}
int main(int argc, char *argv[]) {
if (!init_pkcs11()) {
return 0;
}
if (argc > 1 && strcmp(argv[1], "one") == 0) {
one_sign();
} else {
signLoop();
}
p11->C_Finalize(NULL);
return 0;
}
@nikolkam Sorry I do not have the environment to compile your c++ application :-( I checked the tcti trace of Esys_StartAuthSession. The handle 0x80ffffff was used for the tpm key and the bind key. But transient last is defined as 0x80fffffe. Could you please attach the tcti part of your trace with the call of Esys_StartAuthSession which did not produce an error? Perhaps it would make sense to check the application with valgrind.
@nikolkam I can reproduce the tcti error if i call Tss2_Sys_StarAuthSession with the tpmKey and bind set to 0x80ffffff for an AMD firmware TPM and an Infineon SLB9670 TPM.
@nikolkam In the meantime I found the time to test your program with tpm2-tss and tpm2-pkcs11. I did use the master branches of both and the tpm2-abrmd resource manager. The error did not occur after about 65k signatures:
success: 70000
@nikolkam If i use the kernel resource manager I also get tcti errors with your test example. But they occur quite quickly. The number of sign operations until the error does occur is different. Also the TPM command which causes the failure is not always StartAuthSession. This makes it look like it is a bug in the kernel resource manager (I did use 5.15.0).
@JuergenReppSIT
Thank you for taking a look into the issue.
Just to make sure, when you use your kernel manger, are you testing on your physical machine? The number of sign operations seems to be different on different environments. On vm it takes me about 65k operations, but on physical machines I remember it was below 1k sign operations. Does the number of sign operation to get to fail state differ each time you execute the program?
In our production code, we call tpm2 flushcontext --saved-session
to workaround this issue when this happens. But since one of the program holding the tpm2 handle is a browser and we have no control over it, it results in failing browser's cert sign capability. If this issue looks like a lower level kernel issue which cannot be fixed with tpm2 library, do you have any suggestions or better way to work around this issue?
@nikolkam yes i did run the test on a physical machine. Perhaps a workaround could be to use the tpm2-abrmd resource manager (https://github.com/tpm2-software/tpm2-abrmd). Yes the number of sign operations did differ. I will try to create an example as simple as possible without the pkcs11 part to reproduce the error.
@nikolkam I must correct my previous comments.
I tried to close the session after n
successful operations an call get_key
again. In this case the error occurred earlier the smallern
was.
Did you run your tests with tpm2-abrmd?
@nikolkam The error 0x901 (TPM2_RC_CONTEXT_GAP) in the kernel resource manager did occur in the kernel function tpm2_save_context
:
[ 401.923826] tpm tpm0: tpm2_save_context: failed with a TPM error 0x0901
[ 401.925049] tpm tpm0: A TPM error (459) occurred flushing context
This error is not returned to tss. This error is described in the Architecture Part of the TPM Spec 30.4. There were no calls of ContextSave
from tss. So it seems to be a problem of the kernel resource manager.
The error also occurs in the kernel if tpm2-abrmd is used but tpm2-abrmd handles the error.
@JuergenReppSIT Sorry for the late reply, I tried to run sign loop test in the abrmd environment and it's working all good. Thank you for the help. Should this issue be reported somewhere else in that case?
@nikolkam Also sorry for late replay. I just sent an e-mail related to this problem to linux-integrity@vger.kernel.org
I have two test programs running on a same computer.
I start program 1 and then program 2 and let it keep running. When it reaches around 65k (around 16 bytes maximum number), sign operation using the token starts failing and does not recover unless I close every program holding a session or I run
tpm2 flushcontext --saved-session
. It seems like it fails inside StartAuthSession and fails to read from a tpm device. The error was reproducable on both vm and physical machines so I don't feel like it's a hardware issue. Is there any known limitation/bugs with tpm that I'm not aware of?Here is a trace log of trying to do sign operation on a vm with failed state of tpm.