Assuming pkcs11j is meant to be PKCS11 2.40 compliant (which the comments seem to imply), the structure for the CK_GCM_PARAMS is incorrect. In PKCS11 2.40 it is defined as:
Ok, noticed that pkcs11j defines both CK_AES_GCM_PARAMS and CK_AES_GCM_240_PARAMS, but both give the same error with the HSM so will need to investigate further.
Assuming pkcs11j is meant to be PKCS11 2.40 compliant (which the comments seem to imply), the structure for the CK_GCM_PARAMS is incorrect. In PKCS11 2.40 it is defined as:
typedef struct CK_GCM_PARAMS { CK_BYTE_PTR pIv; CK_ULONG ulIvLen; CK_BYTE_PTR pAAD; CK_ULONG ulAADLen; CK_ULONG ulTagBits; } CK_GCM_PARAMS;
whereas the current code uses the structure from PKCS11 3.0 and above (added ulIvBits field):
typedef struct CK_GCM_PARAMS { CK_BYTE_PTR pIv; CK_ULONG ulIvLen; CK_ULONG ulIvBits; CK_BYTE_PTR pAAD; CK_ULONG ulAADLen; CK_ULONG ulTagBits; } CK_GCM_PARAMS;
Consequently we are getting CKR_MECHANISM_PARAM_INVALID when testing against SafeNet ProtectServer 3 HSM.