While the default primality check implemented here (Baillie-PSW test) is time-tested with no counter-examples found, FIPS standard (https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.186-4.pdf) is more strict. If this library is to be used for RSA, one needs to go through that standard and see if the performed checks, and the details of those checks are compatible with FIPS. In particular,
FIPS requires several Miller-Rabin checks, depending on the prime size, instead of the two we currently perform.
The Lucas test may need to use the Baillie's Method A* for selecting the base instead of currently implemented Method A
Miller-Rabin algorithm needs to be compared with that in FIPS (Sections C.3.1 and C.3.2)
Lucas algorithm needs to be compared with that in FIPS (Section C.3.3)
Supporting algorithms too (sieving, square root, Jacobi symbol etc - see Appendix C in general)
While the default primality check implemented here (Baillie-PSW test) is time-tested with no counter-examples found, FIPS standard (https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.186-4.pdf) is more strict. If this library is to be used for RSA, one needs to go through that standard and see if the performed checks, and the details of those checks are compatible with FIPS. In particular,