apache / cloudstack

Apache CloudStack is an opensource Infrastructure as a Service (IaaS) cloud computing platform
https://cloudstack.apache.org/
Apache License 2.0
1.98k stars 1.09k forks source link

Cloudstack is not FIPS compliant #6246

Open rwdj opened 2 years ago

rwdj commented 2 years ago
ISSUE TYPE
COMPONENT NAME
UI, more?
CLOUDSTACK VERSION
4.16
CONFIGURATION
OS / ENVIRONMENT

RHEL 8 FIPS compliant system

SUMMARY

Cloudstack uses multiple instances of non-FIPS-compliant features. This makes it impossible to work with in a FIPS compliant environment.

STEPS TO REPRODUCE

On a FIPS compliant system, build as instructed in https://docs.cloudstack.apache.org/en/latest/installguide/building_from_source.html:

$ mvn -P deps

On a FIPS compliant RHEL8 system, install and run cloudstack-management.service from https://download.cloudstack.org/centos/8/4.16/ as instructed in https://docs.cloudstack.apache.org/en/latest/installguide/overview/index.html. See #6232 for why I do the chown and chmod steps.

[root@rwdj ~]# systemctl stop firewalld nftables
[root@rwdj ~]# systemctl start iptables
[root@rwdj ~]# dnf install cloudstack-management mysql-connector-java
[root@rwdj ~]# cloudstack-setup-databases cloud:<omitted> --deploy-as=root:<omitted> -i 127.0.0.1
[root@rwdj ~]# chown :cloud /etc/cloudstack/management/key
[root@rwdj ~]# chmod 0640 /etc/cloudstack/management/key
[root@rwdj ~]# cloudstack-setup-management
EXPECTED RESULTS
Able to build or run cloudstack-management.
ACTUAL RESULTS

Build

$ mvn -P deps

[INFO] Running com.cloud.utils.testcase.NioTest                                                                                                                     
2022-04-11 13:55:28,671 INFO  [utils.testcase.NioTest] (main:) Setting up Benchmark Test                                                                            
2022-04-11 13:55:28,695 INFO  [utils.nio.NioServer] (main:) NioServer started and listening on /0:0:0:0:0:0:0:0:41907                                               
2022-04-11 13:55:28,714 DEBUG [utils.testcase.NioTest] (Time-limited test:) 0/2 tests done. Waiting for completion                                                  
2022-04-11 13:55:28,720 INFO  [utils.testcase.NioTest] (MaliciousNioClientHandler-2:) Connecting to 127.0.0.1:41907                                                 
2022-04-11 13:55:28,720 INFO  [utils.nio.NioClient] (NioClientHandler-1:) Connecting to 127.0.0.1:41907                                                             
2022-04-11 13:55:28,720 INFO  [utils.nio.NioClient] (NioClientHandler-2:) Connecting to 127.0.0.1:41907                                                             
2022-04-11 13:55:28,720 INFO  [utils.testcase.NioTest] (MaliciousNioClientHandler-1:) Connecting to 127.0.0.1:41907                                                 
2022-04-11 13:55:29,715 DEBUG [utils.testcase.NioTest] (Time-limited test:) 0/2 tests done. Waiting for completion                                                  
2022-04-11 13:55:30,715 DEBUG [utils.testcase.NioTest] (Time-limited test:) 0/2 tests done. Waiting for completion                                                  
2022-04-11 13:55:31,716 DEBUG [utils.testcase.NioTest] (Time-limited test:) 0/2 tests done. Waiting for completion                                                  
2022-04-11 13:55:32,003 WARN  [utils.nio.Link] (NioClientHandler-2:) Failed to load keystore, using trust all manager                                               
2022-04-11 13:55:32,003 WARN  [utils.nio.Link] (NioClientHandler-1:) Failed to load keystore, using trust all manager                                               
2022-04-11 13:55:32,041 ERROR [utils.nio.Link] (NioTestServer-NioConnectionHandler-1:) CA service is not configured, by-passing CA manager to create SSL engine     
2022-04-11 13:55:32,043 ERROR [utils.nio.NioConnection] (NioClientHandler-1:) Unable to initialize the threads.                                                     
java.io.IOException: Failed to initialise security                                                                                                                  
        at com.cloud.utils.nio.NioClient.init(NioClient.java:82)                                                                                                    
        at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)                                                                                           
        at com.cloud.utils.testcase.NioTest$ThreadedNioClient.run(NioTest.java:172)                                                                                 
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)                                                                        
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)                                                                                       
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)                                                                
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)                                                                
        at java.base/java.lang.Thread.run(Thread.java:829)                                                                                                          
Caused by: java.security.KeyManagementException: FIPS mode: only SunJSSE TrustManagers may be used                                                                  
        at java.base/sun.security.ssl.SSLContextImpl.chooseTrustManager(SSLContextImpl.java:133)                                                                    
        at java.base/sun.security.ssl.SSLContextImpl.engineInit(SSLContextImpl.java:95)                                                                             
        at java.base/javax.net.ssl.SSLContext.init(SSLContext.java:297)                                                                                             
        ... 9 more
[...]
[INFO] Running com.cloud.utils.rest.HttpClientHelperTest                                                                                                            
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.116 s <<< FAILURE! - in com.cloud.utils.rest.HttpClientHelperTest                         
[ERROR] testCreateClient(com.cloud.utils.rest.HttpClientHelperTest)  Time elapsed: 0.115 s  <<< ERROR!                                                              
java.security.KeyManagementException: FIPS mode: only SunJSSE TrustManagers may be used                                                                             
        at com.cloud.utils.rest.HttpClientHelperTest.testCreateClient(HttpClientHelperTest.java:33)                                                                 

[INFO] Running com.cloud.utils.rest.BasicRestClientTest                                                                                                             
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.087 s <<< FAILURE! - in com.cloud.utils.rest.BasicRestClientTest                          
[ERROR] com.cloud.utils.rest.BasicRestClientTest  Time elapsed: 0.087 s  <<< ERROR!                                                                                 
java.security.KeyManagementException: FIPS mode: only SunJSSE TrustManagers may be used                                                                             
        at com.cloud.utils.rest.BasicRestClientTest.setupClass(BasicRestClientTest.java:62)

Run.

INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstac
k-4.16.1.0.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml]                                                                                
WARNING: An illegal reflective access operation has occurred                                                                                                           
WARNING: Illegal reflective access by net.sf.cglib.core.ReflectUtils$1 (file:/usr/share/cloudstack-management/lib/cloudstack-4.16.1.0.jar) to method java.lang.ClassLoa
der.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)                                                                                        
WARNING: Please consider reporting this to the maintainers of net.sf.cglib.core.ReflectUtils$1                                                                         
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations                                                                  
WARNING: All illegal access operations will be denied in a future release                                                                                              
INFO  [c.c.u.d.T.Transaction] (main:null) (logid:) Is Data Base High Availiability enabled? Ans : false                                                                
WARN  [c.c.u.d.T.Transaction] (main:null) (logid:) Unable to load db configuration, using defaults with 5 connections. Falling back on assumed datasource on localhost:
3306 using username:password=cloud:<omitted>. Please check your configuration                                                                                              
org.jasypt.exceptions.EncryptionInitializationException: java.security.NoSuchAlgorithmException: PBEWithMD5AndDES SecretKeyFactory not available                       
        at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.initialize(StandardPBEByteEncryptor.java:773)                                                            
        at org.jasypt.encryption.pbe.StandardPBEStringEncryptor.initialize(StandardPBEStringEncryptor.java:566)                                                        
        at org.jasypt.encryption.pbe.StandardPBEStringEncryptor.decrypt(StandardPBEStringEncryptor.java:718)                                                           
        at org.jasypt.properties.PropertyValueEncryptionUtils.decrypt(PropertyValueEncryptionUtils.java:72)                                                            
        at org.jasypt.properties.EncryptableProperties.decode(EncryptableProperties.java:230)                                                                          
        at org.jasypt.properties.EncryptableProperties.getProperty(EncryptableProperties.java:172)                                                                     
        at com.cloud.utils.db.TransactionLegacy.initDataSource(TransactionLegacy.java:1034)                                                                            
        at com.cloud.utils.db.TransactionLegacy.<clinit>(TransactionLegacy.java:1008)                                                                                  
        at com.cloud.utils.db.Merovingian2.<init>(Merovingian2.java:68)                                                                                                
        at com.cloud.utils.db.Merovingian2.createLockController(Merovingian2.java:88)                                                                                  
        at com.cloud.server.LockControllerListener.<init>(LockControllerListener.java:33)
        ... more
Caused by: com.mysql.cj.exceptions.SSLParamsException: Cannot open file:NONE [NONE (No such file or directory)]                                                       
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)                                                                   
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)                                            
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)                                    
        ... 90 more
Caused by: java.io.FileNotFoundException: NONE (No such file or directory)
        at java.base/java.io.FileInputStream.open0(Native Method)
        at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
        at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
        ... 90 more

Upon checking, I found utils/src/main/java/com/cloud/utils/crypt/DBEncryptionUtil.java#L81, where PBEWithMD5AndDES is not FIPS compliant. I'm concerned there are also other spots to check that I can't check because it doesn't build which brings me to the extent of my knowledge.

FIPS compliance can be a stickler, because the core documentation isn't very clear from what I can find. Regarding PBEWithMD5AndDES, after a few searches, the most clear documentation on approved algorithms I could find was this (which is out of date with FIPS 140-3 existing, but probably still correct enough): https://csrc.nist.gov/csrc/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp2261.pdf And additional resources that may prove useful: https://github.com/jasypt/jasypt/blob/master/jasypt/src/main/java/org/jasypt/util/text/AES256TextEncryptor.java https://docs.oracle.com/en/java/javase/11/docs/specs/security/standard-names.html#cipher-algorithm-names https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/assembly_installing-a-rhel-8-system-with-fips-mode-enabled_security-hardening

weizhouapache commented 2 years ago

@rwdj is this a update of #6232 ? it looks the db configuration file /etc/cloudstack/management/db.properties is missing.

rwdj commented 2 years ago

@weizhouapache, no, #6232 is mostly unrelated.

/etc/cloudstack/management/db.properties isn't missing. Although I noticed those lines in the error as well. But it also doesn't compile.

[root@rwdj ~]# su - cloud -c 'ls -l /etc/cloudstack/management/db.properties'
-rw-r-----. 1 root cloud 3248 Apr 12 09:53 /etc/cloudstack/management/db.properties

It was generated appropriately from:

[root@rwdj ~]# cloudstack-setup-databases cloud:<omitted> --deploy-as=root:<omitted> -i 127.0.0.1
Mysql user name:cloud                                                           [ OK ]                                                                   
Mysql user password:******                                                      [ OK ]                                                                   
Mysql server ip:localhost                                                       [ OK ]                                                                   
Mysql server port:3306                                                          [ OK ]                                                                   
Mysql root user name:root                                                       [ OK ]                                                                   
Mysql root user password:******                                                 [ OK ]                                                                   
Using specified cluster management server node IP 127.0.0.1                     [ OK ]                                                                   
Checking Cloud database files ...                                               [ OK ]                                                                   
Checking local machine hostname ...                                             [ OK ]                                                                   
Checking SELinux setup ...                                                      [ OK ]                                                                   
Preparing /etc/cloudstack/management/db.properties                              [ OK ]                                                                   
Applying /usr/share/cloudstack-management/setup/create-database.sql             [ OK ]                                                                   
Applying /usr/share/cloudstack-management/setup/create-schema.sql               [ OK ]                                                                   
Applying /usr/share/cloudstack-management/setup/create-database-premium.sql     [ OK ]                                                                   
Applying /usr/share/cloudstack-management/setup/create-schema-premium.sql       [ OK ]                                                                   
Applying /usr/share/cloudstack-management/setup/server-setup.sql                [ OK ]                                                                   
Applying /usr/share/cloudstack-management/setup/templates.sql                   [ OK ]                                                                   
Processing encryption ...                                                       [ OK ]                                                                   
Finalizing setup ...                                                            [ OK ]
rwdj commented 2 years ago

A general rule of thumb: if the word "MD5" is ever mentioned at any point, then you'll run into a FIPS issue. FIPS machines aren't able to use MD5 at all. Based on the stacktrace, Java looks to me like it's trying to find files relating to MD5 and they aren't found because Java is running in FIPS-mode. The missing file NONE should be the mentioned "SecretKeyFactory" for PBEWithMD5AndDES, but I'm unaware of the technical details of Java's cipher algorithms.

[rwdj@rwdj ~]$ openssl md5
Error setting digest
140235974117184:error:060800C8:digital envelope routines:EVP_DigestInit_ex:disabled for FIPS:crypto/evp/digest.c:135:

And then there's the issue where it won't compile, flat out refusing on the tests due to FIPS being enabled.

FIPS is a Category 1 (almost no exceptions given) specification of DISA's Security Technical Implementation Guides: https://www.stigviewer.com/stig/red_hat_enterprise_linux_8/2021-12-03/finding/V-230223 or https://stigviewer.com/stig/canonical_ubuntu_20.04_lts/2021-11-19/finding/V-238363

rohityadavcloud commented 2 years ago

Thanks for reporting @rwdj, can you advise how to setup a FIPS complaint EL8 mgmt/KVM host? That is what additional steps were taken before you deployed CloudStack on it, maybe there's a common document that describe this process?

rwdj commented 2 years ago

Regarding additional steps, it would be most of the Red Hat Enterprise Linux 8 Security Technical Implementation Guide, but that's probably too much. Checking full STIG compliance for a system takes way too long for this purpose.

The document https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/assembly_installing-a-rhel-8-system-with-fips-mode-enabled_security-hardening describes enabling FIPS. In general, it's just making sure the system has fips=1 in the kernel parameters during installation/initialization. And then you can use fips-mode-setup --check to verify it worked (which is not the same way it was done in RHEL7, by the way).

Alternatively, an existing system can be made FIPS compliant with as described here, but it's not recommended by Red Hat. I believe there are a number of things done during initialization that are less complex to do during initialization, but that's only my guess.

rwdj commented 2 years ago

Alternatively, here's documentation for an Ubuntu FIPS environment: https://ubuntu.com/security/certifications/docs/fips-enablement

rwdj commented 2 years ago

Found a page on automated DISA STIG full compliance for Ubuntu Pro (including FIPS): https://ubuntu.com/security/disa-stig

sonipl commented 7 months ago

@rohityadavcloud , do you know when FIPS changes are coming ?

weizhouapache commented 7 months ago

@rohityadavcloud , do you know when FIPS changes are coming ?

@sonipl in the past releases, we have implemented a lot of security improvements. For backwards compatibility, some insecure algorithms are still supported (but not default). if possible, can you check and share a list of non-FIPS-compliant code/features in CloudStack ? cc @rwdj @sonipl

weizhouapache commented 7 months ago

7003

New encryptor based on AesGcmJce, which replaced PBEWithMD5AndDes (which is not FIPS-compliant)

8549

use ed25519 instead of rsa when generate public/private keys

rwdj commented 5 months ago

7003 sounds nice.

8719 addresses what I was about to mention about #8549.

I'll setup a couple of fresh FIPS-compliant RHEL machines and see what shakes out. Thanks for the hard work!