Deploy with Vagrant errors.

I've been trying to do a basic setup with vagrant & virtual box (./provision vm) and keep running errors that I'm pretty sure is due to outdated code. I finally ran into a wall and can't move forward. Based off these errors, it wouldn't surprise me if it's broken for all deployments, not just vagrant.

Computer Specs macOS Ventura (13.4.1) Intel Chipset Lastest VirtualBox, Vagrant and Ansible versions on homebrew. /etc/host file has been updated as requested.

First Error

TASK [mongo : Sign certificate with our CA] ********************************************
fatal: [192.168.66.99 -> localhost]: FAILED! => {"changed": false, "module_stderr": "sudo: a password is required\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

I fixed it by adding ansible_become: false to Sign Certificate with our CA (roles/mongo/tasks/certs.yml)

- name: Sign certificate with our CA
  community.crypto.x509_certificate_pipe:
    content: "{{ (certificate.content | b64decode) if certificate_exists.stat.exists else omit }}"
    csr_content: "{{ csr.csr }}"
    provider: ownca
    ownca_path: "{{ inventory_dir }}/secrets/mongo/mongoca.pem"
    ownca_privatekey_path: "{{ inventory_dir }}/secrets/mongo/mongoca.key"
    ownca_privatekey_passphrase: "{{ mongo_ca_passphrase }}"
    ownca_not_after: +365d # valid for one year
    ownca_not_before: "-1d" # valid since yesterday
  delegate_to: localhost
  register: certificate
  **vars:
    ansible_become: false**

Second Error

TASK [mongo : Wait until cluster health is ok] *****************************************
fatal: [192.168.66.99]: FAILED! => {"changed": false, "msg": "You must use pymongo 4+."}

I fixed it by updating it to pymongo 4.0.1

- name: Install pymongo
  ansible.builtin.pip:
    name: pymongo
    version: "4.0.1"

Third Error

TASK [manage_provision_entities : Check if the entity already exists] ******************
FAILED - RETRYING: [192.168.66.99]: Check if the entity already exists (15 retries left).
FAILED - RETRYING: [192.168.66.99]: Check if the entity already exists (14 retries left).
FAILED - RETRYING: [192.168.66.99]: Check if the entity already exists (13 retries left).
FAILED - RETRYING: [192.168.66.99]: Check if the entity already exists (12 retries left).
FAILED - RETRYING: [192.168.66.99]: Check if the entity already exists (11 retries left).
FAILED - RETRYING: [192.168.66.99]: Check if the entity already exists (10 retries left).
FAILED - RETRYING: [192.168.66.99]: Check if the entity already exists (9 retries left).
FAILED - RETRYING: [192.168.66.99]: Check if the entity already exists (8 retries left).
FAILED - RETRYING: [192.168.66.99]: Check if the entity already exists (7 retries left).
FAILED - RETRYING: [192.168.66.99]: Check if the entity already exists (6 retries left).
FAILED - RETRYING: [192.168.66.99]: Check if the entity already exists (5 retries left).
FAILED - RETRYING: [192.168.66.99]: Check if the entity already exists (4 retries left).
FAILED - RETRYING: [192.168.66.99]: Check if the entity already exists (3 retries left).
FAILED - RETRYING: [192.168.66.99]: Check if the entity already exists (2 retries left).
FAILED - RETRYING: [192.168.66.99]: Check if the entity already exists (1 retries left).
fatal: [192.168.66.99]: FAILED! => {"attempts": 15, "cache_control": "no-cache", "changed": false, "connection": "close", "content_length": "107", "content_type": "text/html", "elapsed": 0, "msg": "Status code was 503 and not [200]: HTTP Error 503: Service Unavailable", "redirected": false, "status": 503, "strict_transport_security": "max-age=34214400", "url": "https://manage.vm.openconext.org/manage/api/internal/search/saml20_sp"}

This is the error I cannot get past. The error looks like it's just a status code that isn't matching exactly, but I'm not sure where to fix this. Nor what other errors might come after. I figured it is best to submit a new issue with people who know the system better than I.

Edit: I set up an AWS EC2 instance running with CentOS 7, and ran the whole https://github.com/OpenConext/OpenConext-deploy/wiki/Installation-steps-to-deploy-OpenConext-on-a-single-system-other-than-the-Vagrant-VM-centOS7 steps, complete with the environments and I ran into the same exact errors.

Thanks @heuscherl for bringing this to our attention. I think all the issues are fixed. Can you please test it using the branch feature/devvm_mongo_fixes as mentioned in https://github.com/OpenConext/OpenConext-deploy/pull/437 ?

It is now working properly for Vagrant, but it's still not working when deploying on a host of our choice.

I checked out the branch, and created a new environment to make sure everything was updated properly. When I ran ./provision uedev centos environments-external/uedev/secrets/uedev.yml, I ended up with the following error:

TASK [mongo : Wait until cluster health is ok] ***************************************** fatal: [172.100.8.140]: FAILED! => {"changed": false, "msg": "Unable to get MongoDB server version: Authentication failed., full error: {'ok': 0.0, 'errmsg': 'Authentication failed.', 'code': 18, 'codeName': 'AuthenticationFailed', '$clusterTime': {'clusterTime': Timestamp(1692052514, 1), 'signature': {'hash': b'\\x04\\x94\\x97k9\\xbe\\x88\\x9a\\xeb\\x19\\xd1\\xe0\\xf2\\xd8\\xe4\\xa1\\xdc\\xfe\\xa1\\xeb', 'keyId': 7267300998039732230}}, 'operationTime': Timestamp(1692052514, 1)}"}

I'm pretty sure it's due to the changes made, as I was able to go further into the deployment when I used my old environment. This old environment had the latest code but as the environment had been created from master, it is still running the old versions such as oidcng_version: "6.1.3".

This environment ended up running about 100 more tasks before failing at: TASK [shibboleth : Create tables for shibboleth] *************************************** fatal: [172.100.8.140]: FAILED! => {"changed": false, "msg": "unable to connect to database, check login_user and login_password are correct or /root/.my.cnf has the credentials. Exception message: (1045, u\"Access denied for user 'shibrw'@'ip-172-100-8-140.ec2.internal' (using password: YES)\")"}

While this second error might be the result of us using AWS, the first error does seem to be the result of the changes made in this new branch.

Thank you for your work so far. It was nice to get to play around with it locally, at the very least :)

It turns out that my Authentication Error for the Mongo DB is due to using a different environment prior.

It seems that the code doesn't update passwords for mongo if you try to deploy a new environment with a new password.

Once I copied over the new versions to the old environment, the error I listed above went away.

Now, I'm dealing with the same error as where I got stuck last time: fatal: [172.100.8.140]: FAILED! => {"attempts": 15, "cache_control": "no-cache", "changed": false, "connection": "close", "content_length": "107", "content_type": "text/html", "elapsed": 0, "msg": "Status code was 503 and not [200]: HTTP Error 503: Service Unavailable", "redirected": false, "status": 503, "strict_transport_security": "max-age=34214400", "url": "https://manage.auth.ly/manage/api/internal/search/saml20_sp"}

I'm going to try adding: mongo_tls_host_altname_dnsorip: IP to the group_vars for the environment.

Otherwise... I'm gonna have to try to figure out how to wipe the data, as I can't use a new environment due to the password issue, but I think that is one of the reasons why it's not working.

The error you see means that the app "manage" is not working. I assume that that is caused by a non-working mongo database. Mixing up the passwords in environments can cause the errors.

mongo_tls_host_altname_dnsorip: IP is needed when your inventory points to an ip address in stead of a hostname.

If that is the case, you might be able to solve it adding that parameter.

If this fails, you might want to reinstall mongo completely. Remove all mongo packages, and delete all data in /var/lib/mongo to do so. You can reinstall mongo simply by running only the mongo role (to make this process faster):

./provision vm --tags mongo

Starting from scratch might also be quicker.

I started from scratch and ended up getting the following error again:

TASK [shibboleth : Create tables for shibboleth] *************************************** fatal: [172.100.8.140]: FAILED! => {"changed": false, "msg": "unable to connect to database, check login_user and login_password are correct or /root/.my.cnf has the credentials. Exception message: (1045, u\"Access denied for user 'shibrw'@'ip-172-100-8-140.ec2.internal' (using password: YES)\")"}

After doing some research, it turns out that the deployment doesn't grant permissions in mysql for users using ip addresses outside of localhost (which instructions say to set it to your IP). I had to manually go into the database and grant permissions to shibrw (let me know if I need to do this for other users, as I only did it for shibrw):

GRANT ALL PRIVILEGES ON *.* TO 'USERNAME'@'%' IDENTIFIED BY 'PASSWORD' with grant option;

I also had to set the bind-address to 0.0.0.0.

This allowed me to get further than I have previously, as it actually started some services, such as the mongodb.service and shibd.services.

However, it still ended up failing at the same part as my last post

`TASK [manage_provision_entities : Check if the entity already exists] **********
FAILED - RETRYING: [172.100.8.140]: Check if the entity already exists (15 retries left).
FAILED - RETRYING: [172.100.8.140]: Check if the entity already exists (14 retries left).
FAILED - RETRYING: [172.100.8.140]: Check if the entity already exists (13 retries left).
FAILED - RETRYING: [172.100.8.140]: Check if the entity already exists (12 retries left).
FAILED - RETRYING: [172.100.8.140]: Check if the entity already exists (11 retries left).
FAILED - RETRYING: [172.100.8.140]: Check if the entity already exists (10 retries left).
FAILED - RETRYING: [172.100.8.140]: Check if the entity already exists (9 retries left).
FAILED - RETRYING: [172.100.8.140]: Check if the entity already exists (8 retries left).
FAILED - RETRYING: [172.100.8.140]: Check if the entity already exists (7 retries left).
fatal: [172.100.8.140]: FAILED! => {"attempts": 15, "cache_control": "no-cache", "changed": false, "connection": "close", "content_length": "107", "content_type": "text/html", "elapsed": 0, "msg": "Status code was 503 and not [200]: HTTP Error 503: Service Unavailable", "redirected": false, "status": 503, "strict_transport_security": "max-age=34214400", "url": "https://manage.auth.ly/manage/api/internal/search/saml20_sp"}

PLAY RECAP **************************************************************************************
172.100.8.140              : ok=300  changed=0    unreachable=0    failed=1    skipped=157  rescued=0    ignored=0
`

Using systemctl status, I can see that the mongodb.service, shibd.service and mariadb.services are all running without errors, but httpd.service is throwing errors such as:


`[proxy:error] [pid 753] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:9198 (localhost) failed
 [proxy:error] [pid 757] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:9196 (localhost) failed
[proxy:error] [pid 753] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 0s
[proxy:error] [pid 757] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 0s
proxy_http:error] [pid 753] [client 172.100.8.140:49970] AH01114: HTTP: failed to make connection to backend: localhost
[proxy_http:error] [pid 757] [client 172.100.8.140:58250] AH01114: HTTP: failed to make connection to backend: localhost`

What is weird is that it is trying to connect to 127.0.0.1 as this IP address isn't listed anywhere in our environment.

I'm honestly running out of ideas on how to debug this or to gain more information at this point. Perhaps you have ideas on how to narrow this down, or at least have an idea on where to go from here.

Here is the information about my setup:

AWS Instance running Centos 7 Target IP: 172.100.8.140 Domain: auth.ly

I set the %target_host% as requested to 172.100.8.140. Changed the template file name to 172.100.8.140.yml (as well as all the apache_app_listen_addresses).' Added mongo_tls_host_altname_dnsorip: IP as the inventory points to in IP and not host.

As for the public IP... this is where things might be having issues. There is no public IP. As we want this in our boundary, under our VPN, we use security groups and inbound/outbound rules instead of having a public IP. Thus, the 'public IP' is the same as our target IP/private IP (172.100.8.140). I also removed the restricted flags as requested when only having one IP.

All this being said, I made an elastic IP to use as a public IP, and tried it from scratch, and still ended up with the same errors.

Lastly, we set the DNS entries in the auth.ly domain with the following, as indicated by the instructions:

172.100.8.140 db.auth.ly
172.100.8.140 engine.auth.ly
172.100.8.140 mujina-idp.auth.ly
172.100.8.140 mujina-sp.auth.ly
172.100.8.140 connect.auth.ly
172.100.8.140 profile.auth.ly
172.100.8.140 static.auth.ly
172.100.8.140 teams.auth.ly
172.100.8.140 voot.auth.ly
172.100.8.140 aa.auth.ly
172.100.8.140 pdp.auth.ly
172.100.8.140 engine-api.auth.ly
172.100.8.140 manage.auth.ly

I can ping these address from the host, as well as nslookup them on my computer, which gives me the 172.100.8.140 IP.

Do you have any ideas what could be causing all this?

Well... I think It kinda working...somehow.

I started over from scratch again, and instead of running every role, I went line by line in the provision.yml using tags.

My first issue was the one I expected, with the mysql users privileges not being granted properly.

After that, I ran into the same error I kept getting stuck at with manage.auth.ly 503 error. As I was trying everything, I decided to skip the app_php tag and move on. After finishing app_java and java_apps, I tried app_php again and suddenly it was working.

I still have yet to have a successful full provision. Something always happens, and it's really bizarre things too.

I've had this error pop up even though nothing I have is referencing 127.0.0.1:

TASK [springboot : oidcng-server | wait for oidcng to start] ********************
FAILED - RETRYING: [172.100.8.140]: oidcng-server | wait for oidcng to start (3 retries left).
FAILED - RETRYING: [172.100.8.140]: oidcng-server | wait for oidcng to start (2 retries left).
FAILED - RETRYING: [172.100.8.140]: oidcng-server | wait for oidcng to start (1 retries left).
fatal: [172.100.8.140]: FAILED! => {"attempts": 3, "changed": false, "elapsed": 120, "msg": "Timeout when waiting for 127.0.0.1:9195"}

Yet, I can manually start this service with no issues... I cannot get past this step though. It's just bizarre.

I am having the same issues with: AA, PDP, AALink, [Fri Aug 18 06:06:20.385242 2023] [proxy:error] [pid 30094] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:9189 (localhost) failed

There is something up with the provision where sometimes it defaults to 127.0.0.1 and then fails... and there is no reason for it.

Do you have any advice to stop several of these services from trying to hit 127.0.0.1?

There's lots of things going on here, it's difficult from a distance to determine what's going on.

First of all, the mysql permissions issue is a bit puzzling to me. The task that creates the users uses {{ mysql_host }} as host that is allowed to connect, which is set to localhost in the template. Permissions should be ok for all apps that use mysql (eb, teams, aa, pdp).

I was also wondering, when starting over with new secrets, did you also clean up the mongo and mysql data directories? Otherwise the old passwords are still there.

I have made some changes in the PR with improvements. to make sure that at least the database backend for Shibboleth is disabled. This is only needed in high availability scenario's. You can achieve the same thing by setting:

shibboleth_database_backend: false

in your group_vars/yourenvironment.yml and rerun the shibboleth role. Not really needed by the way if thing are working as expected.

The the problem with the check of the java apps. You can check their logs in /var/log/$appname/$app.log, so: /var/log/oidcng/oidcng.log /var/log/manage/manage.log

I think the issue here might be related to the connection to mongo.

I apologize for not being clearer. By starting from scratch, I meant I reverted the EC2 instance back to the default state (directly after installing the OS). None of the old directories or passwords were kept.

As for the mysql permissions, perhaps it is an AWS issue. All I know is that localhost was not sufficent enough for later tasks. Later connections were all trying to log in with my host (mysql -u %user% -h 172.100.8.140 -p %password%) and not through localhost. This would cause the authentication to fail as that user didn't have permissions on that host.

I ended up fixing this during the build by adding a host attribute for each user in the group_vars:

  users:
    - {
        name: teamsrw,
        db_name: teams,
        password: "{{ mysql_passwords.teams }}",
        hosts: ["172.100.8.140"],
      }

And by modifying the roles/mysql/tasks/main.yml to add an additional task to make sure both localhost & the IP were granted the correct provisions (this code will need to modified for the public as you can't put more than one host in each permission. You need to have two 'separate' users. The join here will cause mysql to fail if you have more than one host per user in the group_var).

- name: Create database user
  mysql_user:
    name: "{{ item[0].name }}"
    host: "{{ item[0].hosts | join(',') if item[0].hosts is defined and (item[0].hosts | length>0) else item[1] }}"
    password: "{{ item[0].password }}"
    priv: "{{ item[0].db_name }}.*:ALL"
    state: present
  no_log: true
  with_nested:
    - "{{ databases.users }}"
    - "{{ database_clients }}"
  tags:
    - mysql

- name: Create database local user
  mysql_user:
    name: "{{ item[0].name }}"
    host: "{{ item[1] }}"
    password: "{{ item[0].password }}"
    priv: "{{ item[0].db_name }}.*:ALL"
    state: present
  no_log: true
  with_nested:
    - "{{ databases.users }}"
    - "{{ database_clients }}"
  tags:
    - mysql

Moving onto my current issues. I was going through the logs yesterday, and found that I had no mongo issues that I can see. Nothing was in the logs, and I could log into each of the users. I was even able to start up the service without any issues. manage.auth.ly was working as intended, but it was broken in certain areas because of the failing java apps.

As I recently started from scratch again, not everything is running as it was previously (as I didn't want to skip over oidcng failing this time). But https://engine.auth.ly/ is working and manage.auth.ly redirects to https://engine.auth.ly/authentication/feedback/no-idps (as we don't have a working IDP as mujina installs after oidcng, which is failing).

Here is the results of some of my logs. oidcng is the only interesting one:

manage: not a single error. shibboleth: not a single error. Mostly just successful setup and the listener starting. httpd: error_log just shows successful configurations of auth.ly, static.auth.ly, metadata.auth.ly. mongod.log: mostly just successful authentication and interruptions.

{"t":{"$date":"2023-08-19T03:15:02.339+02:00"},"s":"I",  "c":"ACCESS",   "id":20250,   "ctx":"conn33","msg":"Authentication succeeded","attr":{"mechanism":"SCRAM-SHA-256","speculative":true,"principalName":"admin","authenticationDatabase":"admin","remote":"127.0.0.1:50466","extraInfo":{}}}
{"t":{"$date":"2023-08-19T03:15:02.341+02:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:50472","uuid":"eeb63e86-8016-4dd5-a646-3cec3138d26f","connectionId":34,"connectionCount":11}}
{"t":{"$date":"2023-08-19T03:15:02.341+02:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:50474","uuid":"16b612a9-c9cd-41d7-9601-5d21fd57195c","connectionId":35,"connectionCount":12}}
{"t":{"$date":"2023-08-19T03:15:02.342+02:00"},"s":"I",  "c":"NETWORK",  "id":23838,   "ctx":"conn34","msg":"SSL mode is set to 'preferred' and connection to remote is not using SSL.","attr":{"connectionId":34,"remote":"127.0.0.1:50472"}}

oidcng: Actual errors:

2023-08-18 20:41:31,761 ERROR [main] o.springframework.boot.SpringApplication:824 - Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'springSecurityFilterChain' defined in class path resource [org/springframework/security/config/annotation/web/configuration/WebSecurityConfiguration.class]: 
Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.servlet.Filter]: Factory method 'springSecurityFilterChain' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: 
Error creating bean with name 'relyingPartyRegistrationRepository' defined in class path resource [oidc/secure/SecurityConfiguration$SamlSecurity.class]: 
Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.security.saml2.provider.service.registration.RelyingPartyRegistrationRepository]: Factory method 'relyingPartyRegistrationRepository' threw exception; nested exception is java.lang.ClassCastException: class org.bouncycastle.asn1.pkcs.PrivateKeyInfo cannot be cast to class org.bouncycastle.openssl.PEMKeyPair (org.bouncycastle.asn1.pkcs.PrivateKeyInfo and org.bouncycastle.openssl.PEMKeyPair are in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @7aec35a)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:658)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:486)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
  at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:734)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:408)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:308)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295)
        at oidc.OidcServerApplication.main(OidcServerApplication.java:18)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.servlet.Filter]: Factory method 'springSecurityFilterChain' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'relyingPartyRegistrationRepository' defined in class path resource [oidc/secure/SecurityConfiguration$SamlSecurity.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.security.saml2.provider.service.registration.RelyingPartyRegistrationRepository]: Factory method 'relyingPartyRegistrationRepository' threw exception; nested exception is java.lang.ClassCastException: class org.bouncycastle.asn1.pkcs.PrivateKeyInfo cannot be cast to class org.bouncycastle.openssl.PEMKeyPair (org.bouncycastle.asn1.pkcs.PrivateKeyInfo and org.bouncycastle.openssl.PEMKeyPair are in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @7aec35a)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
 at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.servlet.Filter]: Factory method 'springSecurityFilterChain' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'relyingPartyRegistrationRepository' defined in class path resource [oidc/secure/SecurityConfiguration$SamlSecurity.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.security.saml2.provider.service.registration.RelyingPartyRegistrationRepository]: Factory method 'relyingPartyRegistrationRepository' threw exception; nested exception is java.lang.ClassCastException: class org.bouncycastle.asn1.pkcs.PrivateKeyInfo cannot be cast to class org.bouncycastle.openssl.PEMKeyPair (org.bouncycastle.asn1.pkcs.PrivateKeyInfo and org.bouncycastle.openssl.PEMKeyPair are in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @7aec35a)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)
        ... 29 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'relyingPartyRegistrationRepository' defined in class path resource [oidc/secure/SecurityConfiguration$SamlSecurity.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.security.saml2.provider.service.registration.RelyingPartyRegistrationRepository]: Factory method 'relyingPartyRegistrationRepository' threw exception; nested exception is java.lang.ClassCastException: class org.bouncycastle.asn1.pkcs.PrivateKeyInfo cannot be cast to class org.bouncycastle.openssl.PEMKeyPair (org.bouncycastle.asn1.pkcs.PrivateKeyInfo and org.bouncycastle.openssl.PEMKeyPair are in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @7aec35a)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:658)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:486)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:233)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1284)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1245)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveBean(DefaultListableBeanFactory.java:494)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:349)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:342)
        at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1172)
        at org.springframework.security.config.annotation.web.configurers.saml2.Saml2LoginConfigurer.getBeanOrNull(Saml2LoginConfigurer.java:431)
        at org.springframework.security.config.annotation.web.configurers.saml2.Saml2LoginConfigurer.getSharedOrBean(Saml2LoginConfigurer.java:422)
        at org.springframework.security.config.annotation.web.configurers.saml2.Saml2LoginConfigurer.relyingPartyRegistrationRepository(Saml2LoginConfigurer.java:291)
at org.springframework.security.config.annotation.web.configurers.saml2.Saml2LoginConfigurer.init(Saml2LoginConfigurer.java:232)
        at org.springframework.security.config.annotation.web.configurers.saml2.Saml2LoginConfigurer.init(Saml2LoginConfigurer.java:112)
        at org.springframework.security.config.annotation.AbstractConfiguredSecurityBuilder.init(AbstractConfiguredSecurityBuilder.java:338)
        at org.springframework.security.config.annotation.AbstractConfiguredSecurityBuilder.doBuild(AbstractConfiguredSecurityBuilder.java:300)
        at org.springframework.security.config.annotation.AbstractSecurityBuilder.build(AbstractSecurityBuilder.java:38)
        at org.springframework.security.config.annotation.web.builders.WebSecurity.performBuild(WebSecurity.java:313)
        at org.springframework.security.config.annotation.web.builders.WebSecurity.performBuild(WebSecurity.java:88)
        at org.springframework.security.config.annotation.AbstractConfiguredSecurityBuilder.doBuild(AbstractConfiguredSecurityBuilder.java:305)
        at org.springframework.security.config.annotation.AbstractSecurityBuilder.build(AbstractSecurityBuilder.java:38)
        at org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration.springSecurityFilterChain(WebSecurityConfiguration.java:125)
        at java.base/jdk.intert java.base/java.lang.reflect.Method.invoke(Method.java:566)
        ... 30 common frames omitted
        at org.springframework.security.config.annotation.web.builders.WebSecurity.performBuild(WebSecurity.java:88)
        at org.springframework.security.config.annotation.AbstractConfiguredSecurityBuilder.doBuild(AbstractConfiguredSecurityBuilder.java:305)
        at org.springframework.security.config.annotation.AbstractSecurityBuilder.build(AbstractSecurityBuilder.java:38)
        at org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration.springSecurityFilterChain(WebSecurityConfiguration.java:125)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
        ... 30 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.security.saml2.provider.service.registration.RelyingPartyRegistrationRepository]: Factory method 'relyingPartyRegistrationRepository' threw exception; nested exception is java.lang.ClassCastException: class org.bouncycastle.asn1.pkcs.PrivateKeyInfo cannot be cast to class org.bouncycastle.openssl.PEMKeyPair (org.bouncycastle.asn1.pkcs.PrivateKeyInfo and org.bouncycastle.openssl.PEMKeyPair are in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @7aec35a)nal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
        ... 30 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.security.saml2.provider.service.registration.RelyingPartyRegistrationRepository]: Factory method 'relyingPartyRegistrationRepository' threw exception; nested exception is java.lang.ClassCastException: class org.bouncycastle.asn1.pkcs.PrivateKeyInfo cannot be cast to class org.bouncycastle.openssl.PEMKeyPair (org.bouncycastle.asn1.pkcs.PrivateKeyInfo and org.bouncycastle.openssl.PEMKeyPair are in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @7aec35a)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)
        ... 63 common frames omitted
Caused by: java.lang.ClassCastException: class org.bouncycastle.asn1.pkcs.PrivateKeyInfo cannot be cast to class org.bouncycastle.openssl.PEMKeyPair (org.bouncycastle.asn1.pkcs.PrivateKeyInfo and org.bouncycastle.openssl.PEMKeyPair are in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @7aec35a)

Lastly, when I run systemctl status httpd -l I do get the following errors:

Aug 20 05:41:12 ip-172-100-8-140.ec2.internal Apache-Oidc-Playground[395]: 172.100.8.140 - - [20/Aug/2023:05:41:12 +0200] "GET /internal/health HTTP/1.0" 503 299 "-" "-" "-"
Aug 20 05:41:12 ip-172-100-8-140.ec2.internal Apache-myconext[383]: [Sun Aug 20 05:41:12.798105 2023] [proxy:error] [pid 3259] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:9189 (localhost) failed
Aug 20 05:41:12 ip-172-100-8-140.ec2.internal Apache-myconext[383]: [Sun Aug 20 05:41:12.798118 2023] [proxy:error] [pid 3259] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 0s
Aug 20 05:41:12 ip-172-100-8-140.ec2.internal Apache-myconext[383]: [Sun Aug 20 05:41:12.798122 2023] [proxy_http:error] [pid 3259] [client 172.100.8.140:46200] AH01114: HTTP: failed to make connection to backend: localhost
Aug 20 05:41:12 ip-172-100-8-140.ec2.internal Apache-myconext[396]: 172.100.8.140 - - [20/Aug/2023:05:41:12 +0200] "GET /internal/health HTTP/1.0" 503 299 "-" "-" "-"
Aug 20 05:41:12 ip-172-100-8-140.ec2.internal Apache-AA[389]: [Sun Aug 20 05:41:12.799038 2023] [proxy:error] [pid 1041] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:9198 (localhost) failed
Aug 20 05:41:12 ip-172-100-8-140.ec2.internal Apache-AA[389]: [Sun Aug 20 05:41:12.799051 2023] [proxy:error] [pid 1041] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 0s
Aug 20 05:41:12 ip-172-100-8-140.ec2.internal Apache-AA[389]: [Sun Aug 20 05:41:12.799056 2023] [proxy_http:error] [pid 1041] [client 172.100.8.140:47230] AH01114: HTTP: failed to make connection to backend: localhost
Aug 20 05:41:12 ip-172-100-8-140.ec2.internal Apache-AA[402]: 172.100.8.140 - - [20/Aug/2023:05:41:12 +0200] "GET /aa/api/health HTTP/1.0" 503 299 "-" "-" "-"
Aug 20 05:41:13 ip-172-100-8-140.ec2.internal Apache-EB[399]: 172.100.8.140 - - [20/Aug/2023:05:41:13 +0200] "GET /health HTTP/1.0" 200 15 "-" "-" "-"

I have to say, I'm still baffled to where it's getting 127.0.0.1 as I don't have a single reference to it in my environment properties. By any chance do I need to grant more access in my iptables? I only put in the following:

 sudo iptables -A INPUT -s 172.100.8.140 -p tcp --destination-port 22 -j ACCEPT
 sudo iptables -A INPUT -s 172.100.8.140 -p tcp --destination-port 443 -j ACCEPT
 sudo iptables -A INPUT -s 172.100.8.140 -p tcp --destination-port 9195 -j ACCEPT
 sudo iptables -A INPUT -s 172.100.8.140 -p tcp --destination-port 3306 -j ACCEPT
 sudo iptables -A INPUT -s 172.100.8.140 -p tcp --destination-port 27017 -j ACCEPT

I'll start from scratch again with the changes you suggested right now, but my guess is it's not going to do anything based off the errors I'm getting (but who knows?)

edit: Yep, tried it with shibboleth_database_backend being set to false and recieved the same errors.

Your mysql issues baffle me: The task that creates users and the connection string all use {{ mysql_host }} as variable to set the hostname. This is set to "localhost" in the template. In any case, you can override the variable database_clients in your template by setting it to multiple ip addresses, or ranges. In your case, to add both the external ip and the localhost you use:

database_clients: [ "localhost", "172.100.8.140" ]

The localhost errors in Apache: Your apache virtual host configs (in /etc/httpd/conf.d) have proxy connections, which forward to the app running on localhost. Every Java app runs on its own port.

The oidcng issue is related to the SAML keypair that is created. I've never seen that error before, I've asked our java developer to look into it.

Is the keypair created and present on the filesystem? You can find it here: /opt/oidcng/oidcsaml.key /opt/oidcng/oidcsaml.crt

Yes, both those files exist.

/opt/oidcng/oidcsaml.crt

/opt/oidcng/oidcsaml.key

I wonder if it's supposed to be an RSA key instead (just an idea based off previous research.)

As for Apache, as long as it's supposed to be referencing IP Address 127.0.0.1, I'll shall ignore it. I was just getting confused to why it was showing up when I am using 172.100.8.140. I wanted to make sure it wasn't related to the oidcng issues as oidcng is referencing 127.0.0.1 when trying to connect as well. "Timeout when waiting for 127.0.0.1:9195"

Indeed, the key is in the wrong format. You can fix that by using:

 openssl rsa -in oidcsaml.key -out oidcsaml.key.new
mv  oidcsaml.key.new   oidcsaml.key

If you restart the service oidng, it shouldn't give you any errors anymore.

If that works, you can put the key in your secrets file, under:

oidcng_private_key

And rerun the deployment. oidcng is needed for several other apps, if it fails, other apps cannot function properly as well.

I hope this will fix the issues, and everything installs.

While changing the key helped me move past oidcng, it seems that myconext hasn't been fully set up for installation.

All the below variables I had to add to my group_vars as they were missing and caused my installation to fail:

myconext_tiqr_encryption: secret myconext_sms_bearer: secret

myconext: rp_client_id: myconext.ala.eduid rs_client_id: myconext.rs apns_teamid: apple-team-id apns_keyid: apple-key-id sms_api_url: https://connect.{{ base_domain }}/sms (the only one I am unsure about). nudge_eduid_app_days: 7 remember_me_question_asked_days: 30

I got my answers from here: https://github.com/OpenConext/OpenConext-myconext

The only one I am unsure about is the sms_api_url.

While I was able to fill out the above variables and move forward, these two Ansible tasks failed as they were missing whole contents and files in roles/myconext-server/tasks/main.yml

- name: copy / create GCM config
  copy:
    **src: "{{ inventory_dir }}/files/myconext/firebase.json"**
    dest: "{{ myconext_dir }}/firebase.json"
    owner: "root"
    group: "myconext"
    mode: 0740
  notify:
    - "restart myconext"

- name: copy / create APNS certificate
  copy:
    **content: "{{ myconext_apns }}"**
    dest: "{{ myconext_dir }}/apns.p8"
    owner: "root"
    group: "myconext"
    mode: 0740
  notify:
    - "restart myconext"

firebase.json isn't located anywhere in your repo, nor is any reference to myconext_apns. I have no idea what is supposed to be here so I am unable to move forward.

Moreover, I tried to skip installing myconext by setting it to false under springboot_services_state but I still got the following error stating that the firebase.json file wasn't found.

TASK [myconext-server : copy / create GCM config] *******************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: If you are using a module and expect the file to exist on the remote, see the remote_src option
fatal: [172.100.8.140]: FAILED! => {"changed": false, "msg": "Could not find or access '/Users/lindseyheuscher/Software/OpenContext/OpenConext-deploy/environments-external/uedev/files/myconext/firebase.json' on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option"}

EDIT:

While not proper, I decided just to comment out myconext from my springboot_server_services and ran it again.

With myconext not included, I was able to finish the installation and load up manage.auth.ly without issues.

If myconext gets fixed, I'll try to install it at a later time.

You can run the whole suite perfectly without myconext. Myconext is the software that powers the eduID project. If you do not need a guest IdP is not really needed. I will disable it in the template as well.

We would eventually like to have myconext, as we want to check out the eduID project as well. However, it's not a priority. If you wish to close out this issue, go right ahead. I can check back later to see if myconext as been implemented for other hosts in the future.

We could add a feature toggle to disable the mobile-app functions, so you could use the myconext guest IdP easier. I've created a story for that.

Please also join the OpenConext Slack Workspace for any questions, suggestions or suppport.

OpenConext / OpenConext-deploy

Deploy with Vagrant errors. #436