migration process in fusionauth - 1.15.8 to 1.47.1

[jcsferraz]

(Unexpected error. We're missing an internal API key)

fusionAuth-App : 1.47.1 version PostgresSQL : 16.1 version ElasticSearch: 7.17.3 (chart version)

Description

I'm trying to migrate FusionAuth to version 1.47.1 implemented in my Kubernetes cluster.

Challenges:

I currently have a database with version 9.6.24 of Postgres.

The current version of fusioauth that I use is 1.15.8

I implemented in parallel a more current version of fusionauth (1.47.1) with PostgresSQL: 16.1 version.

00 - I stopped version 1.47.1 that runs in parallel on my cluster

kubectl scale --replicas=0 deploy/fusion-fusionauth-beta -n default 01 - I dumped my base first corresponding to the previous version:

02 - I returned with this database dump for the clean and unused version of fusionAuth 1.47.1 within my kubernetes cluster.

03 - The restore of the base occurred normally and immediately afterwards I used the schema and ran the migrations through here:

migrations

04 - After initializing my fusionAuth 1.47.1 again I receive the following error:

fusionauth-beta-7f48f96b5d-q8wmh 0/1 CrashLoopBackOff 2 (12s ago) 66s

Unable to start the server. Here's why: [java.lang.RuntimeException: Unexpected error. We're missing an internal API key.] -> [class java.lang.RuntimeException] Unexpected error. We're missing an internal API key. [Guice/ErrorInCustomProvider]: RuntimeException: Unexpected error. We're missing an internal API key. while locating FusionAuthInternalAPIKeyProvider at ServiceModule.configure(ServiceModule.java:304) \_ installed by: FusionAuthModule -> Modules$OverrideModule -> APIModule -> ServiceModule at DefaultEventService.<init>(DefaultEventService.java:87) \_ for 5th parameter while locating DefaultEventService at EventHelper.eventService(EventHelper.java:17) \_ for field eventService Caused by: java.lang.RuntimeException: Unexpected error. We're missing an internal API key. at io.fusionauth.api.domain.guice.FusionAuthInternalAPIKeyProvider.lambda$get$5(FusionAuthInternalAPIKeyProvider.java:39) at java.base/java.util.Optional.orElseThrow(Optional.java:403)

Is there something wrong I'm doing that shouldn't be done?

[robotdan]

From the error you are seeing, it would appear that the 1.46.0.sql migration did not complete correctly. Assuming you ran each migration in order from 1.16.0 -> 1.47.0 in the .zip schema package?

[jcsferraz]

hi ! thank you for your answer.

I followed the next step to migrations :

first:

I took the migrations on the this [link].

second:

I unzip the file.

third:

I entered on this folder after unzip: $ cd $HOME/fusionauth-database-schema-1.47.1/migrations/postgresql

fourth:

I restored the postgres dump on version 9.6.24 to my new postgres deploy version 1.16 from helm chart bitnami/postgres chart version 13.3.0:

PG_PASSWORD="$POSTGRES_PASSWORD" psql --host 127.0.0.1 -U postgres -d fusionauth -p 5432 < dump-postgres-version-9-6-24-2024-01-12-fusionauth-staging-env.sql

I builded the shell script to automation:

$ migrations=$(find -E . -regex './1\.(16|[1][7-9]|[2-5][0-9]).([0-9]{1,}).sql' -print0 | xargs -0 ls)

$ echo $migrations

$ for file in ${migrations[@]}; do PG_PASSWORD="$POSTGRES_PASSWORD" psql --host 127.0.0.1 -U postgres -d fusionauth -p 5432 < $file ; done

and after that receive the error:

Unexpected error. We're missing an internal API key.

[jcsferraz]

I have good news, but first I need to confirm the behavior of FusionAuth:

I followed the following steps without needing to do the migrations manually, so please let me know if this is correct, or if I skipped any steps!

first:

I implemented a new deployment of Postgres 1.16 without any database, that is, a clean installation with a different name than the existing one in use, that is, with the name postgres-fusioauth-beta.

Then, I just restored the base in Postgres 9.6.24 to this new base in Postgres 1.16 without performing migrations.

I implemented a new deployment of elasticsearch 7.17.3, that is, a clean installation with a different name than the existing one in use in the same cluster, that is, with the name elasticsearch-fusionauth-beta.

I implemented the new deployment of fusionauth version 1.47.1 in the same cluster with a different name than the fusionauth in use, that is, I created a deploy with the name fusionauth-beta.

01 - For fusionauth, I used the values.yaml found here.

And I guaranteed the existence of the values below:

app:
     runtimeMode: production
environment:
     - name: FUSIONAUTH_APP_SILENT_MODE
       value: "true"

I implemented fusionauth with chart version 0.12.1 (1.47.1) in the tag: mentioned inside the values.yaml above.

To my surprise:

When I upgraded fusionauth 1.47.1:

• All the data was there, without any errors, and I would like to know if it is correct.

In the logs, I saw this below, but I would like to confirm if it is correct, I would like to understand in the log if the migrations were carried out by fusionauth:

helm upgrade --install fusion-fusionauth-beta fusionauth/fusionauth --version 0.12.1 -f fusionauth-1-47-1-values.yaml -n default Release "fusion-fusionauth-beta" does not exist. Installing it now. NAME: fusion-fusionauth-beta LAST DEPLOYED: Tue Jan 23 15:58:39 2024 NAMESPACE: default STATUS: deployed REVISION: 1 NOTES:

kubectl logs svc/fusion-fusionauth-beta -n default -f
Defaulted container "fusionauth" out of: fusionauth, wait-for-db (init), wait-for-search (init)
Starting fusionauth-app...
  --> Logging to /usr/local/fusionauth/logs/fusionauth-app.log
2024-01-23 06:58:47.339 PM INFO  io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Loading FusionAuth configuration file [/usr/local/fusionauth/config/fusionauth.properties]
2024-01-23 06:58:47.407 PM INFO  io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Dynamically set property [fusionauth-app.url] set to [http://localhost:9011/]
2024-01-23 06:58:47.409 PM INFO  com.inversoft.configuration.BasePropertiesFileInversoftConfiguration -
  - Overriding default value of property [database.mysql.enforce-utf8mb4] with value [true]
  - Overriding default value of property [FUSIONAUTH_APP_RUNTIME_MODE] with value [production]
  - Overriding default value of property [SEARCH_TYPE] with value [elasticsearch]

2024-01-23 06:58:47.477 PM INFO  io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Loading FusionAuth configuration file [/usr/local/fusionauth/config/fusionauth.properties]
2024-01-23 06:58:47.478 PM INFO  io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Dynamically set property [fusionauth-app.url] set to [http://localhost:9011/]
2024-01-23 06:58:47.480 PM INFO  com.inversoft.configuration.BasePropertiesFileInversoftConfiguration -
  - Overriding default value of property [database.mysql.enforce-utf8mb4] with value [true]
  - Overriding default value of property [FUSIONAUTH_APP_RUNTIME_MODE] with value [production]
  - Overriding default value of property [SEARCH_TYPE] with value [elasticsearch]

2024-01-23 06:58:47.717 PM INFO  com.inversoft.maintenance.DefaultMaintenanceModeWorkflow -

---------------------------------------------------------------------------------------------------------
---------------------------------- Entering Silent Configuration Mode -----------------------------------
---------------------------------------------------------------------------------------------------------

2024-01-23 06:58:48.319 PM INFO  com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - [RHuecQ] Attempting to lock database to support multi-node configurations
2024-01-23 06:58:48.373 PM INFO  com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - [RHuecQ] Obtained a database lock
2024-01-23 06:58:48.603 PM INFO  com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - [RHuecQ] Database Version [1.47.0]
2024-01-23 06:58:48.637 PM INFO  com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - [RHuecQ] Latest Migration Version [1.47.0]
2024-01-23 06:58:48.639 PM INFO  com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - [RHuecQ] Attempting to unlock database to support multi-node configurations
2024-01-23 06:58:48.639 PM INFO  com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - [RHuecQ] Unlock completed
2024-01-23 06:58:48.692 PM INFO  com.inversoft.search.ElasticRestClientHelper - Use credentials, username [elastic] password [********]
2024-01-23 06:58:48.695 PM INFO  com.inversoft.search.ElasticRestClientHelper - Connecting to Elasticsearch at [http://elastic-fusionauth-beta.core.svc.cluster.local:9200/]
2024-01-23 06:58:49.838 PM INFO  io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Loading FusionAuth configuration file [/usr/local/fusionauth/config/fusionauth.properties]
2024-01-23 06:58:49.840 PM INFO  io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Dynamically set property [fusionauth-app.url] set to [http://localhost:9011/]
2024-01-23 06:58:49.841 PM INFO  com.inversoft.configuration.BasePropertiesFileInversoftConfiguration -
  - Overriding default value of property [database.mysql.enforce-utf8mb4] with value [true]
  - Overriding default value of property [FUSIONAUTH_APP_RUNTIME_MODE] with value [production]
  - Overriding default value of property [SEARCH_TYPE] with value [elasticsearch]

This is correct form ? I forgot any steps ? Thanks!

[robotdan]

Sounds like everything is working now?

If you start up FusionAuth it will apply the migrations for you based upon your current schema version and the available migrations.

If you want to apply your own migrations prior to starting FusionAuth using the SQL migrations that works too. I think that is what you are doing here.

Assuming your script that you mentioned applies everything in order I would expect this to behave the same way that FusionAuth performs the migration during startup.

Hard to know what went wrong during your manual schema migration, but if you can recreate, please re-open and we can dig in further.