search

advancedtelematic / ota-community-edition

End-to-end Over The Air updates
Mozilla Public License 2.0
54 stars 23 forks source link

campaigner pod restart loop due to outdated migrations #110

Closed ddunlap2005 closed 4 years ago

ddunlap2005 commented 4 years ago

Hi,

When I start the latest OTA Community Edition (28 August 2020) using all defaults (plus 4 changes which I have already documented in previous "issues"), the campaigner pod continuous reboots. Looking at the logs, I see:

$ kubectl logs campaigner-76cf7cc696-gtskl I|2020-09-01 16:51:05,945|akka.event.slf4j.Slf4jLogger|Slf4jLogger started W|2020-09-01 16:51:06,262|akka.util.ManifestInfo|Detected possible incompatible versions on the classpath. Please note that a given Akka version MUST be the same across all modules of Akka that you are using, e.g. if you use [2.5.30] all other modules that are released together MUST be of the same version. Make sure you're using a compatible set of libraries. Possibly conflicting versions [2.5.30, 2.5.26] in libraries [akka-protobuf:2.5.30, akka-actor:2.5.30, akka-slf4j:2.5.26, akka-stream:2.5.30] I|2020-09-01 16:51:06,424|com.zaxxer.hikari.HikariDataSource|database - Started. I|2020-09-01 16:51:06,892|o.f.c.i.license.VersionPrinter|Flyway Community Edition 6.0.8 by Redgate I|2020-09-01 16:51:07,312|akka.event.slf4j.Slf4jLogger|Slf4jLogger started W|2020-09-01 16:51:07,398|akka.util.ManifestInfo|Detected possible incompatible versions on the classpath. Please note that a given Akka version MUST be the same across all modules of Akka that you are using, e.g. if you use [2.5.30] all other modules that are released together MUST be of the same version. Make sure you're using a compatible set of libraries. Possibly conflicting versions [2.5.30, 2.5.26] in libraries [akka-protobuf:2.5.30, akka-actor:2.5.30, akka-slf4j:2.5.26, akka-stream:2.5.30] I|2020-09-01 16:51:07,438|o.f.c.i.database.DatabaseFactory|Database: jdbc:mariadb://mysql:3306/campaigner (MariaDB 10.3) I|2020-09-01 16:51:07,484|akka.event.slf4j.Slf4jLogger|Slf4jLogger started W|2020-09-01 16:51:07,519|akka.util.ManifestInfo|Detected possible incompatible versions on the classpath. Please note that a given Akka version MUST be the same across all modules of Akka that you are using, e.g. if you use [2.5.30] all other modules that are released together MUST be of the same version. Make sure you're using a compatible set of libraries. Possibly conflicting versions [2.5.30, 2.5.26] in libraries [akka-protobuf:2.5.30, akka-actor:2.5.30, akka-slf4j:2.5.26, akka-stream:2.5.30] E|2020-09-01 16:51:07,535|c.a.libats.slick.db.RunMigrations$|16 migrations pending E|2020-09-01 16:51:07,536|c.advancedtelematic.campaigner.Boot$|Outdated migrations, terminating I|2020-09-01 16:51:07,718|c.advancedtelematic.campaigner.Boot$|Starting campaigner/ebc1f5771021ef40fff63f966178da34a5d7ecd9 on http://0.0.0.0:9001

I am attempting to delve into the campaigner source code but I cannot match the hub.docker.com image reference to a particular https://github.com/advancedtelematic/campaigner tag/commit id anymore. It's been a while since a campaigner update so should I assume the latest? Or has the source code moved?

Cheers, David

ddunlap2005 commented 4 years ago

I rebuilt the latest from GitHub.com and also received the same error just starting it up with 'sbt run':

~/$ mkdir campaigner_entrypoint.d
~/$ echo "
CREATE DATABASE campaigner;
GRANT ALL PRIVILEGES ON \`campaigner%\`.* TO 'campaigner'@'%';
FLUSH PRIVILEGES;
" > campaigner_entrypoint.d/db_user.sql
~/$ MYSQL_PORT=3306
~/$ docker run -d \
  --name campaigner-mariadb \
  -p $MYSQL_PORT:3306 \
  -v $(pwd)/campaigner_entrypoint.d:/docker-entrypoint-initdb.d \
  -e MYSQL_ROOT_PASSWORD=root \
  -e MYSQL_USER=campaigner \
  -e MYSQL_PASSWORD=campaigner \
  mariadb:10.2 \
  --character-set-server=utf8 --collation-server=utf8_unicode_ci \
  --max_connections=1000
Unable to find image 'mariadb:10.2' locally
10.2: Pulling from library/mariadb
f08d8e2a3ba1: Pull complete
3baa9cb2483b: Pull complete
94e5ff4c0b15: Pull complete
1860925334f9: Pull complete
76bdaccc96a9: Pull complete
4a33640afbdd: Pull complete
6059c2c04bf3: Pull complete
0ddc66150bff: Pull complete
92a72508f381: Pull complete
b4537efc7e12: Pull complete
1fa6e0b87d6d: Pull complete
c4f9e79a1317: Pull complete
9cb083a24d84: Pull complete
06dd4cd1848e: Pull complete
Digest: sha256:74ec811b4dd9580f6c6a110e46123e72b9d1eac930cff505e2d8e3e658edc804
Status: Downloaded newer image for mariadb:10.2
42b1a931fc110a665d2ee0bd5fcd88d18afdef6b2898343fcd7849787d507cf4
~/$ mysqladmin ping --protocol=TCP -h 127.0.0.1 -P 3306 -u root -proot
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
mysqld is alive
~/$ cd campaigner
~/$ sbt run
[info] Loading settings for project campaigner-build from plugins.sbt,scoverage.sbt ...
[info] Loading project definition from /Users/ddunlap/Work/projects/usa-abb272897_paria/components/campaigner/project
[info] Loading settings for project campaigner from build.sbt ...
[info] Set current project to campaigner (in build file:/Users/ddunlap/Work/projects/usa-abb272897_paria/components/campaigner/)
[warn] insecure HTTP request is deprecated 'http://nexus.advancedtelematic.com:8081/content/repositories/releases'; switch to HTTPS or opt-in as ("ATS Releases" at "http://nexus.advancedtelematic.com:8081/content/repositories/releases").withAllowInsecureProtocol(true)
[warn] insecure HTTP request is deprecated 'http://nexus.advancedtelematic.com:8081/content/repositories/releases'; switch to HTTPS or opt-in as ("ATS Releases" at "http://nexus.advancedtelematic.com:8081/content/repositories/releases").withAllowInsecureProtocol(true)
[warn] insecure HTTP request is deprecated 'http://nexus.advancedtelematic.com:8081/content/repositories/releases'; switch to HTTPS or opt-in as ("ATS Releases" at "http://nexus.advancedtelematic.com:8081/content/repositories/releases").withAllowInsecureProtocol(true)
[warn] insecure HTTP request is deprecated 'http://nexus.advancedtelematic.com:8081/content/repositories/snapshots'; switch to HTTPS or opt-in as ("ATS Snapshots" at "http://nexus.advancedtelematic.com:8081/content/repositories/snapshots").withAllowInsecureProtocol(true)
[warn] insecure HTTP request is deprecated 'http://nexus.advancedtelematic.com:8081/content/repositories/snapshots'; switch to HTTPS or opt-in as ("ATS Snapshots" at "http://nexus.advancedtelematic.com:8081/content/repositories/snapshots").withAllowInsecureProtocol(true)
[warn] insecure HTTP request is deprecated 'http://nexus.advancedtelematic.com:8081/content/repositories/snapshots'; switch to HTTPS or opt-in as ("ATS Snapshots" at "http://nexus.advancedtelematic.com:8081/content/repositories/snapshots").withAllowInsecureProtocol(true)
[info] Compiling 1 Scala source to /Users/ddunlap/Work/projects/usa-abb272897_paria/components/campaigner/target/scala-2.12/classes ...
[info] running com.advancedtelematic.campaigner.Boot
I|2020-09-01 19:08:00,653|akka.event.slf4j.Slf4jLogger|Slf4jLogger started
I|2020-09-01 19:08:00,847|com.zaxxer.hikari.HikariDataSource|database - Started.
I|2020-09-01 19:08:01,068|o.f.c.i.license.VersionPrinter|Flyway Community Edition 6.0.8 by Redgate
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.bouncycastle.jcajce.provider.drbg.DRBG (file:/var/folders/3j/_78k0z9524s8nbm2_sv7wct00000gn/T/sbt_b035bcb8/target/5ca78794/41152d12/bcprov-jdk15on-1.59.jar) to constructor sun.security.provider.Sun()
WARNING: Please consider reporting this to the maintainers of org.bouncycastle.jcajce.provider.drbg.DRBG
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
I|2020-09-01 19:08:01,191|akka.event.slf4j.Slf4jLogger|Slf4jLogger started
I|2020-09-01 19:08:01,236|o.f.c.i.database.DatabaseFactory|Database: jdbc:mariadb://localhost:3306/campaigner (MariaDB 10.2)
I|2020-09-01 19:08:01,287|akka.event.slf4j.Slf4jLogger|Slf4jLogger started
E|2020-09-01 19:08:01,335|c.a.libats.slick.db.RunMigrations$|18 migrations pending
E|2020-09-01 19:08:01,336|c.advancedtelematic.campaigner.Boot$|Outdated migrations, terminating
I|2020-09-01 19:08:01,408|c.advancedtelematic.campaigner.Boot$|Starting campaigner/0.1.0-SNAPSHOT on http://0.0.0.0:8084
[error] (run-main-0) java.lang.IllegalStateException: cannot create children while terminating or terminated
[error] java.lang.IllegalStateException: cannot create children while terminating or terminated
[error]     at akka.actor.dungeon.Children.makeChild(Children.scala:287)
[error]     at akka.actor.dungeon.Children.attachChild(Children.scala:51)
[error]     at akka.actor.dungeon.Children.attachChild$(Children.scala:50)
[error]     at akka.actor.ActorCell.attachChild(ActorCell.scala:447)
[error]     at akka.actor.ActorSystemImpl.systemActorOf(ActorSystem.scala:774)
[error]     at akka.http.scaladsl.HttpExt.<init>(Http.scala:388)
[error]     at akka.http.scaladsl.Http$.createExtension(Http.scala:1123)
[error]     at akka.http.scaladsl.Http$.createExtension(Http.scala:892)
[error]     at akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:1006)
[error]     at akka.actor.ExtensionId.apply(Extension.scala:79)
[error]     at akka.actor.ExtensionId.apply$(Extension.scala:78)
[error]     at akka.http.scaladsl.Http$.apply(Http.scala:1118)
[error]     at com.advancedtelematic.libats.http.ServiceHttpClientSupport.defaultHttpClient(ServiceHttpClient.scala:25)
[error]     at com.advancedtelematic.libats.http.ServiceHttpClientSupport.defaultHttpClient$(ServiceHttpClient.scala:24)
[error]     at com.advancedtelematic.campaigner.Boot$.defaultHttpClient(Boot.scala:41)
[error]     at com.advancedtelematic.campaigner.Boot$.delayedEndpoint$com$advancedtelematic$campaigner$Boot$1(Boot.scala:60)
[error]     at com.advancedtelematic.campaigner.Boot$delayedInit$body.apply(Boot.scala:41)
[error]     at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error]     at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error]     at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error]     at scala.App.$anonfun$main$1$adapted(App.scala:80)
[error]     at scala.collection.immutable.List.foreach(List.scala:392)
[error]     at scala.App.main(App.scala:80)
[error]     at scala.App.main$(App.scala:78)
[error]     at com.advancedtelematic.campaigner.Boot$.main(Boot.scala:41)
[error]     at com.advancedtelematic.campaigner.Boot.main(Boot.scala)
[error]     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error]     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error]     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error]     at java.base/java.lang.reflect.Method.invoke(Method.java:564)
[error] stack trace is suppressed; run last Compile / bgRun for the full output
ddunlap2005 commented 4 years ago

So the outdated migrations is caused by the campaigner daemon being a different image. As soon as I updated the two images to being the same, the problem has gone away.

The campaigner still does not fully work as I cannot create a campaign properly through the web pages (see Andy Doan's campaigner issue from 2 years ago) but, at least, it is no worse than before.

pattivacek commented 4 years ago

Was this fixed by https://github.com/advancedtelematic/ota-community-edition/pull/115?

ddunlap2005 commented 4 years ago

Yes, this issue is fixed by #115.