General discussion

[nickcoast]

a@a-b:~/code/scala.js/scala-webapp-template$ sbt dev-web
a@b:~/scala-webapp-template$ sbt dev-web
Unrecognized VM option 'CMSClassUnloadingEnabled'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
copying runtime jar...
mkdir: cannot create directory ‘’: No such file or directory
Unrecognized VM option 'CMSClassUnloadingEnabled'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Unrecognized VM option 'CMSClassUnloadingEnabled'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

I'm new to Scala and Java so I'm sure the answer should be obvious but I've been struggling with it so I thought I'd ask for some help.

I did some Googling and found that CMSClassUnloadingEnabled option was deprecated before Java 14. Is that true? What version of Java should I use?

https://stackoverflow.com/questions/70214788/unrecognized-vm-option-cmsclassunloadingenabled-after-switching-to-java-17

I tried this:

$ sdk use java 11.0.16-tem $ sbt dev-web

And got a about 8 warnings and some errors.

[info] welcome to sbt 1.6.2 (Eclipse Adoptium Java 11.0.16)
[info] loading settings for project global-plugins from plugins.sbt ...
[info] loading global plugins from /home/a/.sbt/1.0/plugins
[info] loading settings for project scala-webapp-template-build from plugins.sbt ...
[info] loading project definition from /home/a/code/scala.js/scala-webapp-template/project
[info] loading settings for project root from build.sbt ...
[info] BuildInfo settings:
[info] (apiUrl,None)
[info] BuildInfo settings:
[info] (apiUrl,None)
[info] set current project to root (in build file:/home/a/code/scala.js/scala-webapp-template/)
[warn] there are 8 keys that are not used by any other settings/tasks:
[warn]  
[warn] * admin / Compile / fastOptJS / webpackExtraArgs
[warn]   +- /home/a/code/scala.js/scala-webapp-template/build.sbt:199
[warn] * admin / Compile / fullOptJS / webpackExtraArgs
[warn]   +- /home/a/code/scala.js/scala-webapp-template/build.sbt:200
[warn] * apiJS / Compile / stMinimize
[warn]   +- /home/a/code/scala.js/scala-webapp-template/build.sbt:274
[warn] * commonJS / Compile / stMinimize
[warn]   +- /home/a/code/scala.js/scala-webapp-template/build.sbt:249
[warn] * ui / Compile / stMinimize
[warn]   +- /home/a/code/scala.js/scala-webapp-template/build.sbt:310
[warn] * web / Compile / fastOptJS / webpackExtraArgs
[warn]   +- /home/a/code/scala.js/scala-webapp-template/build.sbt:199
[warn] * web / Compile / fullOptJS / webpackExtraArgs
[warn]   +- /home/a/code/scala.js/scala-webapp-template/build.sbt:200
[warn] * web / Compile / stMinimize
[warn]   +- /home/a/code/scala.js/scala-webapp-template/build.sbt:388
[warn]  
[warn] note: a setting might still be used by a command; to exclude a key from this `lintUnused` check
[warn] either append it to `Global / excludeLintKeys` or call .withRank(KeyRanks.Invisible) on the key
[warn] sbt 0.13 shell syntax is deprecated; use slash syntax instead: web / Compile / fastOptJS / startWebpackDevServer
[info] Updating NPM dependencies
[info] Using lockfile /home/a/code/scala.js/scala-webapp-template/web/yarn.lock
[error] Usage: yarn [options]
[error] yarn: error: no such option: --non-interactive
[info] Updating NPM dependencies
[info] Using lockfile /home/a/code/scala.js/scala-webapp-template/lib/ui/yarn.lock
[error] Usage: yarn [options]
[error] yarn: error: no such option: --non-interactive
[error] java.lang.RuntimeException: Non-zero exit code: 2
[error]     at scala.sys.package$.error(package.scala:30)
[error]     at scalajsbundler.util.Commands$.$anonfun$run$9(Commands.scala:38)
[error]     at scala.util.Either.fold(Either.scala:192)
[error]     at scalajsbundler.util.Commands$.run(Commands.scala:38)
[error]     at scalajsbundler.ExternalCommand.run(ExternalCommand.scala:22)
[error]     at scalajsbundler.ExternalCommand$.$anonfun$install$1(ExternalCommand.scala:90)
[error]     at scalajsbundler.ExternalCommand$.syncYarnLockfile(ExternalCommand.scala:46)
[error]     at scalajsbundler.ExternalCommand$.install(ExternalCommand.scala:89)
[error]     at scalajsbundler.sbtplugin.NpmUpdateTasks$.$anonfun$npmInstallDependencies$1(NpmUpdateTasks.scala:59)
[error]     at sbt.util.FileFunction$.$anonfun$cached$1(FileFunction.scala:80)
[error]     at sbt.util.FileFunction$.$anonfun$cached$4(FileFunction.scala:153)
[error]     at sbt.util.Difference.apply(Tracked.scala:414)
[error]     at sbt.util.Difference.apply(Tracked.scala:394)
[error]     at sbt.util.FileFunction$.$anonfun$cached$3(FileFunction.scala:149)
[error]     at sbt.util.Difference.apply(Tracked.scala:414)
[error]     at sbt.util.Difference.apply(Tracked.scala:389)
[error]     at sbt.util.FileFunction$.$anonfun$cached$2(FileFunction.scala:148)
[error]     at scalajsbundler.sbtplugin.NpmUpdateTasks$.npmInstallDependencies(NpmUpdateTasks.scala:62)
[error]     at scalajsbundler.sbtplugin.ScalaJSBundlerPlugin$.$anonfun$perConfigSettings$7(ScalaJSBundlerPlugin.scala:650)
[error]     at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error]     at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error]     at sbt.std.Transform$$anon$4.work(Transform.scala:68)
[error]     at sbt.Execute.$anonfun$submit$2(Execute.scala:282)
[error]     at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23)
[error]     at sbt.Execute.work(Execute.scala:291)
[error]     at sbt.Execute.$anonfun$submit$1(Execute.scala:282)
[error]     at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error]     at sbt.CompletionService$$anon$2.call(CompletionService.scala:64)
[error]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error]     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[error]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[error]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[error]     at java.base/java.lang.Thread.run(Thread.java:829)
[error] java.lang.RuntimeException: Non-zero exit code: 2
[error]     at scala.sys.package$.error(package.scala:30)
[error]     at scalajsbundler.util.Commands$.$anonfun$run$9(Commands.scala:38)
[error]     at scala.util.Either.fold(Either.scala:192)
[error]     at scalajsbundler.util.Commands$.run(Commands.scala:38)
[error]     at scalajsbundler.ExternalCommand.run(ExternalCommand.scala:22)
[error]     at scalajsbundler.ExternalCommand$.$anonfun$install$1(ExternalCommand.scala:90)
[error]     at scalajsbundler.ExternalCommand$.syncYarnLockfile(ExternalCommand.scala:46)
[error]     at scalajsbundler.ExternalCommand$.install(ExternalCommand.scala:89)
[error]     at scalajsbundler.sbtplugin.NpmUpdateTasks$.$anonfun$npmInstallDependencies$1(NpmUpdateTasks.scala:59)
[error]     at sbt.util.FileFunction$.$anonfun$cached$1(FileFunction.scala:80)
[error]     at sbt.util.FileFunction$.$anonfun$cached$4(FileFunction.scala:153)
[error]     at sbt.util.Difference.apply(Tracked.scala:414)
[error]     at sbt.util.Difference.apply(Tracked.scala:394)
[error]     at sbt.util.FileFunction$.$anonfun$cached$3(FileFunction.scala:149)
[error]     at sbt.util.Difference.apply(Tracked.scala:414)
[error]     at sbt.util.Difference.apply(Tracked.scala:389)
[error]     at sbt.util.FileFunction$.$anonfun$cached$2(FileFunction.scala:148)
[error]     at scalajsbundler.sbtplugin.NpmUpdateTasks$.npmInstallDependencies(NpmUpdateTasks.scala:62)
[error]     at scalajsbundler.sbtplugin.ScalaJSBundlerPlugin$.$anonfun$perConfigSettings$7(ScalaJSBundlerPlugin.scala:650)
[error]     at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error]     at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error]     at sbt.std.Transform$$anon$4.work(Transform.scala:68)
[error]     at sbt.Execute.$anonfun$submit$2(Execute.scala:282)
[error]     at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23)
[error]     at sbt.Execute.work(Execute.scala:291)
[error]     at sbt.Execute.$anonfun$submit$1(Execute.scala:282)
[error]     at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error]     at sbt.CompletionService$$anon$2.call(CompletionService.scala:64)
[error]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error]     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[error]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[error]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[error]     at java.base/java.lang.Thread.run(Thread.java:829)
[error] (ui / Compile / npmInstallDependencies) Non-zero exit code: 2
[error] (web / Compile / npmInstallDependencies) Non-zero exit code: 2
[error] Total time: 1 s, completed Sep 27, 2022, 7:00:50 PM

[AlexITC]

I really haven't tried with jdk 17 but there seems to be an unknown jvm option involved.

The recommended way is to leverage https://sdkman.io/ and pick the jdk from https://github.com/wiringbits/scala-webapp-template/blob/master/.sdkmanrc (which seems out of date), java 11.0.16-tem should work.

sbt dev-web

It does print some warnings but the logs you shared show the problem:

[error] Usage: yarn [options]
[error] yarn: error: no such option: --non-interactive

I'd bet you are using an incompatible yarn version, I'd suggest to use https://github.com/nvm-sh/nvm to pick the supported node version (https://github.com/wiringbits/scala-webapp-template/blob/master/.nvmrc), then, install yarn (I have 1.22.11).

If you get more issues, feel free to comment, I'm in the process to update the docs.

[AlexITC]

@nickcoast please check the docs and let us know if you get into any other issues: https://github.com/wiringbits/scala-webapp-template/blob/master/docs/setup-dev-environment.md

[nickcoast]

EDIT: I see you added some docs since the time I started composing this reply. I will take a look. Thanks again.

---

Thank you so much! Your suggestions helped me get past those issues and get this working.

Just in case it might be useful for your documentation, I'll note a couple problems I worked through . Though they may be basic and obvious to others.

• yarn: I had to use global option to fix the yarn problem. npm install --global yarn.

• postgres: Pretty sure I ran this query, CREATE EXTENSION IF NOT EXISTS CITEXT, during set up, but got error on localhost:9000, Ran the query again and clicked the 'resolve' - no change. No user* tables created. Finally, deleted play_evolutions table and restarted the Scala server, and all tables got created.

• Creating a new account worked (db records were added), but I received no confirmation on the frontend (still shows filled in form - see screenshot below). I assume that's because of failure to communicate with the AWS service that would exist in production. I ran the query UPDATE users SET verified_on = '2022-09-29'; and now login works.

Here's the new account error: software.amazon.awssdk.services.ses.model.SesException: The security token included in the request is invalid. (Service: Ses, Status Code: 403, Request ID: 0974de2c-0ab8-463b-a806-f34fbd2b4f93)

Here's the form after submission, with the AWS error:

Also, the Ubuntu Setup section was helpful but, perhaps naively, I assumed all other dependencies would be handled in the build process. As you pointed out, I needed the right node version to get yarn to work. But again, maybe that's just me not having enough experience with these technologies and maybe your target audience is more savvy.

Thanks again!

[AlexITC]

Thanks for getting back and sharing the useful insights, I'll take your suggestions and update the docs again.

yarn: I had to use global option to fix the yarn problem. npm install --global yarn.

I included this a couple of hours ago.

postgres: Pretty sure I ran this query, CREATE EXTENSION IF NOT EXISTS CITEXT, during set up, but got error on localhost:9000, Ran the query again and clicked the 'resolve' - no change. No user* tables created. Finally, deleted play_evolutions table and restarted the Scala server, and all tables got created.

People commonly fails to connect to the database before creating the extension, there is a chance you experienced the same.

Creating a new account worked (db records were added), but I received no confirmation on the frontend (still shows filled in form - see screenshot below). I assume that's because of failure to communicate with the AWS service that would exist in production. I ran the query UPDATE users SET verified_on = '2022-09-29'; and now login works.

Oh, good catch! most emails are sent through a background job but it seems this is not the case when you are creating an account.

Also, the Ubuntu Setup section was helpful but, perhaps naively, I assumed all other dependencies would be handled in the build process. As you pointed out, I needed the right node version to get yarn to work. But again, maybe that's just me not having enough experience with these technologies and maybe your target audience is more savvy.

Which section are you referring to?

all other dependencies would be handled in the build process

There are some native dependencies required, I tried making that clear on the new docs, yarn was particularly hidden.

maybe your target audience is more savvy.

I'm actually trying to make the process as simple as possible, we have had this project mostly for internal usage, hence, the support has been handled through our internal chats.

[AlexITC]

Oh, good catch! most emails are sent through a background job but it seems this is not the case when you are creating an account.

Now I finally remember why it was done this way, most emails are sent through the background job unless they include sensitive information, for example, the token used to verify an email or to recover a password, I created https://github.com/wiringbits/scala-webapp-template/pull/285 to simplify this process, now, there are no more errors when an account is created but AWS is not used.

[nickcoast]

That makes sense, works great.

Your new docs for Architecture are extremely helpful. I've added some models but am was getting a 500 Internal Server error when running server/run and I'm wondering if you can help me find a log that would show me the error text. localhost:9000 was showing some useful information, but after I added some more code I'm just getting a 500 Internal Server Error.

Nothing is logged in ~.logs/application.log, nor in the Postgresql log, for this error.

The error went away after I added 2.sql evolution, but I would still like to know where I can find error information. Thank you!

https://github.com/nickcoast/fbn-scala

Also, by the way, in case you will be adding any IDE docs, I will share a couple things I found helpful in IntelliJ. I created 1 run configuration each for server/run, dev-web, and dev-admin and combined them under a "Compound" configuration that runs all 3 at once. Before that, I was running one in IntelliJ's sbt console, and the other two in IntelliJ terminal windows, which didn't seem like the right way to do it.

Also, I added *.iml to .gitignore to ignore IntelliJ's files, and added .envrc to ignore the direnv file from your docs.

[AlexITC]

Your new docs for Architecture are extremely helpful

Glad you liked that, there is still work to be done to adapt the code to make such separations clear.

I've added some models but am was getting a 500 Internal Server error when running server/run and I'm wondering if you can help me find a log that would show me the error text. localhost:9000 was showing some useful information, but after I added some more code I'm just getting a 500 Internal Server Error.

Logs are printed to stdout, could it be anything related to your compound IDE settings? I tend to run the app by launching 3 terminals, the one where I execute sbt server/run prints the logs, in any case, file logs should be found at ~/logs/application.log (see logback.xml)

The error went away after I added 2.sql evolution, but I would still like to know where I can find error information. Thank you!

This is actually weird, I'd bet there is something else going on but it is hard to tell without logs.

I see you got into some trouble with sql parsers, feel free to ask questions about those.

[nickcoast]

Thank you! Actually it turns my 2.sql evolution did not cause the change in the failure mode. It was a new syntax syntax error from when I git merged, and I guess the web interface for Scala Play stopped at the syntax error. After I fixed it, the 500 error came back, but only if I added my new implicit val Case Class parsers to daos/package.scala. (If commented them out, a different Scala Play error page came up) I am guessing that it's because the parsers rely on bytecode and my IDE did not recompile it yet. Does that sound right?

I don't know if my parsers, etc, are really working until I try some read/write but at least there are no compile or runtime errors yet.

So, the server seems to be working and my next steps will be creating new widgets and/or pages.

By the way, the evolutions error happened because I already created my tables, so evolutions could not CREATE TABLE. I clicked "resolved" on the web interface but nothing changed. I had to DROP my tables, run DELETE FROM play_evolutions WHERE id = 2, and restart the server so that 2.sql could be reapplied and marked successful in the play_evolutions table. Is that normal?

Thanks so much for your help!

[AlexITC]

Actually it turns my 2.sql evolution did not cause the change in the failure mode

Which makes sense, in any case, I'd expect it to cause a failure.

After I fixed it, the 500 error came back, but only if I added my new implicit val Case Class parsers to daos/package.scala. (If commented them out, a different Scala Play error page came up) I am guessing that it's because the parsers rely on bytecode and my IDE did not recompile it yet. Does that sound right?

It is hard to know without logs.

I don't know if my parsers, etc, are really working until I try some read/write but at least there are no compile or runtime errors yet.

Usually, you write a simple test to verify that (check tests for repositories).

By the way, the evolutions error happened because I already created my tables, so evolutions could not CREATE TABLE. I clicked "resolved" on the web interface but nothing changed. I had to DROP my tables, run DELETE FROM play_evolutions WHERE id = 2, and restart the server so that 2.sql could be reapplied and marked successful in the play_evolutions table. Is that normal?

Usually, the UI should work but I tend to update play_evolutions table directly when such details occur, same way, repository tests help you to make sure evolutions can be applied correctly.

[nickcoast]

Hi Alex,

I added some routes, a controller, actions, repos. I'm getting an error related to java.time.Instant but the error output doesn't seem to point to anywhere in my code. Do you have any tips on how I could debug this?

Thank you

Output from sbt server/test

nick@asdf:~/code/scala.js/fbn$ sbt server/test
[info] welcome to sbt 1.6.2 (Eclipse Adoptium Java 11.0.16)
[info] loading settings for project global-plugins from plugins.sbt ...
[info] loading global plugins from /home/nick/.sbt/1.0/plugins
[info] loading settings for project fbn-build from plugins.sbt ...
[info] loading project definition from /home/nick/code/scala.js/fbn/project
[info] loading settings for project root from build.sbt ...
[info] BuildInfo settings:
[info] (apiUrl,None)
[info] BuildInfo settings:
[info] (apiUrl,None)
[info] set current project to root (in build file:/home/nick/code/scala.js/fbn/)
[warn] there are 8 keys that are not used by any other settings/tasks:
[warn]  
[warn] * admin / Compile / fastOptJS / webpackExtraArgs
[warn]   +- /home/nick/code/scala.js/fbn/build.sbt:199
[warn] * admin / Compile / fullOptJS / webpackExtraArgs
[warn]   +- /home/nick/code/scala.js/fbn/build.sbt:200
[warn] * apiJS / Compile / stMinimize
[warn]   +- /home/nick/code/scala.js/fbn/build.sbt:274
[warn] * commonJS / Compile / stMinimize
[warn]   +- /home/nick/code/scala.js/fbn/build.sbt:249
[warn] * ui / Compile / stMinimize
[warn]   +- /home/nick/code/scala.js/fbn/build.sbt:310
[warn] * web / Compile / fastOptJS / webpackExtraArgs
[warn]   +- /home/nick/code/scala.js/fbn/build.sbt:199
[warn] * web / Compile / fullOptJS / webpackExtraArgs
[warn]   +- /home/nick/code/scala.js/fbn/build.sbt:200
[warn] * web / Compile / stMinimize
[warn]   +- /home/nick/code/scala.js/fbn/build.sbt:388
[warn]  
[warn] note: a setting might still be used by a command; to exclude a key from this `lintUnused` check
[warn] either append it to `Global / excludeLintKeys` or call .withRank(KeyRanks.Invisible) on the key
[info] compiling 3 Scala sources to /home/nick/code/scala.js/fbn/server/target/scala-2.13/classes ...
java.lang.ClassNotFoundException: Instant
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
  | => sat java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    at io.swagger.util.ReflectionUtils.loadClassByName(ReflectionUtils.java:53)
    at io.swagger.jackson.ModelResolver.getInnerType(ModelResolver.java:932)
    at io.swagger.jackson.ModelResolver.resolve(ModelResolver.java:468)
    at io.swagger.jackson.ModelResolver.resolve(ModelResolver.java:203)
    at io.swagger.scala.converter.SwaggerScalaModelConverter.resolve(SwaggerScalaModelConverter.scala:90)
    at com.github.dwickern.swagger.ValidationModelConverter.resolve(ValidationModelConverter.scala:63)
    at io.swagger.converter.ModelConverterContextImpl.resolve(ModelConverterContextImpl.java:103)
    at io.swagger.jackson.ModelResolver.resolve(ModelResolver.java:257)
    at io.swagger.jackson.ModelResolver.resolve(ModelResolver.java:203)
    at io.swagger.scala.converter.SwaggerScalaModelConverter.resolve(SwaggerScalaModelConverter.scala:90)
    at com.github.dwickern.swagger.ValidationModelConverter.resolve(ValidationModelConverter.scala:63)
    at io.swagger.converter.ModelConverterContextImpl.resolve(ModelConverterContextImpl.java:103)
    at io.swagger.jackson.ModelResolver.resolveProperty(ModelResolver.java:175)
    at io.swagger.jackson.ModelResolver.resolveProperty(ModelResolver.java:126)
    at io.swagger.scala.converter.SwaggerScalaModelConverter.resolveProperty(SwaggerScalaModelConverter.scala:70)
    at com.github.dwickern.swagger.ValidationModelConverter.resolveProperty(ValidationModelConverter.scala:59)
    at io.swagger.converter.ModelConverterContextImpl.resolveProperty(ModelConverterContextImpl.java:83)
    at io.swagger.converter.ModelConverters.readAsProperty(ModelConverters.java:63)
    at io.swagger.converter.ModelConverters.readAsProperty(ModelConverters.java:57)
    at io.swagger.util.ParameterProcessor.applyAnnotations(ParameterProcessor.java:263)
    at play.modules.swagger.PlayReader.readImplicitParam(PlayReader.java:441)
    at play.modules.swagger.PlayReader.readImplicitParameters(PlayReader.java:408)
    at play.modules.swagger.PlayReader.read(PlayReader.java:208)
    at play.modules.swagger.PlayReader.read(PlayReader.java:76)
    at play.modules.swagger.PlayReader.read(PlayReader.java:70)
    at play.modules.swagger.ApiListingCache.$anonfun$listing$1(ApiListingCache.scala:17)
    at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:454)
    at play.modules.swagger.ApiListingCache.listing(ApiListingCache.scala:13)
    at com.github.dwickern.swagger.SwaggerRunner$.run(SwaggerRunner.scala:31)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.github.dwickern.sbt.SwaggerPlayPlugin$.$anonfun$projectSettings$12(SwaggerPlayPlugin.scala:87)
    at com.github.dwickern.sbt.SwaggerPlayPlugin$$anon$2.run(SwaggerPlayPlugin.scala:172)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at com.github.dwickern.sbt.SwaggerPlayPlugin$.withContextClassLoader(SwaggerPlayPlugin.scala:175)
    at com.github.dwickern.sbt.SwaggerPlayPlugin$.$anonfun$projectSettings$11(SwaggerPlayPlugin.scala:79)
    at scala.Function1.$anonfun$compose$1(Function1.scala:49)
    at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
    at sbt.std.Transform$$anon$4.work(Transform.scala:68)
    at sbt.Execute.$anonfun$submit$2(Execute.scala:282)
    at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23)
    at sbt.Execute.work(Execute.scala:291)
    at sbt.Execute.$anonfun$submit$1(Execute.scala:282)
    at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:64)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
[error] java.lang.IllegalArgumentException: argument "t" is null
[error]     at com.fasterxml.jackson.databind.ObjectMapper._assertNotNull(ObjectMapper.java:4737)
[error]     at com.fasterxml.jackson.databind.ObjectMapper.constructType(ObjectMapper.java:2116)
[error]     at io.swagger.scala.converter.SwaggerScalaModelConverter.resolveProperty(SwaggerScalaModelConverter.scala:24)
[error]     at com.github.dwickern.swagger.ValidationModelConverter.resolveProperty(ValidationModelConverter.scala:59)
[error]     at io.swagger.converter.ModelConverterContextImpl.resolveProperty(ModelConverterContextImpl.java:83)
[error]     at io.swagger.jackson.ModelResolver.resolve(ModelResolver.java:469)
[error]     at io.swagger.jackson.ModelResolver.resolve(ModelResolver.java:203)
[error]     at io.swagger.scala.converter.SwaggerScalaModelConverter.resolve(SwaggerScalaModelConverter.scala:90)
[error]     at com.github.dwickern.swagger.ValidationModelConverter.resolve(ValidationModelConverter.scala:63)
[error]     at io.swagger.converter.ModelConverterContextImpl.resolve(ModelConverterContextImpl.java:103)
[error]     at io.swagger.jackson.ModelResolver.resolve(ModelResolver.java:257)
[error]     at io.swagger.jackson.ModelResolver.resolve(ModelResolver.java:203)
[error]     at io.swagger.scala.converter.SwaggerScalaModelConverter.resolve(SwaggerScalaModelConverter.scala:90)
[error]     at com.github.dwickern.swagger.ValidationModelConverter.resolve(ValidationModelConverter.scala:63)
[error]     at io.swagger.converter.ModelConverterContextImpl.resolve(ModelConverterContextImpl.java:103)
[error]     at io.swagger.jackson.ModelResolver.resolveProperty(ModelResolver.java:175)
[error]     at io.swagger.jackson.ModelResolver.resolveProperty(ModelResolver.java:126)
[error]     at io.swagger.scala.converter.SwaggerScalaModelConverter.resolveProperty(SwaggerScalaModelConverter.scala:70)
[error]     at com.github.dwickern.swagger.ValidationModelConverter.resolveProperty(ValidationModelConverter.scala:59)
[error]     at io.swagger.converter.ModelConverterContextImpl.resolveProperty(ModelConverterContextImpl.java:83)
[error]     at io.swagger.converter.ModelConverters.readAsProperty(ModelConverters.java:63)
[error]     at io.swagger.converter.ModelConverters.readAsProperty(ModelConverters.java:57)
[error]     at io.swagger.util.ParameterProcessor.applyAnnotations(ParameterProcessor.java:263)
[error]     at play.modules.swagger.PlayReader.readImplicitParam(PlayReader.java:441)
[error]     at play.modules.swagger.PlayReader.readImplicitParameters(PlayReader.java:408)
[error]     at play.modules.swagger.PlayReader.read(PlayReader.java:208)
[error]     at play.modules.swagger.PlayReader.read(PlayReader.java:76)
[error]     at play.modules.swagger.PlayReader.read(PlayReader.java:70)
[error]     at play.modules.swagger.ApiListingCache.$anonfun$listing$1(ApiListingCache.scala:17)
[error]     at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:454)
[error]     at play.modules.swagger.ApiListingCache.listing(ApiListingCache.scala:13)
[error]     at com.github.dwickern.swagger.SwaggerRunner$.run(SwaggerRunner.scala:31)
[error]     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error]     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error]     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error]     at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[error]     at com.github.dwickern.sbt.SwaggerPlayPlugin$.$anonfun$projectSettings$12(SwaggerPlayPlugin.scala:87)
[error]     at com.github.dwickern.sbt.SwaggerPlayPlugin$$anon$2.run(SwaggerPlayPlugin.scala:172)
[error]     at java.base/java.security.AccessController.doPrivileged(Native Method)
[error]     at com.github.dwickern.sbt.SwaggerPlayPlugin$.withContextClassLoader(SwaggerPlayPlugin.scala:175)
[error]     at com.github.dwickern.sbt.SwaggerPlayPlugin$.$anonfun$projectSettings$11(SwaggerPlayPlugin.scala:79)
[error]     at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error]     at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error]     at sbt.std.Transform$$anon$4.work(Transform.scala:68)
[error]     at sbt.Execute.$anonfun$submit$2(Execute.scala:282)
[error]     at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23)
[error]     at sbt.Execute.work(Execute.scala:291)
[error]     at sbt.Execute.$anonfun$submit$1(Execute.scala:282)
[error]     at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error]     at sbt.CompletionService$$anon$2.call(CompletionService.scala:64)
[error]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error]     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[error]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[error]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[error]     at java.base/java.lang.Thread.run(Thread.java:829)
[error] (server / swaggerPlayJson) java.lang.IllegalArgumentException: argument "t" is null
[error] Total time: 9 s, completed Oct 8, 2022, 12:51:06 PM
nick@asdf:~/code/scala.js/fbn$ 

[AlexITC]

Oh, I have saw similar errors, please remove any swagger annotations you changed/added to your controller, there are tricky cases that I must document.

[nickcoast]

I've made a lot of progress since removing those annotations. They were indeed causing that issue.

Could you please give me a hint about creating a modal dialog in scala.js? Do I need to add a new library to build.sbt?

Thank you much

[AlexITC]

The project already has everything from https://v3.mui.com/ available, in particular, check dialogs, I have another project where such a component is used.

Good luck!

[nickcoast]

Thanks! One more quick question if you have the time.

Should I be using UUID primary keys for all DB tables? Seems the Admin interface only allows viewing individual records for tables with UUID primary keys.

[AlexITC]

Should I be using UUID primary keys for all DB tables? Seems the Admin interface only allows viewing individual records for tables with UUID primary keys.

It's been a while since I checked the react-admin backend we built but I wouldn't be surprised if that's the case, most of the work we are doing on top of this template defaults to UUID primary keys.

I'm documenting the project slowly, some day, I'll get to the admin integration, in the mean time, feel free to raise any other questions.

[nickcoast]

Thank! For deployment, there's lots of sensitive data that needs to go in /infra/. For a project based on your repo (like mine is), would you add the whole directory to .gitignore? Or remove it from the repo and perhaps commit it to a private repo?

[AlexITC]

I commonly use any of these approaches:

1. Add prod details to gitignore.
2. Use ansible-vault to keep those encrypted.

[nickcoast]

Great, thanks, I will try the ansible-vault. Seems like a good solution.

[nickcoast]

EDIT:

I had some question but made some progress. Deleting for now. Made some progress in debugging. Issues with environment variables and 'host not allowed' warnings.

[AlexITC]

I commented out the frontend value and then ansible deployed my server.yml to the backend value.

This seems weird.

I'm getting this error when I try to start the wiringbits server on either of my EC2 instances:

Check logs (home/play/logs/application.log), most likely reason is that server can't connect to the database.

EDIT: seems to be postgres authentication problem. I thought I created the user and db on both of my test servers but now user is not found. I'll try to fix.

Indeed.

in infra/config/server/dev.env.j2 I see some variables that are coming from the ini file. Can I also put the POSTGRES_PASSWORD and other values as variables in dev.env.js and declare them in the ini file? I've tried doing that but I'm not sure it's working.

While it is doable, I wouldn't recommend you to do that, backend settings can be encrypted with ansible-vault while hosts file is usully tracked by git

If you check the backend settings, there is a line PLAY_SESSION_DOMAIN="{{ web_app_domain }}" which is set at the hosts file web_app_domain="template-demo.wiringbits.net".

At last, web_app_site.j2 needs to be updated to reflect where your backend instances are located.

[nickcoast]

Could there be some problem loading the environment variables with the wiringbits-server service?

I tried running the wiringbits-server executable in /home/play/app/wiringbits-server-0.1.0-SNAPSHOT/bin and it gave postgres authentication error. Then I loaded all the environment variables from /home/play/app/.env ran wiringbits-server without error, and was able to get a response from port :9000

{"error":{"requestId":3,"message":"Host not allowed: xx.xxx.xx.xx:9000"}} (server is responding!)

I can also log in to Posgresql from cli with the user name/password from .env, so they are correct.

ubuntu@asdf:/etc/systemd/system$ psql -U db_user -h 127.0.0.1 server_db
Password for user db_user: 
psql (14.5 (Ubuntu 14.5-0ubuntu0.22.04.1))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.

/etc/systemd/system/wiringbits-server.service has the correct .env file:

[Unit]
Description=wiringbits-server

[Service]
Type=simple
WorkingDirectory=/home/play/app/wiringbits-server-0.1.0-SNAPSHOT.zip
StandardOutput=tty
StandardError=tty
EnvironmentFile=/home/play/app/.env
LimitNOFILE=65535
User=play
ExecStart=/home/play/app/wiringbits-server-0.1.0-SNAPSHOT.zip/bin/wiringbits-server -Dpidfile.path=/dev/null
Restart=on-failure

[Install]
WantedBy=multi-user.target

[AlexITC]

That would be weird but I have experienced something similar in different ubuntu versions, when the app starts, it prints logs with the loaded settings, can you please check those and verify if the variables are being picked up?

[nickcoast]

It turned out I had a typo in my app_directory_name in my hosts.ini file. I found using journalctl -u wiringbits-server that it couldn't find the binary! So, that explains why there were no logs created for the app, since it was never able to be run that way. Whoops!

In your demo-hosts.ini file you have this (with same IP for both servers).

[backend]
test-server ansible_host=64.227.100.33
[backend:vars]
# this is where the environment variables required by the app are defined
# it could be kept encrypted by using ansible-vault
app_env_config_source=config/server/demo.env.j2
.
.
.
[frontend]
test-server ansible_host=64.227.100.33
[frontend:vars]
# this is necessary to get SSL certificates, it is the email for receiving notifications from letsencrypt
letsencrypt_notifications_email=certbot@wiringbits.net
.
.
.

It seems that test-server causes a conflict. The second test-server overwrites the first, which only matters if it's 2 different machines. So when I run my hosts.ini with two different machines:

ansible-playbook -i hosts.ini server.yml web.yml admin.yml

...everything gets deployed to the second test-server.

I changed the name under [backend] to back-server and now ansible has deployed the server app to the right EC2 instance, and the web and admin apps to the other EC2 instance as I intended.

Is it not supposed to work that way? Ansible docs seem to suggest that ansible_* variables are global. Perhaps that's why?

I did some experiments. Using the same name ("test-server") causes the frontend and backend to all get merged under test-server.

Here I am running this command with the ansible_host under [frontend] having IP ending in 66, and the [backend] ending in 33 is not there!:

nick@nick:~/code/scala.js/fbn/infra$ ansible-inventory -i demo-hosts.ini  server.yml  --list
{
    "_meta": {
        "hostvars": {
            "test-server": {
                "admin_api_url": "https://template-demo-admin.wiringbits.net/api",
                "admin_app_domain": "template-demo-admin.wiringbits.net",
                "ansible_host": "64.227.100.66",
                "ansible_ssh_extra_args": "-o StrictHostKeyChecking=no",
                "ansible_user": "ubuntu",
                "app_directory_name": "wiringbits-server-0.1.0-SNAPSHOT",
                "app_env_config_source": "config/server/demo.env.j2",
                "app_source_name": "wiringbits-server-0.1.0-SNAPSHOT.zip",
                "app_startup_script": "wiringbits-server",
                "app_systemd_service_name": "wiringbits-server",
                "app_systemd_service_source": "config/server/server.service.j2",
                "letsencrypt_notifications_email": "certbot@wiringbits.net",
                "nginx_admin_password_file": "config/nginx/admin-app-htpasswd",
                "web_api_url": "https://template-demo.wiringbits.net/api",
                "web_app_domain": "template-demo.wiringbits.net"
            }
        }
    },
    "all": {
        "children": [
            "ungrouped",
            "webapp"
        ]
    },
    "backend": {
        "hosts": [
            "test-server"
        ]
    },
    "frontend": {
        "hosts": [
            "test-server"
        ]
    },
    "webapp": {
        "children": [
            "backend",
            "frontend"
        ]
    }
}

And now again, after changing test-server to back-server under [backend], now both ips appear under their own sections:

nick@nick:~/code/scala.js/fbn/infra$ ansible-inventory -i demo-hosts.ini  server.yml  --list
{
    "_meta": {
        "hostvars": {
            "back-server": {
                "admin_app_domain": "template-demo-admin.wiringbits.net",
                "ansible_host": "64.227.100.33",
                "ansible_ssh_extra_args": "-o StrictHostKeyChecking=no",
                "ansible_user": "ubuntu",
                "app_directory_name": "wiringbits-server-0.1.0-SNAPSHOT",
                "app_env_config_source": "config/server/demo.env.j2",
                "app_source_name": "wiringbits-server-0.1.0-SNAPSHOT.zip",
                "app_startup_script": "wiringbits-server",
                "app_systemd_service_name": "wiringbits-server",
                "app_systemd_service_source": "config/server/server.service.j2",
                "web_app_domain": "template-demo.wiringbits.net"
            },
            "test-server": {
                "admin_api_url": "https://template-demo-admin.wiringbits.net/api",
                "admin_app_domain": "template-demo-admin.wiringbits.net",
                "ansible_host": "64.227.100.66",
                "ansible_ssh_extra_args": "-o StrictHostKeyChecking=no",
                "ansible_user": "ubuntu",
                "letsencrypt_notifications_email": "certbot@wiringbits.net",
                "nginx_admin_password_file": "config/nginx/admin-app-htpasswd",
                "web_api_url": "https://template-demo.wiringbits.net/api",
                "web_app_domain": "template-demo.wiringbits.net"
            }
        }
    },
    "all": {
        "children": [
            "ungrouped",
            "webapp"
        ]
    },
    "backend": {
        "hosts": [
            "back-server"
        ]
    },
    "frontend": {
        "hosts": [
            "test-server"
        ]
    },
    "webapp": {
        "children": [
            "backend",
            "frontend"
        ]
    }
}

Maybe this is all due to some other mistake I've made but changing one test-server to a different name seems to have fixed my deployment issue.

[AlexITC]

It seems that test-server causes a conflict. The second test-server overwrites the first, which only matters if it's 2 different machines. So when I run my hosts.ini with two different machines:

I'd say that you basically found a bug in the scripts, thanks for sharing your details! I checked another project that uses similar ansible scripts but deploying to many instances, and, each server has a different variable name.

[AlexITC]

Given that this has become a general discussion, I'm reopening and pinning the ticket.

Previously, I mentioned that current swagger integration is tricky, I added some docs just in case you are interested in this.

If you have questions about any other topic, please let me know to prioritize documenting those topics.

[nickcoast]

In DatabaseTablesDAO.scala in package package net.wiringbits.webapp.utils.admin.repositories.daos, def find requires UUID primary key. Is there a way I can change it to also accept integer primary keys?

I would tweak it myself but I'm not sure how to deal with compiled sources (I'm new to Java).

  def find(tableName: String, primaryKeyField: String, primaryKeyValue: String)(implicit
      conn: Connection
  ): Option[TableRow] = {
    val sql = s"""
    SELECT *
      FROM $tableName
    WHERE $primaryKeyField = ?
    """
    val preparedStatement = conn.prepareStatement(sql)

    // TODO: UUID from String can fail if the ID field isn't an UUID
    preparedStatement.setObject(1, UUID.fromString(primaryKeyValue))
    val resultSet = preparedStatement.executeQuery()
    Try {
      resultSet.next()
      val numberOfColumns = resultSet.getMetaData.getColumnCount
      val row = for {
        columnNumber <- 1 to numberOfColumns
        cellData = resultSet.getString(columnNumber)
      } yield Cell(Option(cellData).getOrElse(""))
      TableRow(row.toList)
    }.toOption
  }

[AlexITC]

We believe that there shouldn't be so many changes involved, still, the sources are not in this repository, they are at https://github.com/wiringbits/wiringbits-webapp-utils

A simple solution would be to use the module settings to specify the type for the primary key field, then, we can update queries accordingly.

[AlexITC]

@nickcoast I have created https://github.com/wiringbits/wiringbits-webapp-utils/issues/138 to track this improvement.

[nickcoast]

Deployment question.

I changed my subdomain in my hosts.ini for the web app and redeployed now I'm getting 502 Bad Gateway response for requests to the web and admin api urls. And then when I changed back to the previous subdomain, still happening.

So I can load the homepage of the web app, but api requests fail. Can't sign in anymore. And admin is blank since rendering depends on the api response.

Do you think this is an nginx config issue?

[AlexITC]

Did you ran the nginx_site_web.yml and nginx_site_admin.yml playbooks? because that can be the problem. These scripts need to be executed after changing domains.

[nickcoast]

Yeah I ran all the nginx and other ymls:

nginx.yml nginx_site_admin.yml nginx_site_web.yml

Then

server.yml web.yml admin.yml

Repeated several times.

Perhaps I should delete all nginx files on the server, uninstall nginx. Then ansible will reinstall it and set it all up from scratch, right?

Also, no changes in infra have any bearing on the compiled code, right? Nothing from the hosts.ini, etc, inside infra gets pulled into the compilation process, right?

[AlexITC]

Yeah I ran all the nginx and other ymls:

That's weird, I remember changing domain names previously without issues. The steps you performed seem correct, I'd double check that hosts file doesn't have any details about the old domain.

Perhaps I should delete all nginx files on the server, uninstall nginx. Then ansible will reinstall it and set it all up from scratch, right?

The related files are at /etc/nginx/sites-enabled, removing old files from there must be enough, still, those old files must point to the old domain, hence, nginx shouldn't be using them.

You are right that if you uninstall nginx, the ansible playbooks would get it installed again.

Also, no changes in infra have any bearing on the compiled code, right? Nothing from the hosts.ini, etc, inside infra gets pulled into the compilation process, right?

Actually, there is an entry propagated from hosts file to the web project at compile time (see https://github.com/wiringbits/scala-webapp-template/blob/master/infra/web.yml), web_api_url is a variable defined there which is used to specify where to connect to the backend api:

    - name: Build the application
      shell: ./scripts/build-web.sh {{ web_api_url }}
      delegate_to: 127.0.0.1

Still, this value is logged when you load web, hence, opening the browser console will display whether the right value is being used or not.

I'm getting 502 Bad Gateway response for requests to the web and admin api urls

This occurs when nginx can't connect to the backend api, I remember that you were using different servers, please verify that:

1. Your backend service is actually running and accepting external traffic.
2. Nginx can talk to such service, use curl -v [backend-server] on the server running nginx to verify.
3. Check /var/log/nginx/error.log for hints about what's going on.

[nickcoast]

The related files are at /etc/nginx/sites-enabled, removing old files from there must be enough, still, those old files must point to the old domain, hence, nginx shouldn't be using them.

Yes, I looked there and removed the old files just in case but it had no effect.

Still, this value is logged when you load web, hence, opening the browser console will display whether the right value is being used or not.

Okay, so on my development machine, I don't need to run sbt compile before running ansible-playbook if I have, for example, uncommitted changes or have not run sbt compile since my last changes to the application code?

This occurs when nginx can't connect to the backend api, I remember that you were using different servers, please verify that:

I changed it to single-server deployment a couple weeks ago. Frontend and admin were both working fine yesterday. Deployed with my changes to the webapp utils and loaded data with integer primary keys and could view tables and records in admin, and frontend responses were working.

I ran sudo apt purge nginx nginx-common nginx-core on the deployment machine, then ran the ansible-playbook for nginx.yml and it successfully reinstalled. (sudo apt remove nginx resulted in ansible failure - I guess it left the server in a weird state) Then I ran the playbooks for the apps. Then for the nginx sites. All successful

1. wiringbits-server.service? It's running. Doesn't need to accept external traffic if on the same server, right?

ubuntu@ip-172-30-0-242:~$ systemctl status wiringbits-server.service
● wiringbits-server.service - wiringbits-server
     Loaded: loaded (/etc/systemd/system/wiringbits-server.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-11-17 03:28:08 UTC; 1min 2s ago
   Main PID: 110896 (java)
      Tasks: 17 (limit: 1143)
     Memory: 119.3M
        CPU: 1min 1.615s
     CGroup: /system.slice/wiringbits-server.service
             └─110896 java -Duser.dir=/home/play/app/wiringbits-server-0.1.0-SNAPSHOT -Dpidfile.path=/dev/null -cp /home/play/app/wiringbits-server-0.1.0-SNA>

Nov 17 03:28:08 ip-172-30-0-242 systemd[1]: wiringbits-server.service: Scheduled restart job, restart counter is at 13.
Nov 17 03:28:08 ip-172-30-0-242 systemd[1]: Stopped wiringbits-server.
Nov 17 03:28:08 ip-172-30-0-242 systemd[1]: wiringbits-server.service: Consumed 1min 32.945s CPU time.
Nov 17 03:28:08 ip-172-30-0-242 systemd[1]: Started wiringbits-server.

2. Like this?

ubuntu@ip-172-30-0-242:~$ curl -v 172.30.0.242
*   Trying 172.30.0.242:80...
* Connected to 172.30.0.242 (172.30.0.242) port 80 (#0)
> GET / HTTP/1.1
> Host: 172.30.0.242
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Server: nginx/1.18.0 (Ubuntu)
< Date: Thu, 17 Nov 2022 03:41:04 GMT
< Content-Type: text/html
< Content-Length: 162
< Connection: keep-alive
< 
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
* Connection #0 to host 172.30.0.242 left intact

3. Nginx error log has 111 unknown error

2022/11/17 03:13:27 [error] 109938#109938: *4 connect() failed (111: Unknown error) while connecting to upstream, client: xxx.xxx.xx.xx, server: web.futurebabynames.com, request: "GET /api/auth/me HTTP/1.1", upstream: "http://127.0.0.1:9000/auth/me", host: "web.futurebabynames.com", referrer: "https://web.futurebabynames.com/"
2022/11/17 03:13:32 [error] 109941#109941: *7 connect() failed (111: Unknown error) while connecting to upstream, client: xxx.xxx.xx.xx, server: web.futurebabynames.com, request: "PUT /api/babies/name HTTP/1.1", upstream: "http://127.0.0.1:9000/babies/name", host: "web.futurebabynames.com", referrer: "https://web.futurebabynames.com/"
2022/11/17 03:13:45 [error] 109941#109941: *7 connect() failed (111: Unknown error) while connecting to upstream, client: xxx.xxx.xx.xx, server: web.futurebabynames.com, request: "GET /api/auth/me HTTP/1.1", upstream: "http://127.0.0.1:9000/auth/me", host: "web.futurebabynames.com", referrer: "https://web.futurebabynames.com/"
2022/11/17 03:13:49 [error] 109941#109941: *7 connect() failed (111: Unknown error) while connecting to upstream, client: xxx.xxx.xx.xx, server: web.futurebabynames.com, request: "PUT /api/babies/name HTTP/1.1", upstream: "http://127.0.0.1:9000/babies/name", host: "web.futurebabynames.com", referrer: "https://web.futurebabynames.com/"
2022/11/17 03:13:58 [error] 109941#109941: *7 connect() failed (111: Unknown error) while connecting to upstream, client: xxx.xxx.xx.xx, server: web.futurebabynames.com, request: "GET /api/auth/me HTTP/1.1", upstream: "http://127.0.0.1:9000/auth/me", host: "web.futurebabynames.com", referrer: "https://web.futurebabynames.com/"

[nickcoast]

All the api requests fail. But, again, it was all working and all I did was:

Change subdomain from web to www for web_app_domain, and change web_api_domain subdomain also to www from web

Admin domains were not changed, but the admin api still isn't working.

[AlexITC]

Yes, I looked there and removed the old files just in case but it had no effect.

It is hard to tell without looking into it, I don't mind taking a short screen-sharing session to see what's going on (there is a link to my calendar in the repo main page).

Okay, so on my development machine, I don't need to run sbt compile before running ansible-playbook if I have, for example, uncommitted changes or have not run sbt compile since my last changes to the application code?

No, similar to web, server playbook compiles the app before uploading it to the server (see https://github.com/wiringbits/scala-webapp-template/blob/master/infra/server.yml), the linked scripts run sbt compile:

    - name: Build the application
      retries: 10
      delay: 5
      shell: "./scripts/build-server.sh {{ app_source_name }}"
      delegate_to: 127.0.0.1

I ran sudo apt purge nginx nginx-common nginx-core on the deployment machine, then ran the ansible-playbook for nginx.yml and it successfully reinstalled. (sudo apt remove nginx resulted in ansible failure - I guess it left the server in a weird state)

Like I said above, it is hard to tell what's going on without digging further, when I break the servers, I usually just recreate them because that's usually faster.

Everything points that nginx has issues contacting the backend code.

wiringbits-server.service? It's running. Doesn't need to accept external traffic if on the same server, right?

Correct, what you shared is the systemd status, can you check the app logs? I think they are at /home/play/logs/application.log.

Have you tried invoking curl localhost:9000/health?

Nov 17 03:28:08 ip-172-30-0-242 systemd[1]: wiringbits-server.service: Scheduled restart job, restart counter is at 13.

Given this log, I would believe that the app is in an infinite restart loop due to an error, which would explain that nginx can't contact backend.

Like this? `curl -v 172.30.0.242`

No, invoke curl -v localhost:9000/health which is what nginx would invoke, I bet you will get an error because backend is not running, invoking the ip address directly ends up talking to nginx.

Nginx error log has 111 unknown error

Same reason, backend is most likely in a restart loop.

[nickcoast]

Correct, what you shared is the systemd status, can you check the app logs?

Log for yesterday is huge 6.5M. Probably because of the restart loop. Today's log is 684k after about 4 hours. They warn that I have included destructive downs in my play evolutions. I guess should remove those? I think I must have misunderstood their purpose.

2022-11-17 04:21:03,239 [INFO] from net.wiringbits.tasks.NotificationsTask in main - Starting the notifications task
2022-11-17 04:21:54,437 [INFO] from play.api.db.DefaultDBApi in main - Database [default] initialized
2022-11-17 04:21:54,734 [INFO] from play.api.db.HikariCPConnectionPool in main - Creating Pool for datasource 'default'
2022-11-17 04:22:06,456 [WARN] from play.api.db.evolutions.ApplicationEvolutions in main - Your production database [default] needs evolutions, including downs! 

-- !!! WARNING! This script contains DOWNS evolutions that are likely destructive

... (list of queries in evolutions ...

2022-11-17 04:22:06,581 [WARN] from play.api.db.evolutions.ApplicationEvolutions in main - Run with -Dplay.evolutions.db.default.autoApply=true and -Dplay.evolutions.db.default.autoApplyDowns=true if you want to run them automatically, including downs (be careful, especially if your down evolutions drop existing data)
2022-11-17 04:22:31,904 [INFO] from net.wiringbits.modules.ConfigModule in main - Config loaded: AwsConfig(region = us-west-1, accessKeyId = RE...ME, secretAccessKey = RE...ME)
2022-11-17 04:22:31,949 [INFO] from net.wiringbits.modules.ConfigModule in main - Config loaded: EmailConfig(senderAddress = admin@ogredata.com, provider = none)
2022-11-17 04:22:31,950 [INFO] from net.wiringbits.modules.ConfigModule in main - Config loaded: NotificationsConfig(interval = 1 minute)
2022-11-17 04:22:31,952 [INFO] from net.wiringbits.modules.ConfigModule in main - Config loaded: ReCaptchaConfig(secret = 6L...We, siteKey = ReCaptchaSiteKey(XXXOBFUSCATEDXXX))
2022-11-17 04:22:31,952 [INFO] from net.wiringbits.modules.ConfigModule in main - Config loaded: UserTokensConfig(emailVerificationExp = 1 day, resetPasswordExp = 1 day, hmacSecret = CW...M=)
2022-11-17 04:22:31,953 [INFO] from net.wiringbits.modules.ConfigModule in main - Config loaded: WebAppConfig(host = web.futurebabynames.com)
2022-11-17 04:22:36,012 [INFO] from net.wiringbits.modules.ApisModule$EmailApiProvider in main - Mail provider set to none, emails will be printed as logs
2022-11-17 04:22:36,048 [INFO] from net.wiringbits.webapp.utils.admin.tasks.DataExplorerConfigValidatorTask in main - Running task
2022-11-17 04:22:38,743 [INFO] from net.wiringbits.webapp.utils.admin.tasks.DataExplorerConfigValidatorTask in main - Verifying users
2022-11-17 04:22:38,992 [INFO] from net.wiringbits.webapp.utils.admin.tasks.DataExplorerConfigValidatorTask in main - Verifying baby_names
2022-11-17 04:22:39,078 [INFO] from net.wiringbits.webapp.utils.admin.tasks.DataExplorerConfigValidatorTask in main - Verifying eras
2022-11-17 04:22:39,242 [INFO] from net.wiringbits.webapp.utils.admin.tasks.DataExplorerConfigValidatorTask in main - Verifying parent_fact_categories
2022-11-17 04:22:39,389 [INFO] from net.wiringbits.webapp.utils.admin.tasks.DataExplorerConfigValidatorTask in main - Data explorer settings validated
2022-11-17 04:22:39,620 [DEBUG] from controllers.AssetsConfiguration in main - Using the following cache configuration for assets:
     enableCaching = true
     enableCacheControl = true
     defaultCacheControl = public, max-age=3600
     aggressiveCacheControl = public, max-age=31536000, immutable
     configuredCacheControl:

2022-11-17 04:22:40,204 [INFO] from net.wiringbits.tasks.NotificationsTask in main - Starting the notifications task

Have you tried invoking curl localhost:9000/health?

I just tried it and it failed. You are correct.

ubuntu@ip-172-30-0-242:~$ curl localhost:9000/health
curl: (7) Failed to connect to localhost port 9000 after 0 ms: Connection refused

I checked journalctl -u wiringbits-server.service and got this:

Nov 17 04:32:24 ip-172-30-0-242 systemd[1]: wiringbits-server.service: Failed with result 'exit-code'.
Nov 17 04:32:24 ip-172-30-0-242 systemd[1]: wiringbits-server.service: Consumed 1min 34.178s CPU time.
Nov 17 04:32:24 ip-172-30-0-242 systemd[1]: wiringbits-server.service: Scheduled restart job, restart counter is at 31.
Nov 17 04:32:24 ip-172-30-0-242 systemd[1]: Stopped wiringbits-server.
Nov 17 04:32:24 ip-172-30-0-242 systemd[1]: wiringbits-server.service: Consumed 1min 34.178s CPU time.
Nov 17 04:32:24 ip-172-30-0-242 systemd[1]: Started wiringbits-server.
Nov 17 04:33:59 ip-172-30-0-242 systemd[1]: wiringbits-server.service: Main process exited, code=exited, status=255/EXCEPTION
Nov 17 04:33:59 ip-172-30-0-242 systemd[1]: wiringbits-server.service: Failed with result 'exit-code'.
Nov 17 04:33:59 ip-172-30-0-242 systemd[1]: wiringbits-server.service: Consumed 1min 33.148s CPU time.
Nov 17 04:33:59 ip-172-30-0-242 systemd[1]: wiringbits-server.service: Scheduled restart job, restart counter is at 32.
Nov 17 04:33:59 ip-172-30-0-242 systemd[1]: Stopped wiringbits-server.
Nov 17 04:33:59 ip-172-30-0-242 systemd[1]: wiringbits-server.service: Consumed 1min 33.148s CPU time.
Nov 17 04:33:59 ip-172-30-0-242 systemd[1]: Started wiringbits-server.
Nov 17 04:35:10 ip-172-30-0-242 systemd[1]: wiringbits-server.service: Main process exited, code=exited, status=255/EXCEPTION
Nov 17 04:35:10 ip-172-30-0-242 systemd[1]: wiringbits-server.service: Failed with result 'exit-code'.
Nov 17 04:35:10 ip-172-30-0-242 systemd[1]: wiringbits-server.service: Consumed 1min 7.884s CPU time.
Nov 17 04:35:10 ip-172-30-0-242 systemd[1]: wiringbits-server.service: Scheduled restart job, restart counter is at 33.
Nov 17 04:35:10 ip-172-30-0-242 systemd[1]: Stopped wiringbits-server.
Nov 17 04:35:10 ip-172-30-0-242 systemd[1]: wiringbits-server.service: Consumed 1min 7.884s CPU time.
Nov 17 04:35:10 ip-172-30-0-242 systemd[1]: Started wiringbits-server.

It is hard to tell without looking into it, I don't mind taking a short screen-sharing session to see what's going on (there is a link to my calendar in the repo main page).

If I can't figure this out, that'd be great! Thanks!

[AlexITC]

They warn that I have included destructive downs in my play evolutions. I guess should remove those? I think I must have misunderstood their purpose.

You need to make sure to check the official docs (https://www.playframework.com/documentation/2.8.x/Evolutions), looks like your server has applied evolution scripts that were removed/updated after a deployment, which causes this effect, if your current server data is not important, simpler way is to just recreate the database and restart the server, it must work after that. Otherwise, you will need to mark the evolution as applied in the database and make sure to apply the schema changes manually.

[nickcoast]

Hmmm, but does the log indicate that's what is making the server keep restarting?

I did read those docs before. I created that new evolution, which adds columns in Ups and removes them in downs, etc, and deployed those a couple of days ago and all the changes were applied successfully. Then I loaded more data and the site still worked.

So, it seemed like all the evolutions were working well but I understand your point and will check on that. It's just odd if that would be the problem, when everything worked in the days before.

[nickcoast]

Changed all my DNS and my hosts.ini to point to a different AWS server. Ran all the ansible-playbooks. And it worked.

wiringbits-server.service is running smoothly. I loaded my data and everyhing seems to be working. Couple bugs in the code related to NULLs in the DB but the system basically works.

Not sure if it's worth debugging the other server. Could maybe be interesting?

[AlexITC]

Not sure if it's worth debugging the other server. Could maybe be interesting?

I wonder if there was a time where you deployed to this server, then, migrated to the new one after making changes to evolution scripts, this could explain why going back to previous server got issues applying evolutions. I'd bet these two servers use different databases. Assuming there is no useful data a your server db, just recreate the database and everything should be back to normal.

[AlexITC]

Hmmm, but does the log indicate that's what is making the server keep restarting?

Yes, this log you pasted complains about unapplied evolutions, the app will refuse to start without that.

2022-11-17 04:22:06,456 [WARN] from play.api.db.evolutions.ApplicationEvolutions in main - Your production database [default] needs evolutions, including downs! 

-- !!! WARNING! This script contains DOWNS evolutions that are likely destructive

... (list of queries in evolutions ...

So, it seemed like all the evolutions were working well but I understand your point and will check on that. It's just odd if that would be the problem, when everything worked in the days before.

This happens when you create evolution scripts like 1.sql/2.sql and deploy, then, updating any of these files and deploy again, ideally, an evolution script must not be modified once it gets deployed.