Dockerize?

[ProZachJ]

I could help dockerize components or complete deployments of AppSensor. Which would be most helpful?

[ProZachJ]

For example, I could create dockerfiles for the sample apps so you could run them as a container. If so, any preference on which stack i use (tomcat, jetty, other)?

which ones should I do first?

[jtmelton]

@ProZachJ This would be great. I think the first thing I'd like to do is get the following 3 sample apps done:

• https://github.com/jtmelton/appsensor/tree/master/sample-apps/appsensor-ws-rest-server-boot
• https://github.com/jtmelton/appsensor/tree/master/sample-apps/appsensor-ws-rest-server-with-websocket-boot
• https://github.com/jtmelton/appsensor/tree/master/sample-apps/appsensor-ws-rest-client-boot-data-generator

After that, I'd want to get the appsensor-ui project done as well:

• https://github.com/jtmelton/appsensor/tree/master/appsensor-ui

When starting down this path, I was planning to follow instructions here: https://spring.io/guides/gs/spring-boot-docker/ . If that's not a good guide, the key terms are "spring boot" and "docker".

All of the apps above (and probably all the ones we'd want dockerized are spring boot apps, which build a fat jar (everything self contained, including the container). Given that the apps are all the same (boot apps), once they're done, it should be straightforward to build some "starter" apps for folks looking for canned servers they can just grab.

Also, thanks a bunch. This is something that'll help us out a lot.

[ProZachJ]

Starting to look at this tonight. How shall the containers be tested?

In the past, I've done this by running the container and requesting all the important URLs checking response codes or bodies for key info. It takes time to build up to total coverage, but I usually start by establishing some tests that can at least detect a container that is broken in major ways.

What is the unit and integration test story right now?

I'm going to be learning this toolset here so, bear with me.

[jtmelton]

The tests now are split across pure unit tests (junit) and integration tests (spring integration testing). Essentially the integration tests stand up a spring environment based on configuration and then execute the tests with the needed dependencies.

For the applications above, they are web apps, so there are not any tests right now for them. Given the 4 above I think I'd do them in the order listed.

1. rest-server-boot: should be straightforward. There are no parameters to send, just run the packaged application. The testing would essentially be sending an http request hitting a GET endpoint and making sure you get back an empty array. That would tell you the app is up and running. Beyond that, you could send a handful of POSTs which would trigger a response, then make sure that the GET returns 1 object in the responses array.
2. rest-server-with-websocket-boot: similar to (1), but with the addition of passing in a system property. I would test it in much the same way.
3. rest-client-boot-data-generator: this is tougher. You'll have to start one of [1,2], wait until you know they're loaded successfully, then start the client. Success will be measured by waiting a few seconds, then making sure the responses array is non-empty.
4. appsensor-ui: this is the hardest one. It may not be very testable as is. It depends on a mysql DB being available, as well as the [2,3] above being loaded and running. After that, "success" would be defined as logging into the app successfully, and hitting a couple pages and making sure that data is populated.

Lots of info here, let me know if it makes sense, or if you have more questions.

[ProZachJ]

No makes sense, I've just got some reading to do.

Right now just trying to build the first sample app.

[ProZachJ]

I need more details on websocket what do I need to pass in? it seems to be running but don't know how to tell.

[jtmelton]

For the websocket project, the way to run it is:

mvn spring-boot:run -DAPPSENSOR_WEB_SOCKET_HOST_URL=ws://localhost:8085/dashboard

[ProZachJ]

any idea how that will work when running the jar? is that something i can put in env?

[jtmelton]

yep - env vars should work. The full list is here: http://docs.spring.io/spring-boot/docs/1.3.0.M5/reference/htmlsingle/#boot-features-external-config. You can see the resolution order. Env vars are listed, so they should resolve fine.

[ProZachJ]

should I leave the value configurable at launch of the container?

[jtmelton]

yep, that's a good idea.

[ProZachJ]

should i see any output to a browser if I'm running the generator?

[jtmelton]

If you have the server with websocket running and the client data generator running (with no failures), you can connect to the websocket address and see output yes. That's what the appsensor-ui does.

[ProZachJ]

The generator looks great but I do see some errors now and again when I look at the ws server output. I also see it getting events though.

[jtmelton]

at the beginning, it gets a few startup errors due to some "response already committed" issues - from what I can tell it's an internal issue to the library itself, but the events are still getting gathered even when the trace happens

[ProZachJ]

05:30:23.068 [http-nio-8085-exec-5] INFO  o.o.a.r.SimpleLoggingReportingEngine - Reporter observed event by user [susan]
05:30:23.171 [http-nio-8085-exec-6] INFO  o.o.a.r.WebSocketReportingEngine - Connected ... 7
05:30:23.171 [http-nio-8085-exec-5] INFO  o.o.a.r.WebSocketReportingEngine - Connected ... 9
Opened connection with client: 0
Opened connection with client: 3
Opened connection with client: 4
Opened connection with client: 2
Opened connection with client: 1
05:30:23.172 [http-nio-8085-exec-8] INFO  o.o.a.r.WebSocketReportingEngine - Connected ... 5
05:30:23.172 [http-nio-8085-exec-4] INFO  o.o.a.r.WebSocketReportingEngine - Connected ... 8
05:30:23.172 [http-nio-8085-exec-7] INFO  o.o.a.r.WebSocketReportingEngine - Connected ... 6
started and connected
started and connected
started and connected
started and connected
started and connected
05:30:23.263 [http-nio-8085-exec-7] ERROR o.a.c.c.C.[.[.[.[jerseyServlet] - Servlet.service() for servlet [jerseyServlet] in context with path [] threw exception [java.lang.IllegalStateException: The remote endpoint was in state [TEXT_FULL_WRITING] which is an invalid state for called method] with root cause
java.lang.IllegalStateException: The remote endpoint was in state [TEXT_FULL_WRITING] which is an invalid state for called method
    at org.apache.tomcat.websocket.WsRemoteEndpointImplBase$StateMachine.checkState(WsRemoteEndpointImplBase.java:1148) ~[tomcat-embed-websocket-8.0.23.jar:8.0.23]
    at org

[jtmelton]

yep that error is seen a few times at startup ... it should clear up after a few events go across the wire

[jtmelton]

From the research I did, appears this is a tomcat bug, and doesn't happen in jetty. Maybe a possibility for upgrade in the future.

[ProZachJ]

Ok so the websocket server is definitely running and working as confirmed by running the UI.

However when trying to mvn package I get a test failure:

java.lang.IllegalStateException: javax.websocket.server.ServerContainer not available

This thread seems relevant but concerning that it appears to be 'fixed'.

https://jira.spring.io/browse/SPR-12340

Thoughts?

[jtmelton]

So to be clear, this is when running all 3 apps [rest-server-with-websocket, rest-client, and appsensor-ui] in containers? You see the error at that point? Want to try and duplicate on my side.

[jtmelton]

Ok, so I was able to duplicate what issues I'm guessing you were seeing. There were a few things I did to resolve them.

• commented out the test classes in [appsensor-ui, rest-server-with-websocket]. Those tests were not doing anything anyways, so no harm done
• copied the changes you made related to requiresUnpack in the maven boot plugin for rest-server over the the rest-server-with-websocket along with adding a few extra libraries that were necessary
• instead of running with the command line params ( -DABC=XYZ ), I had to actually set env vars ( export ABC=XYZ ) to get it to run via 'java -jar ...' mode.
• bumped the version number on spring boot (not necessary, but while we're at it ...)

Specifically, I ran the following (for appsensor-ui):

export APPSENSOR_REST_REPORTING_ENGINE_URL=http://localhost:8085 export APPSENSOR_CLIENT_APPLICATION_ID_HEADER_NAME=X-Appsensor-Client-Application-Name2 export APPSENSOR_CLIENT_APPLICATION_ID_HEADER_VALUE=myclientapp export APPSENSOR_WEB_SOCKET_HOST_URL=ws://localhost:8085/dashboard env "spring.datasource.url=jdbc:mysql://localhost/appsensor" bash env "spring.datasource.username=appsensor_user" bash env "spring.datasource.password=appsensor_pass" bash

Locally, I use bash so the last few may differ for you.

After making these changes, it seemed to run just fine for me. I'll be pushing these changes shortly.

[jtmelton]

Ok, these changes should be available to pull down now ... let me know if things work for you.

[ProZachJ]

I had already made the similar pom changes, it was just the test cases stopping me. I'll pull and check it out.

[ProZachJ]

Yea worked great for the websocket server. Finishing up readme on that then I'll start working on a branch for ui.

[jtmelton]

awesome

[ProZachJ]

For the UI container I assume we want one with the demo creds first and then later do "prod ready"?

If so, is it ok to commit another .sql file that isn't commented out? I need it to bring into the container and run.

[jtmelton]

If you need to add another sql file, I'm fine with that. I assumed mysql would still be external to the container, and that all the "creds" as well as other things would be passed in via env vars. Does that not work or is this something else?

[ProZachJ]

I can create a root pass, db name, and user pass via env vars yes. But creating the actual web users at the end appears a bit more complex.

[ProZachJ]

To explain, I'm trying to setup instructions for using the official mysql container as the datastore.

[jtmelton]

ah ok, yes I get your point now. my long term goal was to

• have instructions on how to setup an admin user for creating/editing accounts
• setup a UI for user admin

but for now, a canned starter makes sense, yes

[ProZachJ]

Agreed, do the user creation routines already exist and just need UI?

If not I could just show the commands to execute against the mysql container to create additional users, add applications to those users, etc.

[jtmelton]

no, there's no code yet, hence the straight mysql commands that are commented out in the mysql script

[ProZachJ]

Would you prefer documenting how to add users manually to committing another script?

[jtmelton]

I think I'd rather go with adding a sql script for now, so that people can get started. When the UI is built, we can re-address. At that point, we can set up the documentation for how to add a single admin, and then use the UI to create/manage other users.

Having said that, please add a note that users are created by default in the current version so folks know that there are default accounts.

[ProZachJ]

Ok will do.

[ProZachJ]

Got it working all containerized. I need to clean up a couple things to make it easier to deploy and shut down the open port on mysql since I can link it directly to the UI container.

[jtmelton]

awesome - sounds great

[jtmelton]

Made a minor change to address the illegal state exception issue from before. Turns out both tomcat and jetty have issues with the sync and async websocket implementations with respect to multithreaded access. By locking on the client side in the tests - those exceptions go away. Addressed with https://github.com/jtmelton/appsensor/commit/1bf4713973bd1abb155ae94999b9b9a958b31097

[ProZachJ]

Awesome. I should be able to get my latest work ready to merge this week.

On Oct 25, 2015, at 9:57 PM, John Melton notifications@github.com wrote:

Made a minor change to address the illegal state exception issue from before. Turns out both tomcat and jetty have issues with the sync and async websocket implementations with respect to multithreaded access. By locking on the client side in the tests - those exceptions go away.

— Reply to this email directly or view it on GitHub.

[jtmelton]

Never closed this, but @ProZachJ completed and merged this work in late 2015 ... cleaning up