dataflow server keeps crashing right after startup

[balistof]

When I deploy the dataflow server on mesosphere the task crashes a few seconds after the launch.

The last log I see is:

Registered docker executor on 10.227.245.189
Starting task spring-cloud-data-flow.cccf6b33-96d3-11e6-b766-70b3d5800001
2016-10-20 14:45:06.660  INFO 1 --- [           main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@5a888b14: startup date [Thu Oct 20 14:45:06 GMT 2016]; root of context hierarchy
2016-10-20 14:45:07.200  INFO 1 --- [           main] f.a.AutowiredAnnotationBeanPostProcessor : JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2016-10-20 14:45:07.295  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'configurationPropertiesRebinderAutoConfiguration' of type [class org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$8e9cf773] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2016-10-20 14:45:07.853  INFO 1 --- [           main] .c.d.s.c.DefaultEnvironmentPostProcessor : spring.application.name=spring-cloud-dataflow-server-mesos
  ____                              ____ _                __
 / ___| _ __  _ __(_)_ __   __ _   / ___| | ___  _   _  __| |
 \___ \| '_ \| '__| | '_ \ / _` | | |   | |/ _ \| | | |/ _` |
  ___) | |_) | |  | | | | | (_| | | |___| | (_) | |_| | (_| |
 |____/| .__/|_|  |_|_| |_|\__, |  \____|_|\___/ \__,_|\__,_|
  ____ |_|    _          __|___/                 __________
 |  _ \  __ _| |_ __ _  |  ___| | _____      __  \ \ \ \ \ \
 | | | |/ _` | __/ _` | | |_  | |/ _ \ \ /\ / /   \ \ \ \ \ \
 | |_| | (_| | || (_| | |  _| | | (_) \ V  V /    / / / / / /
 |____/ \__,_|\__\__,_| |_|   |_|\___/ \_/\_/    /_/_/_/_/_/

 1.0.1.RELEASE

2016-10-20 14:45:07.952  INFO 1 --- [           main] c.c.c.ConfigServicePropertySourceLocator : Fetching config from server at: http://localhost:8888
2016-10-20 14:45:08.143  WARN 1 --- [           main] c.c.c.ConfigServicePropertySourceLocator : Could not locate PropertySource: I/O error on GET request for "http://localhost:8888/spring-cloud-dataflow-server-mesos/default": Connection refused; nested exception is java.net.ConnectException: Connection refused
2016-10-20 14:45:08.145  INFO 1 --- [           main] o.s.c.d.s.mesos.MesosDataFlowServer      : No active profile set, falling back to default profiles: default
2016-10-20 14:45:08.153  INFO 1 --- [           main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@7c512760: startup date [Thu Oct 20 14:45:08 GMT 2016]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@5a888b14

Marathon UI is telling me following:

Last Task Failure
Task id
spring-cloud-data-flow.138030b5-96d4-11e6-b766-70b3d5800001
State
TASK_FAILED
Message
Container exited with status 137
Host
10.227.245.189
Timestamp
2016-10-20T14:47:14.192Z (Just now)
Version
2016-10-20T14:45:09.960Z (2 minutes ago)
Mesos Details
link

Any suggestions where I can dig?

[sabbyanandan]

@balistof: Are you following the getting-started instructions? Also, according to your stacktrace, the version is printed as 1.0.1.RELEASE, but the latest release of SCDF's mesos-server is at 1.0.0.RELEASE - could you clarify the version and where you downloaded the bits from?

[balistof]

@sabbyanandan: yes I am following the instruction with docker image: springcloud/spring-cloud-dataflow-server-mesos:1.0.0.RELEASE but the logs print 1.0.1 release...

What I have just seen is that if I increase the memory to 1GB it started working.

[trisberg]

The logs show the version of SCDF Core. SCDF for Mesos 1.0.0.RELEASE builds on the 1.0.1.RELEASE version of the core project. Maybe we should look into replacing the banner with an implementation specific one.

[sabbyanandan]

@trisberg: Ah, yeah, looks like you fixed the banner across all the SCDF servers now! We default the memory setting at 512MB, should we explicitly document the memory requirement to be a little more?

[balistof]

I would rather suggest setting the java memory limits on the spring boot application running inside docker. There seems to be general issues with memory allocation and heap sizing as java cannot detect available memory correctly inside docker.

[trisberg]

You could experiment with setting a "_JAVA_OPTIONS" env var for the server deployment in scdf-server.json. We added additional support for providing env vars for each deployment for the Kubernetes deployer, we should probably do that for the Mesos deployer as well.