spring-attic / spring-cloud-dataflow-server-yarn

Spring Cloud Data Flow Implementation for Apache YARN
http://cloud.spring.io/spring-cloud-dataflow-server-yarn/
Apache License 2.0
15 stars 34 forks source link

Spring Cloud Data Flow for Apache YARN Memory leak #168

Open mixergit opened 7 years ago

mixergit commented 7 years ago

20170928110945 Use a period of time yarnModuleDeployerTaskExecutor thread resource occupancy will be ascribed, resulting in memory overflow

mixergit commented 7 years ago
private synchronized YarnCloudAppServiceApplication getApp(String appVersion, String dataflowVersion, CloudAppType cloudAppType,
        List<String> contextRunArgs) {
    contextRunArgs = processContextRunArgs(contextRunArgs);
    String cacheKey = cloudAppType + appVersion + StringUtils.collectionToCommaDelimitedString(contextRunArgs);
    YarnCloudAppServiceApplication app = appCache.get(cacheKey);
    logger.info("Cachekey {} found YarnCloudAppServiceApplication {}", cacheKey, app);
    if (app == null) {
        Properties configFileProperties = new Properties();
        if (StringUtils.hasText(appVersion)) {
            configFileProperties.setProperty("spring.yarn.applicationVersion", appVersion);
        }
        if (StringUtils.hasText(dataflowVersion)) {
            configFileProperties.setProperty("spring.cloud.deployer.yarn.version", dataflowVersion);
        }

        logger.info("Bootsrapping YarnCloudAppServiceApplication with {}", cloudAppType.toString().toLowerCase());
        ArrayList<String> runArgs = new ArrayList<String>();
        runArgs.add("--spring.config.name=" + cloudAppType.toString().toLowerCase());
        runArgs.add("--spring.jmx.enabled=false");

        if (!ObjectUtils.isEmpty(contextRunArgs)) {
            runArgs.addAll(contextRunArgs);
        }

        app = new YarnCloudAppServiceApplication(appVersion, dataflowVersion, "application.properties", configFileProperties,
                runArgs.toArray(new String[0]), initializers);
        try {
            app.afterPropertiesSet();
        } catch (Exception e) {
            throw new RuntimeException("Error initializing YarnCloudAppServiceApplication", e);
        }
        logger.info("Set cache with key {} and YarnCloudAppServiceApplication {}", cacheKey, app);
        appCache.put(cacheKey, app);
    }
    return app;
}

Two issues: One, appCache not Recycling resources. Two, YarnCloudAppServiceApplication context not destroy. 20171012165147