Azure-Samples / hdinsight-java-storm-wordcount

A basic example of a Java-based Apache Storm Topology, and how to deploy and run it with HDInsight.
MIT License
9 stars 9 forks source link

Getting error on logs/access-web-logs/access-web-${sys:daemon.name}.log when running the Flux Example #3

Open thinklarge opened 6 years ago

thinklarge commented 6 years ago

I followed the setup guide and have been able to run the Java example but have been incapable of getting the Flux example running.

It looks like my sample is getting caught up on the log4j somewhere. I'm getting an exception that looks like it's in the logger.

I'm going to include the body of text below to show the exception that I'm getting. The Location it's attempting to drop the logs/access-web is my %STORM_HOME%/logs I'm not sure if I need to configure permissions over there to get things working or not.

I'm running on

Do I need to try another version of python, Java, set permissions? I'm sorry to post a wall of exception, but I'm at a loss as to what to do next. Thanks for any help you can offer.

2017-12-11 16:28:44,792 main ERROR Unable to create file C:\JavaProgs\apache-storm-1.1.1\logs/access-web-${sys:daemon.name}.log java.io.IOException: The filename, dire ctory name, or volume label syntax is incorrect at java.base/java.io.WinNTFileSystem.canonicalizeWithPrefix0(Native Method) at java.base/java.io.WinNTFileSystem.canonicalizeWithPrefix(WinNTFileSystem.java:454) at java.base/java.io.WinNTFileSystem.canonicalize(WinNTFileSystem.java:425) at java.base/java.io.File.getCanonicalPath(File.java:618) at java.base/java.io.File.getCanonicalFile(File.java:643) at org.apache.logging.log4j.core.util.FileUtils.makeParentDirs(FileUtils.java:134) at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:573) at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:554) at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:112) at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114) at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:155) at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:131) at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:60) at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122) at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:952) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:892) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:884) at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:508) at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:232) at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:244) at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:545) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:617) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:634) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:229) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:152) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:122) at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:46) at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383) at org.apache.storm.flux.Flux.(Flux.java:42)

2017-12-11 16:28:44,801 main ERROR Unable to inject fields into builder class for plugin type class org.apache.logging.log4j.core.appender.RollingFileAppender, element RollingFile. java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.a ppender.rolling.RollingFileManager$RollingFileManagerFactory@82de64a] unable to create manager for [C:\JavaProgs\apache-storm-1.1.1\logs/access-web-${sys:daemon.name}.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@ 659499f1[pattern=C:\JavaProgs\apache-storm-1.1.1\logs/access-web-${sys:daemon.name}.log.%i.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=104857600)]), strategy=DefaultRolloverStrateg y(min=1, max=9, useMax=true), advertiseURI=null, layout=%d{yyyy-MM-dd HH:mm:ss.SSS} %c{1.} %t [%p] %msg%n]] at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:114) at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114) at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:155) at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:131) at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:60) at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122) at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:952) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:892) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:884) at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:508) at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:232) at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:244) at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:545) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:617) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:634) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:229) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:152) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:122) at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:46) at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383) at org.apache.storm.flux.Flux.(Flux.java:42)

2017-12-11 16:28:44,803 main ERROR Unable to invoke factory method in class class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile. java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.cor e.appender.RollingFileAppender at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:224) at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:130) at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:952) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:892) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:884) at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:508) at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:232) at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:244) at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:545) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:617) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:634) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:229) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:152) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:122) at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:46) at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383) at org.apache.storm.flux.Flux.(Flux.java:42)

2017-12-11 16:28:44,849 main ERROR Null object returned for RollingFile in appenders. 2017-12-11 16:28:44,856 main ERROR Unable to locate appender "WEB-ACCESS" for logger config "org.apache.storm.logging.filters.AccessLoggingFilter"

Blackmist commented 6 years ago

Hi @thinklarge it looks like this is a side effect of Storm being developed primarily on Linux, and probably not heavily tested on Windows.

The default configuration assumes that there's a web log (I'm guessing the Storm web UI) and tries to log it. But the variable (${sys:daemon.name}) isn't being replaced, and some of those characters aren't legal in a Windows file path/name.

I was able to work around this problem on a Windows system by using the following steps:

  1. Go to the directory that you installed the Storm development environment into.
  2. In the log4j2 subdirectory, edit cluster.xml.
  3. find the line that begins with <RollingFile name="WEB-ACCESS". Remove the string -${sys:daemon.name} from the fileName property of this line. This results in Storm creating a file named access-web.log, which Windows understands. Save the file after making the change.

One other thing that I noticed on Windows is that the Storm command doesn't send the output of the topology to the console. I tried with both cmd.exe and PowerShell. Instead, the output is logged to <storm installation directory>\logs\jar.log.

Can you give this a try and see if it works aroudn the error for you?

NicolasPhan commented 5 years ago

Hi @Blackmist, I encountered the same issue and your workaround fixed the issue ! Thanks a lot ! :)