search

embulk / embulk

Embulk: Pluggable Bulk Data Loader.
https://www.embulk.org/
Apache License 2.0
1.75k stars 201 forks source link

Un-friendly message raised when I exeucute embulk run without liquid suffix #436

Open hiroyuki-sato opened 8 years ago

hiroyuki-sato commented 8 years ago

I use liquid template

When I execute embulk run config.yml without .liquid suffix The following exception raised.

the more user-friendly message need.

config.yml

in:
  type: file
  path_prefix: {{env.my_path}} # <-- liquid
  decoders:
  - {type: gzip}
  parser:
    charset: UTF-8
    newline: CRLF
    type: csv
    delimiter: ','
    quote: '"'
    escape: '"'
    null_string: 'NULL'
    trim_if_not_quoted: false
    skip_header_lines: 1
    allow_extra_columns: false
    allow_optional_columns: false
    columns:
    - {name: id, type: long}
    - {name: account, type: long}
    - {name: time, type: timestamp, format: '%Y-%m-%d %H:%M:%S'}
    - {name: purchase, type: timestamp, format: '%Y%m%d'}
    - {name: comment, type: string}
out: {type: stdout}

embulk run config.yml

org.embulk.exec.PartialExecutionException: org.embulk.config.ConfigException: com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.lang.String out of START_OBJECT token
 at [Source: N/A; line: -1, column: -1]
    at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(org/embulk/exec/BulkLoader.java:363)
    at org.embulk.exec.BulkLoader.doRun(org/embulk/exec/BulkLoader.java:572)
    at org.embulk.exec.BulkLoader.access$000(org/embulk/exec/BulkLoader.java:33)
    at org.embulk.exec.BulkLoader$1.run(org/embulk/exec/BulkLoader.java:374)
    at org.embulk.exec.BulkLoader$1.run(org/embulk/exec/BulkLoader.java:370)
    at org.embulk.spi.Exec.doWith(org/embulk/spi/Exec.java:25)
    at org.embulk.exec.BulkLoader.run(org/embulk/exec/BulkLoader.java:370)
    at org.embulk.EmbulkEmbed.run(org/embulk/EmbulkEmbed.java:180)
    at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
    at RUBY.run(/Users/hsato/.embulk/bin/embulk!/embulk/runner.rb:84)
    at RUBY.run(/Users/hsato/.embulk/bin/embulk!/embulk/command/embulk_run.rb:306)
    at RUBY.<top>(/Users/hsato/.embulk/bin/embulk!/embulk/command/embulk_main.rb:2)
    at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:937)
    at RUBY.(root)(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1)
    at Users.hsato.$_dot_embulk.bin.embulk.embulk.command.embulk_bundle.<top>(file:/Users/hsato/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb:51)
    at java.lang.invoke.MethodHandle.invokeWithArguments(java/lang/invoke/MethodHandle.java:627)
    at org.embulk.cli.Main.main(org/embulk/cli/Main.java:23)
    Suppressed: java.lang.NullPointerException
        at org.embulk.exec.BulkLoader.doCleanup(BulkLoader.java:479)
        at org.embulk.exec.BulkLoader$3.run(BulkLoader.java:410)
        at org.embulk.exec.BulkLoader$3.run(BulkLoader.java:406)
        at org.embulk.spi.Exec.doWith(Exec.java:25)
        at org.embulk.exec.BulkLoader.cleanup(BulkLoader.java:406)
        at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:184)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:452)
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:313)
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:45)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:313)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:140)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:126)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:273)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)
        at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:414)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:348)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
        at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:86)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:210)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:196)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:197)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:313)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
        at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:112)
        at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:99)
        at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:35)
        at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
        at org.jruby.Ruby.runInterpreter(Ruby.java:833)
        at org.jruby.Ruby.loadFile(Ruby.java:2905)
        at org.jruby.runtime.load.LibrarySearcher$ResourceLibrary.load(LibrarySearcher.java:235)
        at org.jruby.runtime.load.LibrarySearcher$FoundLibrary.load(LibrarySearcher.java:34)
        at org.jruby.runtime.load.LoadService.tryLoadingLibraryOrScript(LoadService.java:900)
        at org.jruby.runtime.load.LoadService.smartLoadInternal(LoadService.java:541)
        at org.jruby.runtime.load.LoadService.requireCommon(LoadService.java:426)
        at org.jruby.runtime.load.LoadService.require(LoadService.java:392)
        at org.jruby.RubyKernel.requireCommon(RubyKernel.java:944)
        at org.jruby.RubyKernel.require19(RubyKernel.java:937)
        at org.jruby.RubyKernel$INVOKER$s$1$0$require19.call(RubyKernel$INVOKER$s$1$0$require19.gen)
        at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:380)
        at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:61)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:313)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
        at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:86)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:210)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:196)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:197)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:313)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at Users.hsato.$_dot_embulk.bin.embulk.embulk.command.embulk_bundle.invokeOther66:require(file:/Users/hsato/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb)
        at Users.hsato.$_dot_embulk.bin.embulk.embulk.command.embulk_bundle.RUBY$script(file:/Users/hsato/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb:51)
        at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
        at org.jruby.ir.Compiler$1.load(Compiler.java:111)
        at org.jruby.Ruby.runScript(Ruby.java:817)
        at org.jruby.Ruby.runScript(Ruby.java:809)
        at org.jruby.Ruby.runNormally(Ruby.java:747)
        at org.jruby.Ruby.runFromMain(Ruby.java:569)
        at org.jruby.Main.doRunFromMain(Main.java:415)
        at org.jruby.Main.internalRun(Main.java:310)
        at org.jruby.Main.run(Main.java:239)
        at org.jruby.Main.main(Main.java:201)
        at org.embulk.cli.Main.main(Main.java:23)
Caused by: org.embulk.config.ConfigException: com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.lang.String out of START_OBJECT token
 at [Source: N/A; line: -1, column: -1]
    at org.embulk.config.ModelManager.readObjectWithConfigSerDe(ModelManager.java:75)
    at org.embulk.config.DataSourceImpl.loadConfig(DataSourceImpl.java:220)
    at org.embulk.standards.LocalFileInputPlugin.transaction(LocalFileInputPlugin.java:60)
    at org.embulk.spi.FileInputRunner.transaction(FileInputRunner.java:65)
    at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:513)
    at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:33)
    at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:374)
    at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:370)
    at org.embulk.spi.Exec.doWith(Exec.java:25)
    at org.embulk.exec.BulkLoader.run(BulkLoader.java:370)
    at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:180)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:452)
    at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:313)
    at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:45)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:313)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:140)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:126)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:273)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)
    at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:414)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:348)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
    at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:86)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:210)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:196)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:197)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:313)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
    at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:112)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:99)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:35)
    at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
    at org.jruby.Ruby.runInterpreter(Ruby.java:833)
    at org.jruby.Ruby.loadFile(Ruby.java:2905)
    at org.jruby.runtime.load.LibrarySearcher$ResourceLibrary.load(LibrarySearcher.java:235)
    at org.jruby.runtime.load.LibrarySearcher$FoundLibrary.load(LibrarySearcher.java:34)
    at org.jruby.runtime.load.LoadService.tryLoadingLibraryOrScript(LoadService.java:900)
    at org.jruby.runtime.load.LoadService.smartLoadInternal(LoadService.java:541)
    at org.jruby.runtime.load.LoadService.requireCommon(LoadService.java:426)
    at org.jruby.runtime.load.LoadService.require(LoadService.java:392)
    at org.jruby.RubyKernel.requireCommon(RubyKernel.java:944)
    at org.jruby.RubyKernel.require19(RubyKernel.java:937)
    at org.jruby.RubyKernel$INVOKER$s$1$0$require19.call(RubyKernel$INVOKER$s$1$0$require19.gen)
    at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:380)
    at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:61)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:313)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
    at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:86)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:210)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:196)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:197)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:313)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
    at Users.hsato.$_dot_embulk.bin.embulk.embulk.command.embulk_bundle.invokeOther66:require(file:/Users/hsato/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb)
    at Users.hsato.$_dot_embulk.bin.embulk.embulk.command.embulk_bundle.RUBY$script(file:/Users/hsato/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb:51)
    at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
    at org.jruby.ir.Compiler$1.load(Compiler.java:111)
    at org.jruby.Ruby.runScript(Ruby.java:817)
    at org.jruby.Ruby.runScript(Ruby.java:809)
    at org.jruby.Ruby.runNormally(Ruby.java:747)
    at org.jruby.Ruby.runFromMain(Ruby.java:569)
    at org.jruby.Main.doRunFromMain(Main.java:415)
    at org.jruby.Main.internalRun(Main.java:310)
    at org.jruby.Main.run(Main.java:239)
    at org.jruby.Main.main(Main.java:201)
    at org.embulk.cli.Main.main(Main.java:23)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.lang.String out of START_OBJECT token
 at [Source: N/A; line: -1, column: -1]
    at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:148)
    at com.fasterxml.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:835)
    at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:59)
    at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:12)
    at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3534)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:1894)
    at org.embulk.config.TaskSerDe$TaskDeserializer.deserialize(TaskSerDe.java:161)
    at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3534)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:1870)
    at org.embulk.config.ModelManager.readObjectWithConfigSerDe(ModelManager.java:72)
    ... 77 more

Error: org.embulk.config.ConfigException: com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.lang.String out of START_OBJECT token
 at [Source: N/A; line: -1, column: -1]
aaronpowers commented 8 years ago

I have run into a very similar problem, although with a different root cause (which I am struggling to understand), and a slightly different error message.

As far as I can tell, my error is very similar one pasted above in terms of the same exception being raised, except that above it says "START_OBJECT token", my version says "VALUE_STRING token". I have been unable to debug this and would love to get friendlier errors.

My Exception does occur on the same line at the top level, and in the "Caused by" exception:

org.embulk.exec.PartialExecutionException: org.embulk.config.ConfigException:
com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of
java.util.ArrayList out of VALUE_STRING token
 at [Source: N/A; line: -1, column: -1]
    at
    org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(org/embulk/exec/BulkLoader.java:363)

<snip>

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of
java.util.ArrayList out of VALUE_STRING token
 at [Source: N/A; line: -1, column: -1]
    at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:148)
    at
hiroyuki-sato commented 8 years ago

Hello, @aaronpowers

The message Can not deserialize instance of java.util.ArrayList out of VALUE_STRING mean

Can not deserialize instance of <Expected Type> out of <Wrong Type>

The config requires Array but, You specify String.

For example

in:
  type: file
  path_prefix: /private/tmp/error/csv/sample_
# decoders must be Array
#  decoders:
#  - {type: gzip}
  decoders: gzip # Wrong type

Proper config 

  decoders:
  - {type: gzip}

But Now,

  decoders: gzip

Please check a String part in your configuration.

If you show me your configuration, I might find the problem.

Anyway, error message needs user-friendly.

aaronpowers commented 8 years ago

Thank you, Hiroyuki! I will look at my config for that pattern.