search

GoogleCloudPlatform / fluent-plugin-detect-exceptions

A fluentd plugin that scans line-oriented log streams and combines exceptions stacks into a single log entry.
Apache License 2.0
192 stars 69 forks source link

start fluentd error "superclass mismatch for class BufferQueueLimitError (TypeError)" #25

Closed jushiyuan closed 6 years ago

jushiyuan commented 6 years ago

error log

    from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.42/lib/fluent/plugin/buf_memory.rb:21:in `<top (required)>'
    from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/registry.rb:86:in `search'
    from /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/registry.rb:44:in `lookup'
    from /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin.rb:146:in `new_impl'
    from /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin.rb:112:in `new_buffer'
    from /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin/output.rb:328:in `configure'
    from /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin_helper/event_emitter.rb:73:in `configure'
    from /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.4.1/lib/fluent/plugin/out_elasticsearch.rb:96:in `configure'
    from /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin.rb:164:in `configure'
    from /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/agent.rb:130:in `add_match'
    from /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/agent.rb:72:in `block in configure'
    from /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/agent.rb:64:in `each'
    from /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/agent.rb:64:in `configure'
    from /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/root_agent.rb:112:in `configure'
    from /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/engine.rb:131:in `configure'
    from /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/engine.rb:96:in `run_configure'
    from /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/supervisor.rb:770:in `run_configure'
    from /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/supervisor.rb:553:in `dry_run'
    from /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/supervisor.rb:571:in `supervise'
    from /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/supervisor.rb:476:in `run_supervisor'
    from /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/command/fluentd.rb:310:in `<top (required)>'
    from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/bin/fluentd:8:in `<top (required)>'
    from /usr/bin/fluentd:22:in `load'
    from /usr/bin/fluentd:22:in `<main>'
time="2018-01-20T05:12:38Z" level=error msg="exit status 1"

fluentd config

@include conf.d/*.conf
<match docker.**>
@type detect_exceptions
languages java
</match>
<match docker.**>
@type elasticsearch
hosts http://elasticsearch-logging:9200
reconnect_on_error true
target_index_key @target
type_name fluentd
flush_mode immediate
</match>
igorpeshansky commented 6 years ago

You seem to be trying to use a v0.12 plugin with a v1.1 fluentd gem. What fluentd distribution are you using?

jushiyuan commented 6 years ago

@igorpeshansky Thanks for your reply. I install fluentd with <gem install fluentd -v "~> 0.12.0" --no-ri --no-rdoc> , and "fluentd --version" return "fluentd 1.1.0"

igorpeshansky commented 6 years ago

@jushiyuan, so you're installing the gems individually? What's the installation order? What are all the gems you install on top of a vanilla Ruby package? Can you reproduce this issue on a brand new VM?

jushiyuan commented 6 years ago

If I remove 

<match docker.**> 
@type detect_exceptions 
languages java 
</match>

Fluentd would run successfully, Is there a problem in the config？

jushiyuan commented 6 years ago

I run fluent in the docker，my Docker file:

FROM alpine:3.5

RUN apk update && apk upgrade && \  
  apk add ruby-json ruby-irb && \
  apk add build-base ruby-dev && \
  apk add python && \
  apk add lsof && \
  apk add ca-certificates wget && \
  gem install fluentd -v "~> 0.12.0" --no-ri --no-rdoc && \
  gem install fluent-plugin-elasticsearch --no-ri --no-rdoc && \
  gem install fluent-plugin-detect-exceptions --no-ri --no-rdoc && \
  gem install gelf -v "~> 3.0.0" --no-ri --no-rdoc && \
  gem install aliyun_sls_sdk -v ">=0.0.9" --no-ri --no-rdoc && \
  gem install remote_syslog_logger -v ">=1.0.1" --no-ri --no-rdoc && \
  gem install fluent-plugin-remote_syslog -v ">=0.2.1" --no-ri --no-rdoc && \
  gem install fluent-plugin-kafka --no-ri --no-rdoc && \
  apk del build-base ruby-dev && \
  rm -rf /root/.gem

COPY plugins/ /etc/fluentd/plugins/

VOLUME /etc/fluentd/conf.d

COPY pilot fluentd.tpl entrypoint config.default /pilot/
VOLUME /pilot/pos

EXPOSE 24224
WORKDIR /pilot/
#CMD exec fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT
CMD /pilot/entrypoint
jushiyuan commented 6 years ago

@igorpeshansky The log above may not be very clear.This is my fluentd.conf.

@include` conf.d/*.conf
<match docker.**>
@type detect_exceptions
languages java
</match>

<match docker.**>
@type elasticsearch
hosts http://elasticsearch-logging:9200
reconnect_on_error true
target_index_key @target
type_name fluentd
flush_mode immediate
</match>

If I remove the detect_exceptions part, Fluentd would be fine, so I think my detect_exceptions config is not right. I am sorry I am new to Fluentd.

igorpeshansky commented 6 years ago

It certainly looks like something is pulling in fluentd v1.1.0. Can you provide the output of gem list --all? How about gem dependency fluentd --reverse-dependencies?

jushiyuan commented 6 years ago

@igorpeshansky 

/etc/fluentd # gem list --all

*** LOCAL GEMS ***

addressable (2.5.2)
aliyun_sls_sdk (0.0.9)
beefcake (0.5.0)
connection_pool (2.2.1)
cool.io (1.5.3)
did_you_mean (1.0.0)
dig_rb (1.0.1)
elasticsearch (6.0.0)
elasticsearch-api (6.0.0)
elasticsearch-transport (6.0.0)
excon (0.60.0)
faraday (0.13.1)
file-tail (0.1.4)
fluent-plugin-detect-exceptions (0.0.8)
fluent-plugin-elasticsearch (2.4.1)
fluent-plugin-kafka (0.6.6)
fluent-plugin-remote_syslog (1.0.0)
fluentd (1.1.0, 0.12.42)
gelf (3.0.0)
http_parser.rb (0.6.0)
json (1.8.3.1)
ltsv (0.1.0)
msgpack (1.2.2)
multi_json (1.13.1)
multipart-post (2.0.0)
net-http-persistent (3.0.0)
net-telnet (0.1.1)
power_assert (0.2.6)
psych (2.1.0.1)
public_suffix (3.0.1)
remote_syslog_logger (1.0.3)
remote_syslog_sender (1.2.1)
ruby-hmac (0.4.0)
ruby-kafka (0.5.2)
serverengine (2.0.6)
sigdump (0.2.4)
string-scrub (0.0.5)
strptime (0.2.3)
syslog_protocol (0.9.2)
test-unit (3.1.5)
thread_safe (0.3.6)
tzinfo (1.2.4)
tzinfo-data (1.2017.3)
yajl-ruby (1.3.1)
/etc/fluentd # gem dependency fluentd --reverse-dependencies
Gem fluentd-0.12.42
  cool.io (< 2.0.0, >= 1.2.2)
  flexmock (~> 2.0, development)
  http_parser.rb (< 0.7.0, >= 0.5.1)
  json (>= 1.4.3)
  msgpack (< 2, >= 0.5.11)
  oj (~> 2.14, development)
  parallel_tests (~> 0.15.3, development)
  rake (>= 0.9.2, development)
  rr (~> 1.0, development)
  sigdump (~> 0.2.2)
  simplecov (~> 0.7, development)
  string-scrub (<= 0.0.5, >= 0.0.3)
  test-unit (~> 3.2, development)
  test-unit-rr (~> 1.0, development)
  timecop (~> 0.3, development)
  tzinfo (>= 1.0.0)
  tzinfo-data (>= 1.0.0)
  yajl-ruby (~> 1.0)
  Used by
    fluent-plugin-detect-exceptions-0.0.8 (fluentd (~> 0.10))
    fluent-plugin-kafka-0.6.6 (fluentd (< 2, >= 0.10.58))
    fluent-plugin-remote_syslog-1.0.0 (fluentd (>= 0))

Gem fluentd-1.1.0
  cool.io (< 2.0.0, >= 1.4.5)
  dig_rb (~> 1.0.0)
  flexmock (~> 2.0, development)
  http_parser.rb (< 0.7.0, >= 0.5.1)
  msgpack (< 2.0.0, >= 0.7.0)
  oj (~> 2.14, development)
  parallel_tests (~> 0.15.3, development)
  rake (~> 11.0, development)
  rr (~> 1.0, development)
  serverengine (< 3.0.0, >= 2.0.4)
  sigdump (~> 0.2.2)
  simplecov (~> 0.7, development)
  strptime (< 1.0.0, >= 0.2.2)
  test-unit (~> 3.2, development)
  test-unit-rr (~> 1.0, development)
  timecop (~> 0.3, development)
  tzinfo (~> 1.0)
  tzinfo-data (~> 1.0)
  yajl-ruby (~> 1.0)
  Used by
    fluent-plugin-elasticsearch-2.4.1 (fluentd (>= 0.14.8))
    fluent-plugin-kafka-0.6.6 (fluentd (< 2, >= 0.10.58))
    fluent-plugin-remote_syslog-1.0.0 (fluentd (>= 0))
igorpeshansky commented 6 years ago

Ok, looks like the newer version of fluentd is actually pulled in by fluent-plugin-elasticsearch, but it's this plugin that's holding on to the old (v0.12) version. The latter is #21, and is fixed by 0.0.9. Can you pull in that version of fluent-plugin-detect-exceptions and try again?

jushiyuan commented 6 years ago

@igorpeshansky It has a new error

  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event_router.rb:159:in `emit_events'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event_router.rb:96:in `emit_stream'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event_router.rb:87:in `emit'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/out_detect_exceptions.rb:97:in `block (2 levels) in process_record'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/exception_detector.rb:276:in `update_buffer'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/exception_detector.rb:226:in `push'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/out_detect_exceptions.rb:101:in `block in process_record'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/out_detect_exceptions.rb:133:in `synchronize'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/out_detect_exceptions.rb:88:in `process_record'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/out_detect_exceptions.rb:80:in `block in emit'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event.rb:193:in `block in each'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event.rb:192:in `each'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event.rb:192:in `each'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/out_detect_exceptions.rb:79:in `emit'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/compat/output.rb:164:in `process'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin/output.rb:760:in `emit_sync'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event_router.rb:159:in `emit_events'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event_router.rb:96:in `emit_stream'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event_router.rb:87:in `emit'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/out_detect_exceptions.rb:97:in `block (2 levels) in process_record'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/exception_detector.rb:276:in `update_buffer'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/exception_detector.rb:226:in `push'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/out_detect_exceptions.rb:101:in `block in process_record'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/out_detect_exceptions.rb:133:in `synchronize'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/out_detect_exceptions.rb:88:in `process_record'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/out_detect_exceptions.rb:80:in `block in emit'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event.rb:193:in `block in each'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event.rb:192:in `each'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event.rb:192:in `each'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/out_detect_exceptions.rb:79:in `emit'
  2018-01-20 09:02:02 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/compat/output.rb:164:in `process'
igorpeshansky commented 6 years ago

That's just the stack trace. Where's the error?

jushiyuan commented 6 years ago

@igorpeshansky My error log

5921843 [http-nio-8080-exec-167] 2018-01-16 17:40:40,345 ERROR c.***.controller.***Controller - error
org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:356)
        at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:815)
        at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:720)
        at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:391)
        at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:369)
        at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96)
        at com.***.controller.***Controller.getTileImg(***Controller.java:99)
        at com.***.controller.***Controller$$FastClassBySpringCGLIB$$d6e66a9c.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:747)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:101)
        at com.***.aspect.LogAspect.around(LogAspect.java:46)
        at sun.reflect.GeneratedMethodAccessor216.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:643)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:632)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)
        at com.***.controller.***Controller$$EnhancerBySpringCGLIB$$5c7aab02.getTileImg(<generated>)
        at sun.reflect.GeneratedMethodAccessor221.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:869)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:775)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:981)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:915)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:978)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:870)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:855)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.multipart.support.MultipartFilter.doFilterInternal(MultipartFilter.java:123)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:119)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:115)
        at org.springframework.boot.web.servlet.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59)
        at org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
        at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:134)
        at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
        at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:157)
        at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1267)
        at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:670)
        at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:450)
        at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:388)
        at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:644)
        at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:123)
        at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:235)
        at org.apache.coyote.Response.doWrite(Response.java:541)
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:351)
        ... 97 common frames omitted

Info log

491190720 [http-nio-8080-exec-183] 2018-01-22 08:28:29,222 INFO  c.t.interceptor.AllInterceptor - accessToken = ZgZ6zz1RM1PNaGDeuJjYLg==，userId = 10000977，clientType = 7，userType = 0
491190720 [http-nio-8080-exec-183] 2018-01-22 08:28:29,222 INFO  c.t.interceptor.AllInterceptor - redisKey = ACCESS_TOKEN_0_7_10000977
491190722 [http-nio-8080-exec-183] 2018-01-22 08:28:29,224 INFO  c.t.interceptor.AllInterceptor - head accessToken = ZgZ6zz1RM1PNaGDeuJjYLg==
491190722 [http-nio-8080-exec-183] 2018-01-22 08:28:29,224 INFO  c.t.interceptor.AllInterceptor - redis accessToken = ZgZ6zz1RM1PNaGDeuJjYLg==
491190722 [http-nio-8080-exec-183] 2018-01-22 08:28:29,224 INFO  com.***.aspect.LogAspect - getMedicalByClient|/mrs/mrtypes/2,3,4|GET|***|***|***|{"clienttype":"7","referer":"https://***.***.com/home","x-forwarded-proto":"https","clientversion":"0","accept-language":"zh-CN,zh;q=0.9","deviceos":"0","origin":"https://***.***.com","x-forwarded-port":"443","usertype":"0","x-forwarded-for":"***","phonenum":"***","userid":"10000977","accept":"application/json, text/plain, */*","token":"ZgZ6zz1RM1PNaGDeuJjYLg==","x-real-ip":"***","x-forwarded-host":"***.***.com","deviceosversion":"0","host":"***.***.com","x-original-uri":"/mrs/mrtypes/2,3,4","connection":"close","x-scheme":"https","accept-encoding":"gzip, deflate, br","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36"}|null|0|***

access log

getMedicalByClient|/mrs/mrtypes/2,3,4|GET|***|***|***|{"clienttype":"7","referer":"https://***.***.com/home","x-forwarded-proto":"https","clientversion":"0","accept-language":"zh-CN,zh;q=0.9","deviceos":"0","origin":"https://***.***.com","x-forwarded-port":"443","usertype":"0","x-forwarded-for":"***","phonenum":"***","userid":"10000977","accept":"application/json, text/plain, */*","token":"ZgZ6zz1RM1PNaGDeuJjYLg==","x-real-ip":"***","x-forwarded-host":"***.***.com","deviceosversion":"0","host":"***.***.com","x-original-uri":"/mrs/mrtypes/2,3,4","connection":"close","x-scheme":"https","accept-encoding":"gzip, deflate, br","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36"}|null|0|***
search4Involved|/process/dealtTasks|GET|***|***|***|{"clienttype":"7","referer":"https://***.***.com/login","x-forwarded-proto":"https","clientversion":"0","accept-language":"zh-CN,zh;q=0.9","deviceos":"0","origin":"https://***.***.com","x-forwarded-port":"443","usertype":"0","x-forwarded-for":"***","phonenum":"***","userid":"10000977","accept":"application/json, text/plain, */*","token":"ZgZ6zz1RM1PNaGDeuJjYLg==","x-real-ip":"***","x-forwarded-host":"***.***.com","deviceosversion":"0","host":"***.***.com","x-original-uri":"/process/dealtTasks?startTime=&endTime=&pgIndex=0&pgCnt=10&status=&hospitalId=&medicalType=&keyword=","connection":"close","x-scheme":"https","accept-encoding":"gzip, deflate, br","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36"}|{"medicalType":"","hospitalId":"","pgCnt":"10","startTime":"","endTime":"","keyword":"","pgIndex":"0","status":""}|0|***

Thanks.

igorpeshansky commented 6 years ago

@jushiyuan Sorry to have been unclear. Your earlier comment about the new error showed a Ruby exception trace, but not the first line, which has the actual exception info. I'd need that first line to figure out what's going on.

jushiyuan commented 6 years ago

@igorpeshansky I'm sorry to misunderstand what you mean. the logs:

2018-01-22 02:05:40 +0000 [error]: #0 unexpected error error_class=SystemStackError error="stack level too deep"
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin/filter_record_transformer.rb:309:in `rescue in expand'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin/filter_record_transformer.rb:298:in `expand'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin/filter_record_transformer.rb:146:in `block in expand_placeholders'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin/filter_record_transformer.rb:145:in `each_pair'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin/filter_record_transformer.rb:145:in `expand_placeholders'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin/filter_record_transformer.rb:135:in `reform'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin/filter_record_transformer.rb:101:in `block in filter_stream'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event.rb:193:in `block in each'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event.rb:192:in `each'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event.rb:192:in `each'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin/filter_record_transformer.rb:96:in `filter_stream'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event_router.rb:177:in `block in filter_stream'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event_router.rb:177:in `each'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event_router.rb:177:in `reduce'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event_router.rb:177:in `filter_stream'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event_router.rb:158:in `emit_events'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event_router.rb:96:in `emit_stream'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/event_router.rb:87:in `emit'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/out_detect_exceptions.rb:97:in `block (2 levels) in process_record'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/exception_detector.rb:276:in `update_buffer'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/exception_detector.rb:226:in `push'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/out_detect_exceptions.rb:101:in `block in process_record'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/out_detect_exceptions.rb:133:in `synchronize'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/out_detect_exceptions.rb:88:in `process_record'
  2018-01-22 02:05:40 +0000 [error]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.9/lib/fluent/plugin/out_detect_exceptions.rb:80:in `block in emit'