search

envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
24.32k stars 4.7k forks source link

thrift response error: unknown field type 108 using thrift_proxy #7667

Open happydenn opened 5 years ago

happydenn commented 5 years ago

Title: thrift response error: unknown field type 108 using thrift_proxy

Description: We have a service with Thrift RPC interface. When using thrift_proxy while most requests were successful. There were numerous errors with thrift response error: unknown field type 108 and the field type number has several possible values such as 108, 111, -105 etc. There were no such errors when connecting to the service directly.

Config:

admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address:
      protocol: TCP
      address: 0.0.0.0
      port_value: 9901
static_resources:
  listeners:
  - address:
      socket_address:
        protocol: TCP
        address: 0.0.0.0
        port_value: 12000
    filter_chains:
    - filters:
      - name: envoy.filters.network.thrift_proxy
        config:
          transport: AUTO_TRANSPORT
          protocol: BINARY
          stat_prefix: 'thrift_svc'
          thrift_filters:
          - name: envoy.filters.thrift.router
          route_config:
            routes:
            - match:
                method_name: ''
              route:
                cluster: thrift_svc
  clusters:
  - name: thrift_svc
    connect_timeout: 0.25s
    type: LOGICAL_DNS
    lb_policy: ROUND_ROBIN
    extension_protocol_options:
      envoy.filters.network.thrift_proxy:
        transport: FRAMED
        protocol: TWITTER
    health_checks:
    - interval: 10s
      timeout: 3s
      unhealthy_threshold: 2
      healthy_threshold: 1
      tcp_health_check: {}
    load_assignment:
      cluster_name: thrift_svc
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 'thrift-svc'
                port_value: 5908

Logs:

[2019-07-20 20:09:48.590][11][error][thrift] [source/extensions/filters/network/thrift_proxy/conn_manager.cc:653] [C11] thrift response error: unknown field type 108
[2019-07-20 20:11:42.505][16][error][thrift] [source/extensions/filters/network/thrift_proxy/conn_manager.cc:653] [C64] thrift response error: unknown field type 108
[2019-07-20 20:11:44.725][16][error][thrift] [source/extensions/filters/network/thrift_proxy/conn_manager.cc:653] [C66] thrift response error: unknown field type 111
[2019-07-20 20:11:56.267][16][error][thrift] [source/extensions/filters/network/thrift_proxy/conn_manager.cc:653] [C76] thrift response error: unknown field type 111
[2019-07-20 20:16:51.869][13][error][thrift] [source/extensions/filters/network/thrift_proxy/conn_manager.cc:653] [C116] thrift response error: unknown field type 108
mattklein123 commented 4 years ago

cc @zuercher

zyfjeff commented 4 years ago

@happydenn How to reproduce? I can solve this problem.

happydenn commented 4 years ago

Sorry for late reply. Calling a Thrift service with following IDL. Server is implemented with Scala and Finagle:

service StoreLocatorService {
  StoreLocatorResult findStores(
    1: store.Location coordinates,
    2: double radius,
    3: Filter filter,
    4: i32 offset = 0,
    5: i32 size = 30
    ) throws(1: StoreLocatorException ex)
}

When running behind envoy, calling findStores results in about 10-15% error. The service is running on production, using Linkerd 1.6 as the proxy or connect directly work without any errors. I have limited experience with Thrift as I'm only implementing the client side. Client is written in Go. Client code is generated by Apache Thrift compiler. Let me know if you need any additional information. Thanks for the help.

zuercher commented 4 years ago

I expect the problem is that the framing is off and the codec is looking at bytes that are not actually encoding thrift fields. If you could capture a network trace of the exchange where the error happens it would probably be useful.

happydenn commented 4 years ago

Turned on trace log and got this when the error occurs:

[2019-08-15 01:21:38.019][11][debug][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:367] thrift: 94 bytes available
[2019-08-15 01:21:38.019][11][debug][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:388] thrift: framed(auto) transport started
[2019-08-15 01:21:38.020][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/conn_manager.cc:164] new decoder filter
[2019-08-15 01:21:38.020][11][debug][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:422] thrift: protocol binary, state MessageBegin, 90 bytes available
[2019-08-15 01:21:38.020][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state MessageBegin, 90 bytes available
[2019-08-15 01:21:38.020][11][debug][thrift] [source/extensions/filters/network/thrift_proxy/router/router_impl.cc:227] [C1][S6691463599374253655] cluster 'thrift_svc' match for method 'findStores'
[2019-08-15 01:21:38.020][11][debug][thrift] [source/extensions/filters/network/thrift_proxy/router/router_impl.cc:260] [C1][S6691463599374253655] router decoding request
[2019-08-15 01:21:38.021][11][debug][pool] [source/common/tcp/conn_pool.cc:90] [C2] using existing connection
[2019-08-15 01:21:38.021][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state StructBegin, 68 bytes available
[2019-08-15 01:21:38.021][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldBegin, 68 bytes available
[2019-08-15 01:21:38.021][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldValue, 65 bytes available
[2019-08-15 01:21:38.021][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state StructBegin, 65 bytes available
[2019-08-15 01:21:38.021][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldBegin, 65 bytes available
[2019-08-15 01:21:38.021][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldValue, 62 bytes available
[2019-08-15 01:21:38.021][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldEnd, 54 bytes available
[2019-08-15 01:21:38.021][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldBegin, 54 bytes available
[2019-08-15 01:21:38.021][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldValue, 51 bytes available
[2019-08-15 01:21:38.022][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldEnd, 43 bytes available
[2019-08-15 01:21:38.022][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldBegin, 43 bytes available
[2019-08-15 01:21:38.022][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state StructEnd, 42 bytes available
[2019-08-15 01:21:38.022][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldEnd, 42 bytes available
[2019-08-15 01:21:38.022][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldBegin, 42 bytes available
[2019-08-15 01:21:38.022][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldValue, 39 bytes available
[2019-08-15 01:21:38.022][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldEnd, 31 bytes available
[2019-08-15 01:21:38.022][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldBegin, 31 bytes available
[2019-08-15 01:21:38.022][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldValue, 28 bytes available
[2019-08-15 01:21:38.022][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state StructBegin, 28 bytes available
[2019-08-15 01:21:38.022][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldBegin, 28 bytes available
[2019-08-15 01:21:38.022][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldValue, 25 bytes available
[2019-08-15 01:21:38.022][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldEnd, 24 bytes available
[2019-08-15 01:21:38.023][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldBegin, 24 bytes available
[2019-08-15 01:21:38.023][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldValue, 21 bytes available
[2019-08-15 01:21:38.023][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state SetBegin, 21 bytes available
[2019-08-15 01:21:38.023][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state SetValue, 16 bytes available
[2019-08-15 01:21:38.023][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state SetEnd, 16 bytes available
[2019-08-15 01:21:38.023][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldEnd, 16 bytes available
[2019-08-15 01:21:38.023][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldBegin, 16 bytes available
[2019-08-15 01:21:38.023][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state StructEnd, 15 bytes available
[2019-08-15 01:21:38.023][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldEnd, 15 bytes available
[2019-08-15 01:21:38.023][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldBegin, 15 bytes available
[2019-08-15 01:21:38.023][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldValue, 12 bytes available
[2019-08-15 01:21:38.023][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldEnd, 8 bytes available
[2019-08-15 01:21:38.023][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldBegin, 8 bytes available
[2019-08-15 01:21:38.023][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldValue, 5 bytes available
[2019-08-15 01:21:38.023][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldEnd, 1 bytes available
[2019-08-15 01:21:38.023][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldBegin, 1 bytes available
[2019-08-15 01:21:38.024][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state StructEnd, 0 bytes available
[2019-08-15 01:21:38.024][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state MessageEnd, 0 bytes available
[2019-08-15 01:21:38.024][11][trace][connection] [source/common/network/connection_impl.cc:394] [C2] writing 117 bytes, end_stream false
[2019-08-15 01:21:38.024][11][debug][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:446] thrift: framed(auto) transport ended
[2019-08-15 01:21:38.024][11][trace][connection] [source/common/network/connection_impl.cc:458] [C2] socket event: 2
[2019-08-15 01:21:38.024][11][trace][connection] [source/common/network/connection_impl.cc:543] [C2] write ready
[2019-08-15 01:21:38.024][11][trace][connection] [source/common/network/raw_buffer_socket.cc:66] [C2] write returns: 117
[2019-08-15 01:21:38.078][11][trace][connection] [source/common/network/connection_impl.cc:458] [C2] socket event: 3
[2019-08-15 01:21:38.078][11][trace][connection] [source/common/network/connection_impl.cc:543] [C2] write ready
[2019-08-15 01:21:38.078][11][trace][connection] [source/common/network/connection_impl.cc:496] [C2] read ready
[2019-08-15 01:21:38.078][11][trace][connection] [source/common/network/raw_buffer_socket.cc:23] [C2] read returns: 5720
[2019-08-15 01:21:38.078][11][trace][connection] [source/common/network/raw_buffer_socket.cc:37] [C2] read error: Resource temporarily unavailable
[2019-08-15 01:21:38.078][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/router/router_impl.cc:299] [C1][S6691463599374253655] reading response: 5720 bytes
[2019-08-15 01:21:38.078][11][debug][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:367] thrift: 5720 bytes available
[2019-08-15 01:21:38.078][11][debug][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:388] thrift: framed transport started
[2019-08-15 01:21:38.078][11][debug][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:422] thrift: protocol twitter, state MessageBegin, 5716 bytes available
[2019-08-15 01:21:38.078][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state MessageBegin, 5716 bytes available
[2019-08-15 01:21:38.078][11][debug][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:367] thrift: 5716 bytes available
[2019-08-15 01:21:38.078][11][debug][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:388] thrift: unframed transport started
[2019-08-15 01:21:38.078][11][debug][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:422] thrift: protocol binary, state MessageBegin, 5716 bytes available
[2019-08-15 01:21:38.078][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state MessageBegin, 5716 bytes available
[2019-08-15 01:21:38.079][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state StructBegin, 5716 bytes available
[2019-08-15 01:21:38.079][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldBegin, 5716 bytes available
[2019-08-15 01:21:38.079][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state StructEnd, 5715 bytes available
[2019-08-15 01:21:38.079][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state MessageEnd, 5715 bytes available
[2019-08-15 01:21:38.079][11][debug][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:446] thrift: unframed transport ended
[2019-08-15 01:21:38.079][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state StructBegin, 5693 bytes available
[2019-08-15 01:21:38.079][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldBegin, 5693 bytes available
[2019-08-15 01:21:38.079][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldValue, 5690 bytes available
[2019-08-15 01:21:38.079][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state StructBegin, 5690 bytes available
[2019-08-15 01:21:38.079][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldBegin, 5690 bytes available
[2019-08-15 01:21:38.079][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldValue, 5687 bytes available
[2019-08-15 01:21:38.079][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state ListBegin, 5687 bytes available
[2019-08-15 01:21:38.079][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state ListValue, 5682 bytes available
[2019-08-15 01:21:38.079][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state StructBegin, 5682 bytes available

(... many similar lines removed ...)

[2019-08-15 01:21:38.273][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state ListBegin, 71 bytes available
[2019-08-15 01:21:38.273][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state ListValue, 66 bytes available
[2019-08-15 01:21:38.273][11][debug][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:426] thrift: wait for data
[2019-08-15 01:21:38.273][11][trace][connection] [source/common/network/connection_impl.cc:458] [C2] socket event: 3
[2019-08-15 01:21:38.273][11][trace][connection] [source/common/network/connection_impl.cc:543] [C2] write ready
[2019-08-15 01:21:38.273][11][trace][connection] [source/common/network/connection_impl.cc:496] [C2] read ready
[2019-08-15 01:21:38.273][11][trace][connection] [source/common/network/raw_buffer_socket.cc:23] [C2] read returns: 16384
[2019-08-15 01:21:38.274][11][trace][connection] [source/common/network/raw_buffer_socket.cc:23] [C2] read returns: 1717
[2019-08-15 01:21:38.274][11][trace][connection] [source/common/network/raw_buffer_socket.cc:37] [C2] read error: Resource temporarily unavailable
[2019-08-15 01:21:38.274][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/router/router_impl.cc:299] [C1][S6691463599374253655] reading response: 18101 bytes
[2019-08-15 01:21:38.274][11][debug][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:367] thrift: 18167 bytes available
[2019-08-15 01:21:38.274][11][debug][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:422] thrift: protocol twitter, state ListValue, 18167 bytes available
[2019-08-15 01:21:38.274][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state ListValue, 18167 bytes available
[2019-08-15 01:21:38.275][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state ListValue, 18048 bytes available
[2019-08-15 01:21:38.275][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state ListEnd, 18048 bytes available
[2019-08-15 01:21:38.275][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldEnd, 18048 bytes available
[2019-08-15 01:21:38.275][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldBegin, 18048 bytes available
[2019-08-15 01:21:38.275][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state StructEnd, 18047 bytes available
[2019-08-15 01:21:38.275][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldEnd, 18047 bytes available
[2019-08-15 01:21:38.275][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldBegin, 18047 bytes available
[2019-08-15 01:21:38.275][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state StructEnd, 18046 bytes available
[2019-08-15 01:21:38.275][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state ListValue, 18046 bytes available
[2019-08-15 01:21:38.275][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state StructBegin, 18046 bytes available
[2019-08-15 01:21:38.275][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldBegin, 18046 bytes available
[2019-08-15 01:21:38.276][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state StructEnd, 18045 bytes available
[2019-08-15 01:21:38.276][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state ListValue, 18045 bytes available
[2019-08-15 01:21:38.276][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state StructBegin, 18045 bytes available
[2019-08-15 01:21:38.276][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldBegin, 18045 bytes available
[2019-08-15 01:21:38.276][11][trace][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldValue, 18042 bytes available
[2019-08-15 01:21:38.277][11][error][thrift] [source/extensions/filters/network/thrift_proxy/conn_manager.cc:653] [C1] thrift response error: unknown field type 115
[2019-08-15 01:21:38.277][11][trace][connection] [source/common/network/connection_impl.cc:394] [C1] writing 63 bytes, end_stream true
[2019-08-15 01:21:38.277][11][debug][connection] [source/common/network/connection_impl.cc:101] [C1] closing data_to_write=63 type=0
[2019-08-15 01:21:38.277][11][debug][thrift] [source/extensions/filters/network/thrift_proxy/router/router_impl.cc:315] [C1][S6691463599374253655] upstream reset
[2019-08-15 01:21:38.277][11][debug][connection] [source/common/network/connection_impl.cc:101] [C2] closing data_to_write=0 type=1
[2019-08-15 01:21:38.277][11][debug][connection] [source/common/network/connection_impl.cc:190] [C2] closing socket: 1
[2019-08-15 01:21:38.277][11][debug][pool] [source/common/tcp/conn_pool.cc:123] [C2] client disconnected
[2019-08-15 01:21:38.278][11][trace][main] [source/common/event/dispatcher_impl.cc:158] item added to deferred deletion list (size=1)
[2019-08-15 01:21:38.278][11][debug][connection] [source/common/network/connection_impl.cc:101] [C1] closing data_to_write=63 type=1
[2019-08-15 01:21:38.280][11][trace][connection] [source/common/network/raw_buffer_socket.cc:66] [C1] write returns: 63
[2019-08-15 01:21:38.281][11][debug][connection] [source/common/network/connection_impl.cc:190] [C1] closing socket: 1
[2019-08-15 01:21:38.283][11][debug][thrift] [source/extensions/filters/network/thrift_proxy/conn_manager.cc:141] [C1] local close with active request

Hope this helps. It seems to me that this error mostly occurs when envoy receives additional data for the same response from the upstream server.

zuercher commented 4 years ago

If the inbound and outbound connections are all finagle-thrift, I'd recommend configuring the thrift proxy not to auto-detect the transport/protocol (should be framed/ttwitter) and setting the cluster's extension_protocol_options to contain a envoy.filters.network.thrift_proxy entry with a ThriftProtocolOptions struct that also sets the upstreams to use framed/ttwitter. It looks like something's causing the decoder the get confused.

happydenn commented 4 years ago

I've tried with

filter_chains:
- filters:
  - name: envoy.filters.network.thrift_proxy
    config:
      transport: FRAMED
      protocol: TWITTER

and also

clusters:
- name: thrift_svc
  connect_timeout: 0.25s
  type: LOGICAL_DNS
  lb_policy: ROUND_ROBIN
  extension_protocol_options:
    envoy.filters.network.thrift_proxy:
      transport: FRAMED
      protocol: TWITTER

still getting the same error though.

Servers are built with Finagle while we have clients in multiple languages such as JavaScript/Node.js, Go, Python and Java

zuercher commented 4 years ago

If you're not using the twitter protocol from the clients, then you can probably just use the binary protocol, and eliminate some complexity. Also, can you share the IDL for the StoreLocatorResult type? Because that seems to be where the problem occurs.

happydenn commented 4 years ago

Thanks for your help. I've attached some of the related IDLs. If you need more just let me know :) 

typedef list<locator_data.StoreDetail> StoreList
struct StoreLocatorResult {
    1: required StoreList stores;
}

struct StoreDetail {
    1: required store.Store store;
    2: list<Coupon> coupon;
    3: optional Rating rating;
    4: list<RelatedArticle> relatedArticles;
    5: list<Review> reviews;
    6: list<TimeLine> timeLine;
}

struct Store {
    1: required i64 spotId;
    2: optional double distance;
    3: required Location location;
    4: required StoreType category;
    5: required string name;
    6: optional string address;
    7: list<string> images;
    8: optional string thumbnail;
    9: optional timestamp lastModifiedDate;
   10: optional i32 minimumCharge;
   11: optional string businessInfo;
   12: optional string officialWeb;
   13: optional string officialWeb2;
   14: set<Payment> payments;
   15: optional string phone;
   16: set<Facility> facilities;
   17: optional ServiceHours serviceHours;
   18: optional StoreStatus storeStatus;
   19: optional bool acceptReservation;
   20: optional string tableTimeLimit;
   21: optional string lineId;
   22: optional i32 botId;
   23: optional string botMid;
   24: optional bool twentyFourSeven = false;
   25: optional i32 seats;
   26: list<OpeningInterval> openingHours;
   27: list<image.ImageDetail> imageDetails = [];
   28: optional image.ImageDetail thumbnailDetails;
   29: list<Tag> tags = [];
}

struct Location {
    1: required double lat;
    2: required double lon;
}

enum StoreType {
    HEALTH_AND_BEAUTY,
    BEAUTY_AND_SALON,
    FASHION_AND_BOUTIQUE,
    REAL_ESTATE,
    RETAIL,
    TRAVEL,
    FOOD_AND_BEVERAGE,
    TELECOMMUNICATION_AND_SOFTWARE,
    FINANCE_AND_INSURANCE,
    TRANSPORTATION,
    LIFE_SERVICE,
    MEDIA,
    MEDICAL_SERVICES,
    ENTERTAINMENT,
    EDUCATION,
    BEVERAGE
}

typedef i64 (js.type = 'Date') timestamp

enum Payment {
    CASH,
    CREDIT_CARD,
    LINE_PAY,
    IPASS
}

struct Coupon {
    1: required string title;
    3: store.timestamp endTime;
    4: optional string image;
    5: optional string url;
    6: optional double originPrice;
    7: optional double specialPrice;
    8: optional string vendorId;
}

struct Rating {
    1: double avgScore;
    2: i32 totalVoterNumber;
    3: VoterDetails voterDetails;
}

struct VoterDetails {
    1: i32 one;
    2: i32 two;
    3: i32 three;
    4: i32 four;
    5: i32 five;
}
ngtuna commented 3 years ago

@happydenn How did you end up with this error ?

I am having different issue but still with Thrift/Finagle. My client and server are both ThriftMux/Finagle, and envoy thrift filter is setting up as same as yours. And envoy stucks at thrift decoding with below error:

[2021-03-09 09:48:31.941][17][debug][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:398] thrift: 19 bytes available
[2021-03-09 09:48:31.941][17][debug][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:419] thrift: framed transport started
[2021-03-09 09:48:31.941][17][debug][thrift] [source/extensions/filters/network/thrift_proxy/decoder.cc:453] thrift: protocol binary, state MessageBegin, 15 bytes available
[2021-03-09 09:48:31.942][17][error][thrift] [source/extensions/filters/network/thrift_proxy/conn_manager.cc:78] [C20] thrift error: invalid binary protocol version 0x7f00 != 0x8001