zeek / spicy

C++ parser generator for dissecting protocols & files.
https://docs.zeek.org/projects/spicy
Other
246 stars 35 forks source link

compile error when using mime_header_list type. #335

Closed buboi closed 4 years ago

buboi commented 4 years ago

http.evt doesn’t have http_all_headers event, could you give us a sample, how to handle type mime_header_list?

I tried this in rtsp protocol parser.

zeek type:
type mime_header_rec: record {
    name: string;   ##< The header name.
    value: string;  ##< The header value.
};
type mime_header_list: table[count] of mime_header_rec;

I define correspond spicy type:

var header_list: map< uint64, tuple<bytes,bytes> >;

run: spicyz -D zeek rtsp.evt rtsp.spicy got two errors:

[debug/zeek] Beginning pre-script initialization
[debug/zeek] Done with pre-script initialization
[debug/zeek] Beginning post-script initialization
[debug/zeek] Initializing driver
[debug/zeek] Loading input file “rtsp.evt”
[debug/zeek] Loading EVT file “/Users/buboi/works/wudun/test/netprobe/sensor/plugins/spicy/rtsp/src/rtsp.evt”
[debug/zeek] Loading events from /Users/buboi/works/wudun/test/netprobe/sensor/plugins/spicy/rtsp/src/rtsp.evt
[debug/zeek]  Got protocol analyzer definition for spicy_RTSP
[debug/zeek]  Got module Zeek_RTSP to import
[debug/zeek]  Got event definition for rtsp_request
[debug/zeek]  Got event definition for rtsp_reply
[debug/zeek]  Got event definition for rtsp_header
[debug/zeek]  Got event definition for rtsp_all_headers
[debug/zeek]  Got event definition for rtsp_sdp
[debug/zeek]  Got event definition for rtsp_body
[debug/zeek] Loading input file “rtsp.spicy”
[debug/zeek] Loading Spicy file “/Users/buboi/works/wudun/test/netprobe/sensor/plugins/spicy/rtsp/src/rtsp.spicy”
[debug/zeek] Loading input file “zeek_rtsp.spicy”
[debug/zeek] Loading Spicy file “/Users/buboi/works/wudun/test/netprobe/sensor/plugins/spicy/rtsp/src/zeek_rtsp.spicy”
[debug/zeek] Compiling input files
[debug/zeek]  Running Spicy driver
[debug/zeek]   Got unit type ‘RTSP::Version’
[debug/zeek]   Got unit type ‘RTSP::RequestLine’
[debug/zeek]   Got unit type ‘RTSP::Header’
[debug/zeek]   Got unit type ‘RTSP::Message’
[debug/zeek]   Got unit type ‘RTSP::Body’
[debug/zeek]   Got unit type ‘RTSP::Request’
[debug/zeek]   Got unit type ‘RTSP::ReplyLine’
[debug/zeek]   Got unit type ‘RTSP::Reply’
[debug/zeek]  Adding protocol analyzer ‘spicy_RTSP’
[debug/zeek]  Adding Spicy hook ‘RTSP::RequestLine::0x25_done’ for event rtsp_request
[debug/zeek]  Adding Spicy hook ‘RTSP::ReplyLine::0x25_done’ for event rtsp_reply
[debug/zeek]  Adding Spicy hook ‘RTSP::Header::0x25_done’ for event rtsp_header
[debug/zeek]  Adding Spicy hook ‘RTSP::Message::end_of_hdr’ for event rtsp_all_headers
[debug/zeek]  Adding Spicy hook ‘RTSP::Message::0x25_done’ for event rtsp_sdp
[debug/zeek]  Adding Spicy hook ‘RTSP::Message::0x25_done’ for event rtsp_body
In file included from RTSP.cc:6:
In file included from /Users/buboi/zeek/aux/spicy/hilti/include/hilti/rt/libhilti.h:5:
In file included from /Users/buboi/zeek/aux/spicy/hilti/include/hilti/rt/configuration.h:8:
In file included from /Users/buboi/zeek/aux/spicy/hilti/include/hilti/rt/util.h:32:
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/3rdparty/tinyformat/tinyformat.h:351:13: error: call to function ‘operator<<’ that is neither visible in the template definition nor found by argument-dependent lookup
    out << value;
      ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/3rdparty/tinyformat/tinyformat.h:537:13: note: in instantiation of function template specialization ‘tinyformat::formatValue<std::__1::tuple<hilti::rt::Bytes, hilti::rt::Bytes> >’ requested here
      formatValue(out, fmtBegin, fmtEnd, ntrunc, *static_cast<const T*>(value));
      ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/3rdparty/tinyformat/tinyformat.h:513:27: note: in instantiation of function template specialization ‘tinyformat::detail::FormatArg::formatImpl<std::__1::tuple<hilti::rt::Bytes, hilti::rt::Bytes> >’ requested here
      m_formatImpl(&formatImpl<T>),
             ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/3rdparty/tinyformat/tinyformat.h:995:32: note: in instantiation of function template specialization ‘tinyformat::detail::FormatArg::FormatArg<std::__1::tuple<hilti::rt::Bytes, hilti::rt::Bytes> >’ requested here
      m_formatterStore { FormatArg(args)... }
                ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/3rdparty/tinyformat/tinyformat.h:1044:12: note: in instantiation of function template specialization ‘tinyformat::detail::FormatListN<2>::FormatListN<SafeInt<unsigned long long, hilti::rt::integer::detail::SafeIntException>, std::__1::tuple<hilti::rt::Bytes, hilti::rt::Bytes> >’ requested here
  return detail::FormatListN<sizeof...(args)>(args...);
      ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/3rdparty/tinyformat/tinyformat.h:1080:23: note: in instantiation of function template specialization ‘tinyformat::makeFormatList<SafeInt<unsigned long long, hilti::rt::integer::detail::SafeIntException>, std::__1::tuple<hilti::rt::Bytes, hilti::rt::Bytes> >’ requested here
  vformat(out, fmt, makeFormatList(args...));
           ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/3rdparty/tinyformat/tinyformat.h:1089:5: note: (skipping 3 contexts in backtrace; use -ftemplate-backtrace-limit=0 to see all)
  format(oss, fmt, args...);
  ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/rt/extension-points.h:31:12: note: in instantiation of function template specialization ‘hilti::rt::detail::adl::to_string<SafeInt<unsigned long long, hilti::rt::integer::detail::SafeIntException>, std::__1::tuple<hilti::rt::Bytes, hilti::rt::Bytes> >’ requested here
  return to_string(x, detail::adl::tag{});
      ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/rt/types/struct.h:45:47: note: in instantiation of function template specialization ‘hilti::rt::to_string<hilti::rt::Map<SafeInt<unsigned long long, hilti::rt::integer::detail::SafeIntException>, std::__1::tuple<hilti::rt::Bytes, hilti::rt::Bytes> > >’ requested here
    fields += fmt(“$%s=%s”, k, hilti::rt::to_string(v));
                       ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/rt/types/struct.h:48:7: note: in instantiation of function template specialization ‘__hlt::RTSP::Message::__visit<(lambda at /Users/buboi/zeek/aux/spicy/hilti/include/hilti/rt/types/struct.h:39:23)>’ requested here
  x.__visit(render_one);
   ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/rt/extension-points.h:31:12: note: in instantiation of function template specialization ‘hilti::rt::detail::adl::to_string<__hlt::RTSP::Message, nullptr>’ requested here
  return to_string(x, detail::adl::tag{});
      ^
RTSP.cc:2135:120: note: in instantiation of function template specialization ‘hilti::rt::to_string<__hlt::RTSP::Message>’ requested here
static auto __hlt::RTSP::operator<<(std::ostream& o, const __hlt::RTSP::Message& x) -> std::ostream& { o << hilti::rt::to_string(x); return o; }
                                                            ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/rt/types/tuple.h:27:22: note: ‘operator<<’ should be declared prior to the call site or in an associated namespace of one of its arguments
inline std::ostream& operator<<(std::ostream& out, const T& x) {
           ^
In file included from RTSP.cc:6:
In file included from /Users/buboi/zeek/aux/spicy/hilti/include/hilti/rt/libhilti.h:5:
In file included from /Users/buboi/zeek/aux/spicy/hilti/include/hilti/rt/configuration.h:8:
In file included from /Users/buboi/zeek/aux/spicy/hilti/include/hilti/rt/util.h:32:
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/3rdparty/tinyformat/tinyformat.h:283:9: error: call to function ‘operator<<’ that is neither visible in the template definition nor found by argument-dependent lookup
  tmp << value;
    ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/3rdparty/tinyformat/tinyformat.h:348:17: note: in instantiation of function template specialization ‘tinyformat::detail::formatTruncated<std::__1::tuple<hilti::rt::Bytes, hilti::rt::Bytes> >’ requested here
    detail::formatTruncated(out, value, ntrunc);
        ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/3rdparty/tinyformat/tinyformat.h:537:13: note: in instantiation of function template specialization ‘tinyformat::formatValue<std::__1::tuple<hilti::rt::Bytes, hilti::rt::Bytes> >’ requested here
      formatValue(out, fmtBegin, fmtEnd, ntrunc, *static_cast<const T*>(value));
      ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/3rdparty/tinyformat/tinyformat.h:513:27: note: in instantiation of function template specialization ‘tinyformat::detail::FormatArg::formatImpl<std::__1::tuple<hilti::rt::Bytes, hilti::rt::Bytes> >’ requested here
      m_formatImpl(&formatImpl<T>),
             ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/3rdparty/tinyformat/tinyformat.h:995:32: note: in instantiation of function template specialization ‘tinyformat::detail::FormatArg::FormatArg<std::__1::tuple<hilti::rt::Bytes, hilti::rt::Bytes> >’ requested here
      m_formatterStore { FormatArg(args)... }
                ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/3rdparty/tinyformat/tinyformat.h:1044:12: note: in instantiation of function template specialization ‘tinyformat::detail::FormatListN<2>::FormatListN<SafeInt<unsigned long long, hilti::rt::integer::detail::SafeIntException>, std::__1::tuple<hilti::rt::Bytes, hilti::rt::Bytes> >’ requested here
  return detail::FormatListN<sizeof...(args)>(args...);
      ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/3rdparty/tinyformat/tinyformat.h:1080:23: note: (skipping 4 contexts in backtrace; use -ftemplate-backtrace-limit=0 to see all)
  vformat(out, fmt, makeFormatList(args...));
           ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/rt/extension-points.h:31:12: note: in instantiation of function template specialization ‘hilti::rt::detail::adl::to_string<SafeInt<unsigned long long, hilti::rt::integer::detail::SafeIntException>, std::__1::tuple<hilti::rt::Bytes, hilti::rt::Bytes> >’ requested here
  return to_string(x, detail::adl::tag{});
      ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/rt/types/struct.h:45:47: note: in instantiation of function template specialization ‘hilti::rt::to_string<hilti::rt::Map<SafeInt<unsigned long long, hilti::rt::integer::detail::SafeIntException>, std::__1::tuple<hilti::rt::Bytes, hilti::rt::Bytes> > >’ requested here
    fields += fmt(“$%s=%s”, k, hilti::rt::to_string(v));
                       ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/rt/types/struct.h:48:7: note: in instantiation of function template specialization ‘__hlt::RTSP::Message::__visit<(lambda at /Users/buboi/zeek/aux/spicy/hilti/include/hilti/rt/types/struct.h:39:23)>’ requested here
  x.__visit(render_one);
   ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/rt/extension-points.h:31:12: note: in instantiation of function template specialization ‘hilti::rt::detail::adl::to_string<__hlt::RTSP::Message, nullptr>’ requested here
  return to_string(x, detail::adl::tag{});
      ^
RTSP.cc:2135:120: note: in instantiation of function template specialization ‘hilti::rt::to_string<__hlt::RTSP::Message>’ requested here
static auto __hlt::RTSP::operator<<(std::ostream& o, const __hlt::RTSP::Message& x) -> std::ostream& { o << hilti::rt::to_string(x); return o; }
                                                            ^
/Users/buboi/zeek/aux/spicy/hilti/include/hilti/rt/types/tuple.h:27:22: note: ‘operator<<’ should be declared prior to the call site or in an associated namespace of one of its arguments
inline std::ostream& operator<<(std::ostream& out, const T& x) {
           ^
2 errors generated.
[error] jit: failed to execute compilation action.
[error] jit: failed to compile C++ code unit RTSP to bitcode
[error] <Spicy Plugin for Zeek>: JIT compilation failed
rsmmr commented 4 years ago

So turns out support for sending Spicy maps to Zeek wasn't implemented yet. I just pushed a fix for that to #339. With that, I can make http_all_headers work per the following diff:

diff --git a/spicy/lib/protocols/http.spicy b/spicy/lib/protocols/http.spicy
index 4655028..143535b 100644
--- a/spicy/lib/protocols/http.spicy
+++ b/spicy/lib/protocols/http.spicy
@@ -79,6 +79,7 @@ type Message = unit(body_default: bool, is_request: bool) {
     var delivery_mode: DeliveryMode = DeliveryMode::EndOfData;
     var multipart_boundary: bytes &optional;
     var transfer_encoding: bytes &optional;
+    var header_list: map<uint64, tuple<bytes,bytes>>;

     on %init {
         self.has_body = body_default;
@@ -134,6 +135,11 @@ type Header = unit(inout msg: Message) {
             msg.content_type_parameter = ct[1].strip();
             }
         }
+
+    on %done {
+        msg.header_list[|msg.header_list|] = (self.name, self.content);
+    }
+
 };

 type Body = unit(inout msg: Message, delivery_mode: DeliveryMode, is_request: bool) {
diff --git a/zeek/plugin/lib/protocols/http.evt b/zeek/plugin/lib/protocols/http.evt
index a3ae3e5..a336c24 100644
--- a/zeek/plugin/lib/protocols/http.evt
+++ b/zeek/plugin/lib/protocols/http.evt
@@ -10,6 +10,7 @@ import Zeek_HTTP;

 on HTTP::Message::%init        -> event http_begin_entity($conn, $is_orig) &priority=5;
 on HTTP::Message               -> event http_end_entity($conn, $is_orig) &priority=5;
+on HTTP::Message               -> event http_all_headers($conn, $is_orig, self.header_list) &priority=5;
 on HTTP::Message::%error       -> event http_end_entity($conn, $is_orig) &priority=5;

 on HTTP::Message               -> event http_message_done($conn, $is_orig, Zeek_HTTP::create_http_message_stats(self));

(I may commit this later as well)

rsmmr commented 4 years ago

As you had asked for an example, I added a different way of doing this as well: #346 implements http_all_headers() completely on the plugin's side, with no changes to http.spicy.