Would you provide examples of tcp stream or http?

[yangc91]

Hi，when http data is long, I have try a lot but still can not combination tcppacket to get full http data，would you provide examples，thanks！

[ZaNkss]

I want to do this too.But my code can not work.

[ZaNkss]

是要自己拼TCP包吗，我的诉求是解析出请求的方法（Get/Post之类的）、请求体、状态码等这些http的header

[valsong]

I want to do this too.But my code can not work.

I not sure it is correct.

@NoArgsConstructor
@AllArgsConstructor
@Data
@Builder
public class HostPort {

    private String host;

    private Integer port;

}

@NoArgsConstructor
@AllArgsConstructor
@Data
@Builder
public class HttpPacket {

    private long arrivalTime;

    private String destIp;

    private int destPort;

    private Buffer payload;

}

public class HttpPacketParser {

    public static final String URL_SEPARATOR = " ";

    public static List<HttpPacket> parse(File file) throws IOException {
        return parse(Pcap.openStream(file), null);
    }

    /**
     * @param file      pcap 文件
     * @param hostPorts 根据host和port进行过滤
     * @return
     * @throws IOException
     */
    public static List<HttpPacket> parse(File file, List<HostPort> hostPorts) throws IOException {
        return parse(Pcap.openStream(file), hostPorts);
    }

    /**
     * @param pcap      pcap
     * @param hostPorts 根据host和port进行过滤
     * @return
     * @throws IOException
     */
    public static List<HttpPacket> parse(Pcap pcap, List<HostPort> hostPorts) throws IOException {

        List<HttpPacket> httpPackets = new LinkedList<>();

        Map<Long, HttpPacket> halfTcpPacketMap = new HashMap<>();

        pcap.loop((packet) -> {

            if (packet.hasProtocol(Protocol.TCP) && packet.hasProtocol(Protocol.IPv4)) {

                TCPPacket tcpPacket = (TCPPacket) packet.getPacket(Protocol.TCP);

                Buffer payload = tcpPacket.getPayload();

                if (payload == null) {
                    return true;
                }

                String destIP = tcpPacket.getDestinationIP();
                int destPort = tcpPacket.getDestinationPort();

                if (hostPorts != null && !hostPorts.isEmpty()) {

                    boolean notMatch = true;
                    for (HostPort hostPort : hostPorts) {
                        if (hostPort != null) {
                            if ((hostPort.getHost() == null || hostPort.getHost().equals(destIP))
                                    && (hostPort.getPort() == null || hostPort.getPort().equals(destPort))) {
                                notMatch = false;
                            }
                        }
                    }
                    if (notMatch) {
                        return true;
                    }
                }

                long arrivalTime = tcpPacket.getArrivalTime();

                boolean psh = tcpPacket.isPSH();

                // FIXME ? 考虑根据 acknowledgementNumber来合并TCP包
                long acknowledgementNumber = tcpPacket.getAcknowledgementNumber();

                if (psh) {
                    HttpPacket httpPacket;
                    if ((httpPacket = halfTcpPacketMap.remove(acknowledgementNumber)) != null) {
                        httpPacket.setPayload(Buffers.wrap(httpPacket.getPayload(), payload));
                    } else {
                        httpPacket = HttpPacket.builder().arrivalTime(arrivalTime).destIp(destIP).destPort(destPort).payload(payload).build();
                    }

                    System.out.println(httpPacket.getPayload()
                            .toString());

                    String[] httpContents = httpPacket.getPayload()
                            .toString()
                            .split(System.lineSeparator());
                    String firstLine = httpContents[0];
                    String[] schema = firstLine.split(URL_SEPARATOR);
                    if (schema.length != 3) {
                        // 该包有问题，需要丢弃
                        System.out.println("### illegal packet: " + httpPacket.getPayload());
                        return true;
                    }
                    httpPackets.add(httpPacket);

                } else {
                    halfTcpPacketMap.compute(acknowledgementNumber, (k, p) -> {
                        if (p == null) {
                            p = HttpPacket.builder().arrivalTime(arrivalTime).destIp(destIP).destPort(destPort).payload(payload).build();
                        } else {
                            p.setPayload(Buffers.wrap(p.getPayload(), payload));
                        }
                        return p;
                    });
                }

            }
            return true;
        });

        return httpPackets;
    }

}

@Getter
@Setter
@ToString
public class SimpleHttpRequest {

    private URI uri;

    private HttpMethod httpMethod;

    private String protocol;

    private HttpHeaders httpHeaders;

    private String body;

    public SimpleHttpRequest(HttpPacket httpPacket) {
        this(httpPacket.getDestIp(), httpPacket.getDestPort(), httpPacket.getPayload());
    }

    public SimpleHttpRequest(String destIp, Integer destPort, Buffer byteBuffer_) {
        try {
            Buffer byteBuffer = byteBuffer_.clone();
            String uriLine = byteBuffer.readLine().toString();
            String[] uriSplits = uriLine.split(" ");
            this.httpMethod = HttpMethod.valueOf(uriSplits[0]);

            //FIXME
            String uriStr = String.format("http://%s:%s%s", destIp, destPort, uriSplits[1]);
            try {
                this.uri = new URI(uriStr);
            } catch (Exception e) {
                throw new IllegalArgumentException(e.getMessage(), e);
            }

            this.protocol = uriSplits[2];
            this.httpHeaders = new DefaultHttpHeaders();

            Buffer line;

            StringBuilder builder = new StringBuilder();

            boolean header = true;
            while ((line = byteBuffer.readLine()) != null) {

                if (line.toString().equals("")) {
                    header = false;
                    continue;
                }

                if (header) {
                    String[] headerKv = line.toString().split(": ");
                    httpHeaders.add(headerKv[0], headerKv[1]);
                } else {
                    builder.append(line);
                }
            }
            this.body = builder.toString();
        } catch (Exception e) {
            throw new IllegalArgumentException(e.getMessage(), e);
        }
    }

    public Map<String, List<String>> getParameterMap() {
        if (uri == null || StringUtils.isEmpty(uri.getQuery())) {
            return Collections.emptyMap();
        }
        String[] params = uri.getQuery().split("&|&&");
        Map<String, List<String>> parameterMap = new HashMap<>();
        for (String p : params) {
            String[] pKv = p.split("=");

            parameterMap.compute(pKv[0], (k, v) -> {
                if (v == null) {
                    v = new ArrayList<>();
                }
                if (pKv.length == 2) {
                    v.add(pKv[1]);
                }
                return v;
            });
        }
        return parameterMap;
    }

}

public class PktsReadHttpPacketTest {

    public static void main(String[] args) throws IOException {

        String path = "/opt/apps/test2_20200707_153957.pcap";

        List<HttpPacket> httpPackets = HttpPacketParser.parse(new File(path),
                Arrays.asList(HostPort.builder().host("100.100.100.100").port(9999).build()));

        List<SimpleHttpRequest> simpleHttpRequests = httpPackets.stream().map(SimpleHttpRequest::new).collect(Collectors.toList());

        simpleHttpRequests.forEach(e->{
            System.out.println(e);
            System.out.println(e.getParameterMap());
            System.out.println(e.getUri().getPath());
            System.out.println(e.getHttpHeaders().get(HttpHeaderNames.HOST));
        });

    }

}

[valsong]

是要自己拼TCP包吗，我的诉求是解析出请求的方法（Get/Post之类的）、请求体、状态码等这些http的header

这个打印出来是这样子的

POST /app/baz?bar=qux&foo=1 HTTP/1.1
RemoteIp: 100.100.100.100
Host: foo.com
X-Forwarded-For: 111.111.111.111
Connection: close
Content-Length: 92
uid: 847d98464fb8a802

{
  "age": 9867,
  "email": "9867",
  "height": 9867,
  "id": 9867,
  "name": "9867"
}

接下来不就简单了嘛 第一行是Method 和 URL 下面是header 然后跳过一行，最后是body

[ZaNkss]

谢谢，我的代码也可以打印出这些文本消息。不过我希望得到的是http的header与payload结构化分离，比如HttpPacket中有requstMethod和host等字段。

[valsong]

谢谢，我的代码也可以打印出这些文本消息。不过我希望得到的是http的header与payload结构化分离，比如HttpPacket中有requstMethod和host等字段。

这段代码是为了解决请求过长时多个tcppacket合并的问题

[valsong]

谢谢，我的代码也可以打印出这些文本消息。不过我希望得到的是http的header与payload结构化分离，比如HttpPacket中有requstMethod和host等字段。

刚好我这边也要做这个需求，简单写了一个，上面的代码我更新了，新增了一个将ByteBuffer转化成SimpleHttpRequest的功能，我简单测了下没啥问题

结果类似这样的

SimpleHttpRequest(uri=http://100.100.100.100:9999/foo/bar?qux=baz,
 httpMethod=POST, 
 protocol=HTTP/1.1, 
 httpHeaders=DefaultHttpHeaders[User-Agent: curl/7.29.0, Host: 111.111.111.111:9999, Accept: */*, foo: 123, Content-Type: application/json, Content-Length: 25], 
 body={"foo":"123","bar":"456"})

[ZaNkss]

谢谢，我的代码也可以打印出这些文本消息。不过我希望得到的是http的header与payload结构化分离，比如HttpPacket中有requstMethod和host等字段。

刚好我这边也要做这个需求，简单写了一个，上面的代码我更新了，新增了一个将ByteBuffer转化成SimpleHttpRequest的功能，我简单测了下没啥问题

对，对http的拓展需要先对tcp包进行拼接，然后把tcp的payload转换为Http协议相关的，就是定义一个http帧的概念。但是我现在对实现还没有一个好办法

[valsong]

谢谢，我的代码也可以打印出这些文本消息。不过我希望得到的是http的header与payload结构化分离，比如HttpPacket中有requstMethod和host等字段。

刚好我这边也要做这个需求，简单写了一个，上面的代码我更新了，新增了一个将ByteBuffer转化成SimpleHttpRequest的功能，我简单测了下没啥问题

对，对http的拓展需要先对tcp包进行拼接，然后把tcp的payload转换为Http协议相关的，就是定义一个http帧的概念。但是我现在对实现还没有一个好办法

你跑下我的代码看看有没有啥bug，不过要指定拦截的ip和port，我这边测了合并tcp包是没啥问题了， 然后url和header以及body等也转化成SimpleHttRequest了

[jonbo372]

Hi all,

Sorry, a year later so I hope this has been resolved already but just some pointers just in case.

In short, your questions has nothing to do with pkts.io per se. It has to do with how to frame an HTTP message, which is slightly annoying since, unlike many binary protocols where there is a header stating how many bytes the message is, HTTP doesn't have this. SIP has similar "issues" and you would have to do something similar to what I have for my sip stack here: https://github.com/aboutsip/sipstack/blob/master/sipstack-netty-codec-sip/src/main/java/io/sipstack/netty/codec/sip/SipMessageStreamDecoder.java which is really using this https://github.com/aboutsip/sipstack/blob/master/sipstack-netty-codec-sip/src/main/java/io/sipstack/netty/codec/sip/RawMessage.java.

Anyway, I hope you managed to solve your problem a year ago!