FR-002: Hide lvs nodes on the topology graph

[eddietanshuo]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Description

Description

We wish deepflow provides a availiability to hide lvs nodes and show client - server connection directly.

Use Case

For Xiaomi company, A lot of lvs nodes could confuse our user to understand their business flow. To simplify the complexity, deepflow could provide a option to hide lvs nodes.

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[sharang]

About TOA (TCP Option Address)

In the FULLNAT scenario, it is a common requirement for the server to obtain the real client IP address. Driven by this demand, the TOA (TCP Option Address) mechanism appeared.

The TOA mechanism is widely used by cloud vendors and is also a mature kernel module. It was originally introduced by LVS, and was later supported by HAProxy. Using the TOA mechanism, LVS can inject the client IP and Port into the TCP Option when forwarding traffic. By installing the toa kernel module on the RS (Real Server) and hijacking the inet_getname system call, the application can get the real client IP address insensibly.

refs:

• https://github.com/iqiyi/dpvs/blob/master/doc/tutorial.md#simple-full-nat-two-arm
• https://github.com/Huawei/TCP_option_address/blob/master/src/toa.c#L93

code in alibaba/LVS: https://github.com/alibaba/LVS/blob/master/kernel/net/netfilter/ipvs/ip_vs_proto_tcp.c#L1055

    /*
     * for syn packet
     * 1. remove tcp timestamp opt,
     *    because local address with diffrent client have the diffrent timestamp;
     * 2. recompute tcp sequence
     * 3. add toa
     */
    if (tcph->syn & !tcph->ack) {
        tcp_opt_remove_timestamp(tcph);
        tcp_in_init_seq(cp, skb, tcph);
#ifdef CONFIG_IP_VS_IPV6
        if (cp->af == AF_INET6)
            tcp_opt_add_toa_v6(cp, skb, &tcph);
        else
#endif
            tcp_opt_add_toa(cp, skb, &tcph);
    }

    /* TOA: add client ip */
    if ((sysctl_ip_vs_toa_entry == 1)
        && (ntohl(tcph->ack_seq) == cp->fnat_seq.fdata_seq)
        && !tcph->syn && !tcph->rst && !tcph->fin) {
#ifdef CONFIG_IP_VS_IPV6
        if (cp->af == AF_INET6)
            tcp_opt_add_toa_v6(cp, skb, &tcph);
        else
#endif
            tcp_opt_add_toa(cp, skb, &tcph);
    }

code in iqiyi/DPVS: https://github.com/iqiyi/dpvs/blob/master/src/ipvs/ip_vs_proto_tcp.c#L711

    /*
     * for SYN packet
     * 1. remove tcp timestamp option
     *    laddress for different client have diff timestamp.
     * 2. save original TCP sequence for seq-adjust later.
     *    since TCP option will be change.
     * 3. add TOA option
     *    so that RS with TOA module can get real client IP.
     */
    if (th->syn && !th->ack) {
        tcp_in_remove_ts(th);
        tcp_in_init_seq(conn, mbuf, th);
        tcp_in_add_toa(conn, mbuf, th);
    }

    /* add toa to first data packet */
    if (ntohl(th->ack_seq) == conn->fnat_seq.fdata_seq
            && !th->syn && !th->rst /*&& !th->fin*/)
        tcp_in_add_toa(conn, mbuf, th);

How DeepFlow will use TOA

Almost every TCP packet contains TOA information, and its wire protocol can refer to the following kernel module code: https://github.com/Huawei/TCP_option_address/blob/master/src/toa.h#L40

#define TCPOPT_TOA  254

/* MUST be 4n !!!! */
#define TCPOLEN_TOA 8       /* |opcode|size|ip+port| = 1 + 1 + 6 */

/* MUST be 4 bytes alignment */
struct toa_data {
    __u8 opcode;
    __u8 opsize;
    __u16 port;
    __u32 ip;
};

We plan to add two fields, real_client_ip and real_client_port, to TaggedFlow, and replace the client IP/Port of Flow when generating aggregated metrics.

The processing logic is as follows:

1. Get TOA in MetaPacket and record toa_offset
2. Add a new field to save TOA in Flow
3. In FlowGenerator's new_tcp_node and update_tcp_flow store the TOA information of MetaPacket into Flow
4. ... TODO

[sharang]

already done