antctl traceflow doesn't work for traffic to service

Describe the bug antctl traceflow doesn't work for destination service.
The injected packet is not processed as packets to service(clusterip) in ovs pipeline.

$ antctl traceflow -S elearning/rating-88bd94c44-xz4cf -D elearning/course -f tcp,tcp_dst=80
name: elearning-rating-88bd94c44-xz4cf-to-elearning-course-dx8p4szf
phase: Succeeded
source: elearning/rating-88bd94c44-xz4cf
destination: elearning/course
results:
- node: k8s3-node01
  timestamp: 1597917738
  observations:
  - component: SpoofGuard
    action: Forwarded
  - component: NetworkPolicy
    componentInfo: EgressDefaultRule
    action: Dropped

To Reproduce Do antctl traceflow for traffic to service like below.

$ antctl traceflow -S elearning/rating-88bd94c44-xz4cf -D elearning/course -f tcp,tcp_dst=80
name: elearning-rating-88bd94c44-xz4cf-to-elearning-course-dx8p4szf
phase: Succeeded
source: elearning/rating-88bd94c44-xz4cf
destination: elearning/course
results:
- node: k8s3-node01
  timestamp: 1597917738
  observations:
  - component: SpoofGuard
    action: Forwarded
  - component: NetworkPolicy
    componentInfo: EgressDefaultRule
    action: Dropped

Expected The injected packet is loadbalanced and delivered to the pod. Based on analysis with @gran-vmv , the following flag must be set to 2 when traffic is sent out to service.

transportHeader:
      tcp:
        srcPort: xx
        dstPort: xx
        flags: 2

Then It should work like below.

$ k get tf elearning-rating-to-elearning-course-svc -o yaml
apiVersion: ops.antrea.tanzu.vmware.com/v1alpha1
kind: Traceflow
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"ops.antrea.tanzu.vmware.com/v1alpha1","kind":"Traceflow","metadata":{"annotations":{},"name":"elearning-rating-to-elearning-course-svc"},"spec":{"destination":{"namespace":"elearning","service":"course"},"packet":{"ipHeader":{"protocol":6},"transportHeader":{"tcp":{"dstPort":80,"flags":2,"srcPort":10002}}},"source":{"namespace":"elearning","pod":"rating-88bd94c44-xz4cf"}}}
  creationTimestamp: "2020-08-20T11:21:00Z"
  generation: 1
  managedFields:
  - apiVersion: ops.antrea.tanzu.vmware.com/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
      f:spec:
        .: {}
        f:destination:
          .: {}
          f:namespace: {}
          f:service: {}
        f:packet:
          .: {}
          f:ipHeader:
            .: {}
            f:protocol: {}
          f:transportHeader:
            .: {}
            f:tcp:
              .: {}
              f:dstPort: {}
              f:flags: {}
              f:srcPort: {}
        f:source:
          .: {}
          f:namespace: {}
          f:pod: {}
    manager: kubectl
    operation: Update
    time: "2020-08-20T11:21:00Z"
  - apiVersion: ops.antrea.tanzu.vmware.com/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        .: {}
        f:dataplaneTag: {}
        f:phase: {}
        f:results: {}
    manager: antrea-controller
    operation: Update
    time: "2020-08-20T11:21:05Z"
  name: elearning-rating-to-elearning-course-svc
  resourceVersion: "528512"
  selfLink: /apis/ops.antrea.tanzu.vmware.com/v1alpha1/traceflows/elearning-rating-to-elearning-course-svc
  uid: e2252307-c3d6-4407-b251-1ff92f77ded1
spec:
  destination:
    namespace: elearning
    service: course
  packet:
    ipHeader:
      protocol: 6
    transportHeader:
      tcp:
        dstPort: 80
        flags: 2
        srcPort: 10002
  source:
    namespace: elearning
    pod: rating-88bd94c44-xz4cf
status:
  dataplaneTag: 1
  phase: Succeeded
  results:
  - node: k8s3-node01
    observations:
    - action: Forwarded
      component: SpoofGuard
    - action: Forwarded
      component: LB
      pod: elearning/course-6576dc8cb4-rg6w4
      translatedDstIP: 172.29.2.46
    - action: Forwarded
      component: NetworkPolicy
      componentInfo: IngressRule
      networkPolicy: /course-api
    - action: Delivered
      component: Forwarding
      componentInfo: Output
    timestamp: 1597922465

Actual behavior The injected packet is not processed as packets to service(clusterip) in ovs pipeline.

Versions: Please provide the following information:

Antrea version (Docker image tag). v0.9.0

Kubernetes version (use kubectl version). If your Kubernetes components have different versions, please provide the version for all of them.

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:12:48Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:04:18Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Container runtime: which runtime are you using (e.g. containerd, cri-o, docker) and which version are you using? Docker 19.0.3.4
Linux kernel version on the Kubernetes Nodes (uname -r). 4.15.0-112-generic
If you chose to compile the Open vSwitch kernel module manually instead of using the kernel module built into the Linux kernel, which version of the OVS kernel module are you using? Include the output of modinfo openvswitch for the Kubernetes Nodes.

Additional context Add any other context about the problem here, such as Antrea logs, kubelet logs, etc.

(Please consider pasting long output into a GitHub gist or any other pastebin.)

antrea-io / antrea

antctl traceflow doesn't work for traffic to service #1126