Issue with Endpoints Port Forwarding outside of Otel.

[VenuEmmadi]

Component(s)

No response

What happened?

PortForwardIssueOpenTelemetry.docx

Collector version

v0.109.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04") Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
     otlp:
       protocols:
         grpc:
   extensions:
     health_check:
       path: "/"
   service:
     extensions: [health_check]
     pipelines:
       metrics:
         receivers: [otlp]

Log output

curl -v http://localhost:13133
  * processing: http://localhost:13133
  *   Trying [::1]:13133...
  * Connected to localhost (::1) port 13133
  > GET / HTTP/1.1
  > Host: localhost:13133
  > User-Agent: curl/8.2.1
  > Accept: */*
  >
  * Recv failure: Connection was aborted
  * Closing connection
  curl: (56) Recv failure: Connection was aborted

Additional context

Title: Issues with Endpoints Port Forwarding outside of OpenTelemetry Collector application

Title: Issues with Endpoints Port Forwarding outside of OpenTelemetry Collector application

1. Description: I am encountering issues with Endpoints like the health check and gRPC endpoints in the OpenTelemetry Collector. The problems are particularly evident when these endpoints are not explicitly configured, resulting in connection aborts. Below are the detailed observations and configurations related to this issue.

Issue:

When attempting to access the health check and gRPC endpoints of the OpenTelemetry Collector, the connections are being aborted Example:

• Health Check Endpoint: http://localhost:13133
• gRPC Endpoint: http://localhost:4318

Expected Behavior according to documentation:

• Health Check Endpoint: Should respond with a status indicating the health of the collector.
• gRPC Endpoint: Should be accessible for receiving gRPC requests.

2. Actual Behavior:

• Health Check Endpoint: When accessed without an explicitly defined endpoint, the response is an error indicating that the connection was aborted. The following curl command results in:

  curl -v http://localhost:13133/

  • processing: http://localhost:13133/
  • Trying [::1]:13133...
  • Connected to localhost (::1) port 13133

    GET / HTTP/1.1 Host: localhost:13133 User-Agent: curl/8.2.1 Accept: /

  • Recv failure: Connection was aborted
  • Closing connection curl: (56) Recv failure: Connection was aborted From Browser the output is as below: Its Showwing “ERR_EMPTY_RESPONSE”.

• gRPC Endpoint: Similarly, when not configured explicitly, attempts to connect to the gRPC endpoint result in connection aborts. However, when explicitly configured, the connection succeeds.

Configuration Files:

1. Configuration Without Explicit Endpoint Definitions:

   receivers:
    otlp:
      protocols:
        grpc:
   extensions:
    health_check:
      path: "/"
   service:
    extensions: [health_check]
    pipelines:
      metrics:
        receivers: [otlp]

   With this configuration, the health check and gRPC endpoints fail to respond properly as shown below. curl -v http://localhost:13133/

   • processing: http://localhost:13133/
   • Trying [::1]:13133...
   • Connected to localhost (::1) port 13133

     GET / HTTP/1.1 Host: localhost:13133 User-Agent: curl/8.2.1 Accept: /

   • Recv failure: Connection was aborted
   • Closing connection curl: (56) Recv failure: Connection was aborted From Browser the output is as below: Its Showwing “ERR_EMPTY_RESPONSE”.

2. Configuration With Explicit Endpoint Definitions:

   receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "[0.0.0.0:4318](https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0.0.0.0%3A4318%2F&data=05%7C02%7Cvenumadhav.emmadi%40infosys.com%7C53c4f24a94a5494fd36e08dcd8f33866%7C63ce7d592f3e42cda8ccbe764cff5eb6%7C0%7C0%7C638623786360026286%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=nYm5wAc1ovdmsl0%2FiDx3Qa%2Bczc4RU79%2F0Fvm7cR6zFA%3D&reserved=0)"
   extensions:
    health_check:
      endpoint: "[0.0.0.0:13133](https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0.0.0.0%3A13133%2F&data=05%7C02%7Cvenumadhav.emmadi%40infosys.com%7C53c4f24a94a5494fd36e08dcd8f33866%7C63ce7d592f3e42cda8ccbe764cff5eb6%7C0%7C0%7C638623786360048365%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=TyHVueKfQN%2FHUg5%2FpeHVxDY81%2FPEzgi%2BsSlTZ2t2HBU%3D&reserved=0)"
      path: "/"
   service:
    extensions: [health_check]
    pipelines:
      metrics:
        receivers: [otlp] 

   With this configuration, both endpoints work correctly. Specifically:

   curl -v http://localhost:13133/
   * processing: http://localhost:13133/
   *   Trying [::1]:13133...
   * Connected to localhost (::1) port 13133
   > GET / HTTP/1.1
   > Host: localhost:13133
   > User-Agent: curl/8.2.1
   > Accept: */*
   >
   < HTTP/1.1 200 OK
   < Content-Type: application/json
   < Date: Thu, 19 Sep 2024 21:32:46 GMT
   < Content-Length: 98
   <
   {"status":"Server available","upSince":"2024-09-19T21:24:23.69654773Z","uptime":"8m23.117058632s"}

   From Browser the output is as below:

3. Expected Behavior:

• Health Check Endpoint: Should respond with a status indicating the health of the collector.
• gRPC Endpoint: Should be accessible for receiving gRPC requests. Below output should be given by the server even without explicitly given the endpoint. curl -v http://localhost:13133/
  • processing: http://localhost:13133/
  • Trying [::1]:13133...
  • Connected to localhost (::1) port 13133

    GET / HTTP/1.1 Host: localhost:13133 User-Agent: curl/8.2.1 Accept: /

    < HTTP/1.1 200 OK < Content-Type: application/json < Date: Thu, 19 Sep 2024 21:32:46 GMT < Content-Length: 98 < {"status":"Server available","upSince":"2024-09-19T21:24:23.69654773Z","uptime":"8m23.117058632s"} From Browser:

4. Additional Information:

The issue seems to be related to the default behavior of endpoint configurations in the OpenTelemetry Collector. When endpoints are not explicitly defined, the connections to the health check and gRPC services fail. Explicitly defining these endpoints resolves the issue, allowing for proper communication and status responses.

5. Resolution:

->Deploy the OpenTelemetry Collector using the configuration without explicit endpoint definitions. ->Attempt to access the health check endpoint at http://localhost:13133. ->Attempt to access the gRPC endpoint at http://localhost:4318. -> Observe the connection abort errors. -> Modify the configuration to include explicit endpoint definitions for both services. ->Retry accessing the endpoints and confirm that they respond correctly.

This issue needs further investigation to understand why its behaving like above in new versions.

References:

• OpenTelemetry Collector Documentation
• Issue Tracker

[atoulme]

This issue is too long - please break it into smaller issues that can be assigned to a specific component for review.

[VenuEmmadi]

Hi @atoulme,

As per your suggestion, I have broken down the issue into smaller, more focused parts and created new issues for each specific concern. Additionally, I have edited the original issue above to make it shorter and more precise.

Please let me know if further adjustments are needed or if any of the newly created issues require additional details.

Below are the individual issues for review:

1. Health Check Endpoint Issue Title: Health Check Endpoint Fails Without Explicit Configuration in OpenTelemetry Collector v0.109.0 Description: Focus on the /health_check endpoint failing when not explicitly defined. Provide details on the expected behavior versus the observed error and the workaround.

Key Points: Configuration used without explicit health_check endpoint. Steps to reproduce using curl. Error logs and behavior. Explicit configuration as a workaround. Mention environment details and ask whether this behavior is intended.

---

2. gRPC Endpoint Issue Title: gRPC Endpoint Not Accessible Without Explicit Configuration in OpenTelemetry Collector v0.109.0 Description: Focus on the gRPC endpoint's behavior when not explicitly configured. Highlight the default behavior expected and the error observed.

Key Points:

Include the configuration used without endpoint for grpc. Steps to reproduce the issue. Logs and observed errors. Explicit configuration as a workaround. Ask for clarification if this behavior aligns with the intended default setup.

---

3. Default Endpoint Behavior Title: Investigate Default Endpoint Behavior in OpenTelemetry Collector v0.109.0 Description: Raise a generic issue to request clarification or changes regarding default endpoint behavior when configurations are omitted.

Key Points:

Consolidate observations about both health check and gRPC endpoints. Emphasize how explicit configurations resolve issues. Request documentation updates or confirmation of expected behavior.