Introduced `includeTrailerHeader` to enable `TrailerHeaders` in response

[vivek807]

Introduced includeTrailerHeader to enable TrailerHeaders in response If enabled, a header X-Error-Message will be added to indicate reasons for partial results.

Description

Currently, when requesting large amounts of data, the result may be truncated due to query timeouts or other issues. To address this, we are using a non-standard solution.

    // Terminate the last output line, then write an extra blank line, so users can tell the response was not cut off.
    outputStream.write(new byte[]{'\n', '\n'});

To handle this in a standard way, we can implement a trailer header that indicates partial results and the reason for the truncation.

This PR has:

• [x] been self-reviewed.
  • [ ] using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• [ ] added documentation for new or modified features or behaviors.
• [ ] a release note entry in the PR description.
• [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• [ ] added or updated version, license, or notice information in licenses.yaml
• [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• [ ] added integration tests.
• [ ] been tested in a test Druid cluster.

[abhishekagarwal87]

You may want to look into an old PR of mine (now stale) - https://github.com/apache/druid/pull/13492 - The solution in the PR stopped working after query resources were refactored and I didn't get the time to dig into the reason. But given you are in this area, maybe you can explore that patch.

[vivek807]

You may want to look into an old PR of mine (now stale) - #13492 - The solution in the PR stopped working after query resources were refactored and I didn't get the time to dig into the reason. But given you are in this area, maybe you can explore that patch.

Thanks @abhishekagarwal87 for your comment. We added a trailer header to indicate truncated results when fetching large amounts of data. For instance, if a query times out after streaming some results, the count and the actual result may mismatch. To signal this, we included an error message in the trailer header to indicate the reason for truncated results, which can be enabled in query context

[nozjkoitop]

Hi @abhishekagarwal87,

While it's not straightforward to resolve the query result truncation issue, this solution is aimed at highlighting when results are truncated and helping to identify the reason. The rationale is pretty legit - we don't want to truncate the query results silently.

@vivek807, could you please revert the formatting changes in the documentation?

[vivek807]

Hi @abhishekagarwal87,

While it's not straightforward to resolve the query result truncation issue, this solution is aimed at highlighting when results are truncated and helping to identify the reason. The rationale is pretty legit - we don't want to truncate the query results silently.

@vivek807, could you please revert the formatting changes in the documentation?

Updated, thanks @nozjkoitop