search

apache / trafficserver

Apache Traffic Server™ is a fast, scalable and extensible HTTP/1.1 and HTTP/2 compliant caching proxy server.
https://trafficserver.apache.org/
Apache License 2.0
1.82k stars 805 forks source link

Content-Type is lost after TCP_REFRESH_HIT/200 of cached response with duplicated headers from origin #11784

Closed lukenowak closed 2 weeks ago

lukenowak commented 1 month ago

Initially the backend replies with response like:

HTTP/1.1 200 OK
Date: Mon, 23 Sep 2024 14:22:14 GMT
X-Reveal-Duplicate: same
Cache-Control: max-age=30, stale-if-error=604800, stale-while-revalidate=15, public
Content-Length: 14475
Content-Type: application/javascript
Expires: Mon, 23 Sep 2024 14:22:44 GMT
Last-Modified: Mon, 23 Sep 2024 14:22:14 GMT
X-Reveal-Duplicate: other

For the TCP_MISS/200 trafficserver query, the reply from traffic server has all expected headers:

HTTP/1.1 200 OK                                                                                                                                      
Date: Mon, 23 Sep 2024 14:22:14 GMT
X-Reveal-Duplicate: same
Cache-Control: max-age=30, stale-if-error=604800, stale-while-revalidate=15, public
Content-Length: 14475
Content-Type: application/javascript
Expires: Mon, 23 Sep 2024 14:22:44 GMT
Last-Modified: Mon, 23 Sep 2024 14:22:14 GMT
X-Reveal-Duplicate: other
Age: 0

All queries until max-age is reached are the same and correct.

After the max-age time passes trafficserver is doing TCP_REFRESH_HIT/200 to the backend with If-Modified-Since, which replies:

HTTP/1.1 304 Not Modified
Date: Mon, 23 Sep 2024 14:25:16 GMT
X-Reveal-Duplicate: same
Cache-Control: max-age=30, stale-if-error=604800, stale-while-revalidate=15, public
Content-Type: application/javascript
Expires: Mon, 23 Sep 2024 14:25:46 GMT
Last-Modified: Mon, 23 Sep 2024 14:25:16 GMT
X-Reveal-Duplicate: other

But then the Trafficserver replies to the client with headers:

HTTP/1.1 200 OK
Date: Mon, 23 Sep 2024 14:25:16 GMT
Content-length: 14475
X-Reveal-Duplicate: same
Cache-Control: max-age=30, stale-if-error=604800, stale-while-revalidate=15, public
Expires: Mon, 23 Sep 2024 14:25:46 GMT
Last-Modified: Mon, 23 Sep 2024 14:25:16 GMT
X-Reveal-Duplicate: other
Age: 0

Please see, that it replies without Content-Type header, which can be seen also in the squid.log, as the line is logged like:

1727101089.801 2 127.0.0.1 TCP_REFRESH_HIT/200 14933 GET http://example.com - DIRECT/127.0.0.1 - When the duplicated header X-Reveal-Duplicate has the same value, it results with same problem not transmitting Content-Type.

When the backend replies instead with 200 response to the If-Modified-Since then the Content-Type header is correctly transmitted.

Seems like dropping the Content-Type header there is a bug.

Reproduced on trafficserver 9.2.5, configure options:

  --with-openssl=provided-openssl
  --with-pcre=provided-pcre
  --with-ncurses=provided-ncurses
  --with-tcl=provided-tcl
  --with-luajit=provided-luajit
  --with-lzma=provided-xzutils
  --with-zlib=provided-zlib
  --disable-curl
  --disable-hwloc
  --enable-experimental-plugins
  --disable-posix-cap
jasmine-nahrain commented 1 month ago

I have not been able to replicate your issue. Could you please provide more detail about your setup?

What I also find interesting is that your duplicate headers are returned as two seperate lines. Whereas in my testing they were returning on a single header. E.g. X-Duplicate-Header: same, other

lukenowak commented 1 month ago

I have not been able to replicate your issue. Could you please provide more detail about your setup?

Sure.

  • What openssl are you using?

3.0.15

  • Are you using any plugins? If so, which ones?

My plugin.config is empty, but I added extracts of my configuration.

storage.config.clean.txt records.config.clean.txt remap.config.txt

  • Does the origin server respond with a content-type header on cache fill

Yes, I provided full reply of the origin for cache fill with 200 and refresh with 304.

  • Did you only test this in 9.2.5 or any other version?

Only 9.2.5.

What I also find interesting is that your duplicate headers are returned as two seperate lines. Whereas in my testing they were returning on a single header. E.g. X-Duplicate-Header: same, other

Interesting. I am ready to assist you with trying out different configurations and versions. Using 10 would require for me to migrate the configuration a lot, so staying on 9.x will be much easier.

jasmine-nahrain commented 1 month ago

Thanks for the extra info! A few more questions

lukenowak commented 1 month ago

Thanks for the extra info! A few more questions

  • I see that in the response sent to the client for stale hit, the l in content length is not capitalised. Was this you copying the response over and missing the capital or is this what your origin server is sending back?

The Content-Length sent by backend is correctly capitalized, I made a mistake while pasting here.

  • Are you able to replicate this issue and send me the logs? Specifically http and cache logs. In records.config
 proxy.config.diags.debug.tags: 'http*|cache*'

Thanks for the hint, here it goes: diag-for-ticket.log

  • Is the header X-Reveal-Duplicate what you are actually sending or is it used for the example? If it’s not what you are sending, can you describe what the header is? e.g is it completely unique to you or is it some standard header.

The header X-Reveal-Duplicate is exactly that one, I have separate environment to reproduce those problems.

vpelletier commented 1 month ago

(lukenowak colleague here)

The header X-Reveal-Duplicate is exactly that one, I have separate environment to reproduce those problems.

In the original setup which triggered the issue, the duplicate header was actually X-Frame-Options. Using X-Reveal-Duplicate in the reproduction environment was IIRC to check whether such standard(-ish ?) header was being handled differently - and it is not, as the failure was reproduced with such artificial header.