search

4chan / 4chan-API

Documentation for 4chan's read-only JSON API.
http://www.4chan.org/
1.03k stars 74 forks source link

last_modified of HEAD and GET are not always in sync #81

Open tthverst opened 4 years ago

tthverst commented 4 years ago

In the text below you can see the first entry which shows the headers of the HEAD response where the last_modified header is Wed, 15 Jul 2020 20:08:32 GMT. The second entry is the if-modified-since header for the GET request. The third entry shows the headers of the GET response where the last_modified header is Wed, 15 Jul 2020 20:07:46 GMT The url is https://a.4cdn.org/tv/catalog.json.

Why are these two last_modified headers different?

1: {connection: keep-alive, last-modified: Wed, 15 Jul 2020 20:08:32 GMT, cache-control: public, must-revalidate, proxy-revalidate, cf-request-id: 03f5b0e4fe0000faa0c4955200000001, date: Wed, 15 Jul 2020 20:08:33 GMT, access-control-allow-origin: http://boards.4chan.org, vary: Accept-Encoding,Origin, Accept-Encoding, content-encoding: gzip, cf-cache-status: EXPIRED, expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct", access-control-max-age: 1728000, content-type: application/json, server: cloudflare, access-control-allow-headers: If-Modified-Since, accept-ranges: bytes, access-control-allow-methods: GET, OPTIONS, content-length: 76148, cf-ray: 5b361db4cdd3faa0-AMS, etag: "5f0f6240-12974", expires: -1} 2: {if-modified-since: Wed, 15 Jul 2020 20:07:46 GMT} 3: {connection: keep-alive, cache-control: public, must-revalidate, proxy-revalidate, last-modified: Wed, 15 Jul 2020 20:07:46 GMT, cf-request-id: 03f5b0e8d20000fa38d4806200000001, date: Wed, 15 Jul 2020 20:08:33 GMT, access-control-allow-origin: http://boards.4chan.org, vary: Accept-Encoding,Origin, Accept-Encoding, age: 35, cf-cache-status: UPDATING, expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct", access-control-max-age: 1728000, server: cloudflare, access-control-allow-headers: If-Modified-Since, access-control-allow-methods: GET, OPTIONS, cf-ray: 5b361dbae89bfa38-AMS, etag: "5f0f6212-12885", expires: -1}

desuwa commented 4 years ago

Works fine here.

How exactly are you doing your requests? Make sure you send the same Origin header every time.

tthverst commented 4 years ago

Thank you for your quick response. The weird thing is, is that it does not happen every request.

I'm making an app with flutter. http is a dart package. url is https://a.4cdn.org/tv/catalog.json. previous_last_modified is Wed, 15 Jul 2020 20:07:46 GMT.

final head = await http.head(url); final last_modified = head.headers[HttpHeaders.lastModifiedHeader]; // Wed, 15 Jul 2020 20:08:32 GMT final response = await http.get(url, headers: { HttpHeaders.ifModifiedSinceHeader: previous_last_modified }); final response_last_modified = response.headers[HttpHeaders.lastModifiedHeader] // Wed, 15 Jul 2020 20:07:46 GMT

I'll try to explicitly state the origin header. I'll also check if previous_last_modified is always the same as response_last_modified or if it was just a coincidence this time.

EDIT: Explicitly stating the origin header did not change anything. Also previous_last_modified was always the same as response_last_modified.

tthverst commented 4 years ago

I will also make the requests via Postman to make sure it is not a Flutter problem.

EDIT: I've found that when the HEAD and GET are received in the same second this problem occurs (See also the date headers in my first comment). I could not reproduce this in Postman because I'm not fast enough. If I add a delay of 1 second between the requests the problem does not occur. But this is obviously not viable.