how to access response header info?

[Julianoe]

reading @swaroopsm issue about caching headers I realize I'm trying to achieve the exact same thing: I need to know the total page (x-wp-totalpages) from the response header. Except I can't find how I'm suppose to do it. When I try to read the header it returns an error : WordPress API call failed: TypeError: Cannot read properties of undefined (reading 'get')

My codes reads as follows:

const wordpressAPIpages = 'https://example.com/wp-json/wp/v2/posts?orderby=date&order=desc&_fields=id&page=1';
async function wpPostPages() {
  try {
    const res = await EleventyFetch(wordpressAPIpages, { duration: "2d", type: "json" });
    return res.headers.get('x-wp-totalpages') || 0;
  }
  catch(err) {
    console.log(`WordPress API call failed: ${err}`);
    return 0;
  }
}

I tried to read things like this. But it did not help. Is it normal I don't have my header coming back? I need guidance here 🤔

[pdehaan]

I'd probably try using something like axios to fetch the data+headers and manually manage the cache using https://www.11ty.dev/docs/plugins/fetch/#manually-store-your-own-data-in-the-cache.

[pdehaan]

Something like this might work (or not, I don't have a WordPress feed, just a HaveIBeenPwned API endpoint, which doesn't require pagination):

// src/_data/hibp.js
const { AssetCache } = require("@11ty/eleventy-fetch");
const axios = require("axios");

module.exports = async function() {
  const asset = new AssetCache("hibp_breaches");
  if (asset.isCacheValid("2h")) {
    console.log("Using cached value");
    return asset.getCachedValue();
  }
  const url = "https://haveibeenpwned.com/api/v3/breaches/";
  const { headers, data } = await axios.get(url);

  const value = { headers, data, now: new Date() };
  await asset.save(value, "json");
  return value;
};

---
# src/index.liquid
title: cache test
---

<pre data-length>{{ hibp.data.length }}</pre>
<pre data-headers>{{ hibp.headers | json }}</pre>

powered by: {{ hibp.headers["x-powered-by"] }}
fetch date: {{ hibp.now }}

OUTPUT

<pre data-length>629</pre>
<pre data-headers>{"date":"Sat, 10 Sep 2022 01:01:28 GMT","content-type":"application/json; charset=utf-8","transfer-encoding":"chunked","connection":"close","cf-ray":"74843b8b5ec1088d-SEA","access-control-allow-origin":"*","cache-control":"public, max-age=600","strict-transport-security":"max-age=31536000; includeSubDomains; preload","vary":"Accept-Encoding","cf-cache-status":"DYNAMIC","arr-disable-session-affinity":"True","x-content-type-options":"nosniff","x-powered-by":"ASP.NET","set-cookie":["__cf_bm=C0ZYu_fanPaebCmnnRWAM02T4FwRj5TFmJTLNerhUIo-1662771688-0-AbUGoyqeMJ0aM75U+4rKCUH6ABXBR0XBCJ8nEk4y95CVWF8b6qGBpF9xDqyJwaNPi/xmL3C9Z8X8uQ7QPgi0+Eo=; path=/; expires=Sat, 10-Sep-22 01:31:28 GMT; domain=.haveibeenpwned.com; HttpOnly; Secure; SameSite=None"],"server":"cloudflare"}</pre>

powered by: ASP.NET
fetch date: 2022-09-10T01:01:28.356Z

[Julianoe]

That works well for what I'm trying to achieve. I've grabbed the info from the headers. Now what's the syntax to use it not in a template but in another data file? I created the ./_data/wppages.js file and the ./_data/dosomethingwithwp/.js file where I need the wppages info and something like the following does not seem to work:

  const { wppostspages } = require { './wpwppages' };
or
  const wppostspages = require ('./wpwppages' );

[pdehaan]

It's hard to say without seeing code. But if the ./_data/wpwppages.js file returns an async function, you'll need to make sure you do an await and see if it fetches data or returns a cached value.

I'm not sure I understand what you're trying to do. Are you just trying to fetch all the paginated results from WordPress and cache a combined array of results? Not sure I still have any active WordPress blogs that I can try figuring out the API or wp-json whatever. Unless you have a public endpoint I can poke at.

[pdehaan]

Ah, think I found one. https://wordpress.org/news/wp-json/wp/v2/posts?per_page=3

---

UPDATE: And my quick code to recursively fetch every page is:

const fs = require("node:fs/promises");
const axios = require("axios");

main();

async function main() {
  const posts = await fetchPages();
  await fs.writeFile("wp.json", JSON.stringify(posts));
  console.log(`Done! ${posts.length} entries`);
}

async function fetchPages(page = 1) {
  const res = [];
  let totalPages = 1;
  do {
    console.log(`Fetching page ${page}/${totalPages}`);
    const { data, headers } = await axios.get("https://wordpress.org/news/wp-json/wp/v2/posts", {params: {per_page:50, page}});
    res.push(data);
    totalPages = headers['x-wp-totalpages'];
    page += 1;
  } while (page <= totalPages);
  return res.flat();
}

And that seems to write a 7.9 MB minified data JSON file to the local directory, and takes ~20s to complete (for 830 posts and 17 network requests). Ouch.

> time node index

Fetching page 1/1
Fetching page 2/17
Fetching page 3/17
Fetching page 4/17
Fetching page 5/17
Fetching page 6/17
Fetching page 7/17
Fetching page 8/17
Fetching page 9/17
Fetching page 10/17
Fetching page 11/17
Fetching page 12/17
Fetching page 13/17
Fetching page 14/17
Fetching page 15/17
Fetching page 16/17
Fetching page 17/17
Done! 830 entries

real    0m20.935s
user    0m0.323s
sys     0m0.142s

And now you have [at least] two options:

1. Wrap that general caching logic around the const { AssetCache } = require(https://github.com/11ty/eleventy-fetch); code from above so before doing any fetching of assets, so it only loads the data if it is older than the specified age (ie: 2 hours in the sample above).
2. Leave the fetching+writing code in a separate script that you run manually. In a lot of cases if my data doesn't change often, I'll just manually run my "fetch-data.js" script and save the output as a static JSON file in "./src/_data/wppages.json", or wherever.

But again, I don't know your use case or exactly what you're trying to do or why you need headers. But it was an interesting challenge.

[pdehaan]

Also noticed there is a headers.link header [in the response] which looks like this, with prev and next style navigation:

<https://wordpress.org/news/wp-json/wp/v2/posts?per_page=100&orderby=date&order=desc&_fields=id,title&page=3>; rel="prev", <https://wordpress.org/news/wp-json/wp/v2/posts?per_page=100&orderby=date&order=desc&_fields=id,title&page=5>; rel="next"

So we could also do something like this where we fetch the first page, and then keep fetching until there is no more next navigation (warning: RegExp ahead):

const axios = require("axios");

main();

async function main() {
  const posts = [];
  let url = "https://wordpress.org/news/wp-json/wp/v2/posts?per_page=100&orderby=date&order=desc&_fields=id,title";
  while (url) {
    const { data, headers } = await axios.get(url);
    posts.push(...data);
    url = getNav(headers.link).next;
  }
  return posts;
}

function getNav(link = "") {
  const relNavRegExp = /<(?<href>.*?)>; rel="(?<rel>prev|next)"/g;
  return [...link.matchAll(relNavRegExp)]
    .reduce((acc, { groups: { rel, href } }) => Object.assign(acc, { [rel]: href }), {});
}

[Julianoe]

Hey @pdehaan thanks for you comprehensive answers to my newbie question! As my own request does not fetch as much posts as you tested out, I started by implementing a separate script that assembles the json and puts it in the _data/ folder. And then I can run the script to build the site. Works like clockwork!

Once I get the other features I want rolling, I'll add some caching to it. Thanks for the help!

[zachleat]

This is an automated message to let you know that a helpful response was posted to your issue and for the health of the repository issue tracker the issue will be closed. This is to help alleviate issues hanging open waiting for a response from the original poster.

If the response works to solve your problem—great! But if you’re still having problems, do not let the issue’s closing deter you if you have additional questions! Post another comment and we will reopen the issue. Thanks!

[groenroos]

I think maybe this issue should be kept open. While the example code here can indeed work as a workaround, appears that most of it doesn't even use EleventyFetch and re-implements the fetching separately - so it feels like the original problem (how to access response headers) still ultimately remains a gap in the EleventyFetch project.