Sitemaps still creating auto generated urls, even when a function is provided to create our own from an API

[toddpadwick]

Describe the bug

I am using an async function to generate our sitemap URLs. we do not want to use the pre-render crawling feature. But it still seems to be generating an auto-created sitemap instead of from the function:

Our config:

import { getSitemapUrls } from './utils/sitemaps';

export default defineNuxtConfig({
  sitemap: {
    sitemaps:getSitemaps()
  },

and here is the getSitemaps() function:

export const getSitemaps = async () => {
  console.info('Fetching sitemaps');
  let sitemaps = {
    about: {
      urls:[
        `${baseUrl}/about`
      ]
    },
    careers:{
      urls:[
        `${baseUrl}/careers`
      ]
    },
    contact:{
      urls:[`${baseUrl}/about`]
    }
  }

  const data = await client.fetch(sitemapQuery);
  if (data && data.length) {
    console.info("sitemaps generated");
    data.forEach(section => {
      if (!section.name) return;
      sitemaps[section.name] = {
        urls:section.entries?.map(entry => `${baseUrl}${section.basePath ? section.basePath : '' }${entry.category?.slug?.current ? '/'+entry.category.slug.current : ''}/${entry.slug?.current}`)
      }
    })

  } else {
    console.info("No sitemaps found, setting basic sitemaps only");
  }
  console.log(sitemaps);
  return sitemaps;
};

The above function generates something like this:

{ about: { urls: [ 'https://localhost:3000/about' ] },                                                                                                                                      18:08:50
  careers: { urls: [ 'https://localhost:3000/careers' ] },
  contact: { urls: [ 'https://localhost:3000/about' ] },
  blog:
   { urls:
      [ 'https://localhost:3000/blog/work/antler_deep_dive',
        'https://localhost:3000/blog/news/gramafilm-goes-green',
        'https://localhost:3000/blog/news/gramafilm-premiere',
        'https://localhost:3000/blog/work/christianahjones',
        'https://localhost:3000/blog/work/neighbourhood-botanicals',
        'https://localhost:3000/blog/news/screenprinting',
        'https://localhost:3000/blog/news/article-demo',
        'https://localhost:3000/blog/work/bees-and-refugees',
        'https://localhost:3000/blog/news/femaleledproduction',
        'https://localhost:3000/blog/news/gramanews-issue-16' ] },
  work:
   { urls:
      [ 'https://localhost:3000/work/red-bull-beneath-the-surface',
        'https://localhost:3000/work/neighbourhood-botanicals',
        'https://localhost:3000/work/youtube-music-bespoke-campaigns',
        'https://localhost:3000/work/youtube-music',
        'https://localhost:3000/work/bitpanda-ios-launch',
        'https://localhost:3000/work/content-demo',
        'https://localhost:3000/work/red-bull-f1',
        'https://localhost:3000/work/cooking-fever',
        'https://localhost:3000/work/siemens-x-intel',
        'https://localhost:3000/work/android-developer-stories',
        'https://localhost:3000/work/christianahjones',
        'https://localhost:3000/work/allegran-botox',
        'https://localhost:3000/work/youtube-100-voices',
        'https://localhost:3000/work/nakheel',
        'https://localhost:3000/work/google-playtime-2019',
        'https://localhost:3000/work/way-back-home-with-danny-macaskill',
        'https://localhost:3000/work/android-developer-challenge',
        'https://localhost:3000/work/youtube-stayhome',
        'https://localhost:3000/work/don-t-look-down',
        'https://localhost:3000/work/lg-styler',
        'https://localhost:3000/work/intel-the-tempest',
        'https://localhost:3000/work/bafta-campaign',
        'https://localhost:3000/work/red-bull-barnstorming',
        'https://localhost:3000/work/twitter-behindthecode',
        'https://localhost:3000/work/android-10-year',
        'https://localhost:3000/work/teoxane-treatments',
        'https://localhost:3000/work/youtube-creative-economy',
        'https://localhost:3000/work/entyvio',
        'https://localhost:3000/work/amazon-embraceequality',
        'https://localhost:3000/work/antler-luggage',
        'https://localhost:3000/work/google-animations',
        'https://localhost:3000/work/amazon',
        'https://localhost:3000/work/google-i-o-opener',
        'https://localhost:3000/work/android-imakeapps',
        'https://localhost:3000/work/google-playtime-2018',
        'https://localhost:3000/work/twitter-buildwhatsnext' ] },
  services:
   { urls:
      [ 'https://localhost:3000/services/localisation',
        'https://localhost:3000/services/film',
        'https://localhost:3000/services/design',
        'https://localhost:3000/services/brand-experiences',
        'https://localhost:3000/services/social',
        'https://localhost:3000/services/photography',
        'https://localhost:3000/services/event' ] },
  partners:
   { urls:
      [ 'https://localhost:3000/partners/intel',
        'https://localhost:3000/partners/amazon',
        'https://localhost:3000/partners/redbull',
        'https://localhost:3000/partners/google',
        'https://localhost:3000/partners/youtube',
        'https://localhost:3000/partners/dbb-remedy',
        'https://localhost:3000/partners/android',
        'https://localhost:3000/partners/bbc',
        'https://localhost:3000/partners/twitter' ] },
  legal:
   { urls:
      [ 'https://localhost:3000/legal/cookie-policy',
        'https://localhost:3000/legal/privacy-policy' ] } }

Reproduction

No response

System / Nuxt Info

System:
    OS: macOS 11.7.4
    CPU: (16) x64 Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
    Memory: 1.20 GB / 32.00 GB
    Shell: 3.2.57 - /bin/bash
  Binaries:
    Node: 14.19.0 - ~/.nvm/versions/node/v14.19.0/bin/node
    npm: 6.14.16 - ~/.nvm/versions/node/v14.19.0/bin/npm
  Browsers:
    Chrome: 114.0.5735.133
    Safari: 16.3
    Safari Technology Preview: 15.4

[harlan-zw]

Hey @toddpadwick, thanks for the issue.

You can look at disabling inferStaticPagesAsRoutes, I'll need to add a new config to skip the prerender discovery.

Can I ask the use case though? The only time I know this is useful is for i18n when you're overriding the routes.

[toddpadwick]

Hey @harlan-zw thanks for responding so quickly. I have tried the prerendering discovery and although its awesome, it just doesn't have the control I need. Also, unless I'm mistaken, it doesn't seem to pick up all entries, such as ones that are not linked anywhere. I know in general, every page should be linked somewhere for SEO and for static generation, but sometimes the client doesn't do this. so I treat the sitemap as a failsafe so ensure they are still crawlable. I noticed, for example, when I query the CMS for generating URLs, I get 120 or so. but the auto generated discovery returns 96. So this is also a useful thing to refer back to the sitemap as the source of truth for the client to ensure all pages are linked too.

regarding your suggestion, does this require the config update to the module before I can action it?

[lud-hu]

Hi @harlan-zw, Hi @toddpadwick,

I've run into the same issue. We want to create the sitemap only from a list of dynamic URLs (via a request to our CMS). This might also include URLs not visible from the static generated pages, and might also exclude pages that are visible from the static generated pages.

Therefore I came up with sth like this: https://stackblitz.com/edit/github-segxhh-l6rytt?file=nuxt.config.ts But also with having the server/api/_sitemap-urls.ts script as well as the inferStaticPagesAsRoutes: false config in nuxt.config.ts in place, it does still include the unwanted statically found urls like https://test.com/another.

<url>
        <lastmod>2023-10-25T06:36:53+00:00</lastmod>
        <loc>https://test.com/</loc>
    </url>
    <url>
        <lastmod>2023-10-25T06:36:53+00:00</lastmod>
        <loc>https://test.com/another</loc>
    </url>
    <url>
        <lastmod>2023-10-25T06:36:53+00:00</lastmod>
        <loc>https://test.com/test/hello</loc>
    </url>

But I want it to look like

    <url>
        <lastmod>2023-10-25T06:36:53+00:00</lastmod>
        <loc>https://test.com/test/hello</loc>
    </url>

as seen in

export default defineEventHandler(async () => {
  return [
    {
      loc: 'test/hello',
    },
  ];
});

[harlan-zw]

Please try out the new v4 release and read the updated documentation. If you still have issues please let me know.

(You should use excludeAppSources: true)