Sitemap randomly misses urls

[blugri]

The sitemap generated by keystone-express-sitemap is alway missing random urls. The urls that are missing changes over time.

Used implementation:

// Setup Route Bindings
exports = module.exports = function(app) {
    // Sitemap
    app.get('/sitemap.xml', function(req, res) {
            sitemap.create(keystone, req, res);
        });

    // Views
    app.get('/', routes.views.index);
    app.get('/about', routes.views.about);
    app.get('/app/:app', routes.views.app);
    app.get('/blog/:category?', routes.views.blog);
    app.get('/blog/post/:post', routes.views.post);
    app.get('/cookiepolicy', routes.views.cookie);
    app.get('/privacy/:language', routes.views.privacy);
};

[s-plum]

Can you please provide examples of URLs that go missing? Are there any URLs that are never missing?

[blugri]

It looks like the root url, about and cookiepolicy are never missing (i.e. the urls without paramaters). All the other type of urls have missing ones.

I cannot find a pattern in it.

It does not change on every request. But from time to time the urls are changing. Without any changes (code and content) to the site I had these results:

monday: 15 urls tuesday: 35 urls wednesdag: 28 urls

Cheers Tom

[s-plum]

A few questions to help debug:

• Are you using the default settings for MongoDB with Keystone, or do you have any customizations?
• Does the sitemap always vary by day, or do you see the variations with individual refreshes on the same day?
• What are the models and properties of those models in your app?

[blugri]

Hi,

sorry for the late reply.

• The set up is not quite an out of the box setup. I use docker containers and I have a docker container for mongodb and another one for my keystonejs project.
• I cannot say how often the sitemap changes. All I can say is that it is not on each refresh.
• The models are nothing special, only text and cloud images. An example is:

`var keystone = require('keystone'); var Types = keystone.Field.Types;

/**

• Post Model
• ========== */

var Post = new keystone.List('Post', { map: { name: 'title' }, autokey: { path: 'slug', from: 'title', unique: true } });

Post.add({ title: { type: String, required: true }, state: { type: Types.Select, options: 'draft, published, archived', default: 'draft', index: true }, author: { type: Types.Relationship, ref: 'User', index: true }, publishedDate: { type: Types.Date, index: true, dependsOn: { state: 'published' } }, image: { type: Types.CloudinaryImage }, content: { brief: { type: Types.Html, wysiwyg: true, height: 150 }, extended: { type: Types.Html, wysiwyg: true, height: 400 } }, categories: { type: Types.Relationship, ref: 'PostCategory', many: true } });

Post.schema.virtual('content.full').get(function() { return this.content.extended || this.content.brief; });

Post.defaultColumns = 'title, state|20%, author|20%, publishedDate|20%'; Post.register();`

Cheers Tom

[lomegg]

I can confirm that this is happening indeed. A lot. Had to switch back to static sitemap because can't afford to have 70% of the urls missing randomly.

[lomegg]

Strangely that only happening in production. For example,

filters: {
    'Page': function(page) {
        console.log('page', page.title.en, 'state:', page.state, 'returning', page.state == 'published');
        return page.state == 'published';
    }
}

The console.log was never called in production, though some of the Page related urls were processed.

[rodoflho]

I also confirm. But in my case, all the dynamic URL's are missing, excepting one. I can assure I followed the guidelines. To get sitemap ok again, i have to restart server, but after some time the problem comes back.

Unfortunately, I can't recommend this Library. Instead, use npm sitemap and add your urls manually.