Unable to access index from getDataSlice in custom plugin; "not found" error

[dnotes]

I can't seem to write a getDataSlice function that is aware of the index. Below is my attempt, based on the devto plugin, but I've probably missed something simple.

Even if this worked, I'm not sure how I would access the index if I were writing a one-off plugin in exports.plugins, so I wonder: would you be open to including the requested index as a parameter in the call from readConfig.getDataSlice to the getDataSlice plugin function?

const { slugify } = require('./helpers')
const fs = require('fs')
const matter = require('gray-matter')
const normalMD = require('markdown-it')('commonmark')
  .enable('table')
  .use(require('markdown-it-attrs'))
  .use(require('markdown-it-footnote'))

module.exports = function(options) {
  let opts = Object.assign({
    folders: false,
    section: '',
    md: normalMD,
  },
  options || {})
  if (typeof opts.folders === 'string') opts.folders = [opts.folders]

  let allPosts = []

  async function createIndex(idx) {
    idx.ssgCoreData.filter(o => {
      o.type = o.metadata.type || o.shortFilePath.split('/')[0]
      if (o.type.indexOf('.') > -1) o.type = 'page'
      return !opts.folders || (opts.folders.indexOf(o.type) > -1)
    }).forEach(o => {
      if (!o.metadata.slug && o.metadata.title) o.metadata.slug = slugify(o.metadata.title)
      o.slug = o.metadata.slug || o.slug
      o.href = o.metadata.href || [
        (opts.section || o.metadata.section), // section (music)
        (o.metadata.type || o.type),          // type (song)
        o.slug                                // slug (the-earth-and-the-eagle)
      ]
        .filter(Boolean)
        .join('/')
        .replace(/^page\//, '')

      allPosts.push(o)
    })
    return allPosts // <---------allPosts contains data
  }

  async function getDataSlice(slug, coreDataPlugin) {
    let item = allPosts.find(i => i.slug === slug) // <---------allPosts is empty
    if (item) {
      let rawData = await fs.readFile(item.filePath)
      let fileData = matter(rawData)
      fileData.html = opts.md.render(fileData.content)
      return fileData
    }
  }

  return {
    createIndex,
    getDataSlice,
  }
}

[swyxio]

ok this took me a while to understand too, haha. the reason allPosts is empty is because you cant share it between invocations of ssg.config.js under the Sapper model. this is one reason (there are other reasons) i split the data fetching into a createIndex stage and then a getDataSlice stage.

i think whats going on ssg.config.js is called in TWO phrases - once when getting the main index, then every time a slice is required from Sapper. throw in a console.log at the start and it should be called twice. so allPosts doesn't persist between calls specifically of custom plugins inside ssg.config.js. perhaps a design flaw that i never got round to figuring out, bc it works fine as an node module.

i think throwing in the index makes sense. tbh i dont know why i didnt do that when i made it, seems like a very reasonable thing to do.

this is a tricky time for this project. i intend to do a total rewrite, so idk whether we should change this now. if its important to you, i can take a PR, or else you can feel free to fork it.

[dnotes]

OK that makes sense, but in that case I don't understand how the devto plugin is functioning for your other sites! —and it does seem to be functioning... but anyhow, I also see your point about working on a full rewrite. I have some ideas along those lines as well. For example, I'd love to be able to hook into the createIndex function at a far more basic level; I'd love something like a preIndex function that would just get the file stats for every file in the content directory, and from there they would be placed into the index (or not) based on a chain of rules that can be configured, preferably with declarative json or yaml, or supplemented with new rules where necessary. That way one wouldn't have to write a whole separate indexing function for different file types, but basically just a handler. It seems like this would save on both boilerplate and filesystem calls. Anyhow, I'm not sure if I'll be able to spend any time fleshing these ideas out as I'm moving continents this month, but man I just love your addition of an index / render paradigm to sapper, and I have so many ideas for it. :)

On Thu, Jul 30, 2020 at 12:08 PM swyx notifications@github.com wrote:

ok this took me a while to understand too, haha. the reason allPosts is empty is because you cant share it between invocations of ssg.config.js under the Sapper model. this is one reason (there are other reasons) i split the data fetching into a createIndex stage and then a getDataSlice stage.

i think whats going on ssg.config.js is called in TWO phrases - once when getting the main index, then every time a slice is required from Sapper. throw in a console.log at the start and it should be called twice. so allPosts doesn't persist between calls specifically of custom plugins inside ssg.config.js. perhaps a design flaw that i never got round to figuring out, bc it works fine as an node module.

i think throwing in the index makes sense. tbh i dont know why i didnt do that when i made it, seems like a very reasonable thing to do.

this is a tricky time for this project. i intend to do a total rewrite, so idk whether we should change this now. if its important to you, i can take a PR, or else you can feel free to fork it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/sw-yx/ssg/issues/24#issuecomment-666276044, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADXEEVZGT3DNROUYJX3DQDR6FBDLANCNFSM4PMLTNLQ .

[swyxio]

sure, np. fwiw i intend to drop sapper, bc it has race conditions and i dont need the clientside routing :)