Cross-referencing global data in js data files

[stevenk99]

Can I ask anyone's feedback of referencing eleventy global data in one JavaScript data file from another. By way of a much simplified made-up example, I've used two global data files, to represent football divisions and teams:

_data/divisions.js - nosql array of all divisions

// returns [{"id":1,name:"Premiership",{"id":2,name:"Division 2"},...]
const fetch = require('node-fetch');
module.exports = async function() {
    let resp = await fetch('/_api.asp?a=divisionsget');
    return await resp.json();
}

_data/teams.js - nosql array of all teams

// returns [{"id":1,name:"Liverpool",divid:1},{"id":2,name:"Arsenal",divid:1},...]
const fetch = require('node-fetch');
module.exports = async function() {
    let resp = await fetch('/_api.asp?a=teamsget');
    return await resp.json();
}

Now I understand it is possible to 'cross-reference' global data in a layout, for example, using a filter, say, to pick out the relevant team's division:

filters/findinarray.js

module.exports = function(arr, propkey, propval) {
    return arr.find(item => item[propkey] === propval);
}

.elevently.js

module.exports = function(eleventyConfig) {
    eleventyConfig.addFilter("findinarray", require("./filters/findinarray"));
}

teams.liquid - eg "Liverpool play in the Premiership"

{% assign this_team_div = divisions | findinarray: "divid", team.divid %}
{{ team.name }} play in the {{ this_team_div.name }}

but something similar in the data file does not work, throwing a reference error "divisions is not defined":

_data/teams.js

const fetch = require('node-fetch');
module.exports = async function() {
    let resp = await fetch('/_api.asp?a=teamsget');
    let data = await resp.json();
    data.map(team => {
        team.division = findinarray(divisions, "divid", team.divid);
        return team;
    });
    return data;
}

This is likely obvious to you why it fails, but I'm not sure. Maybe it is possible somehow, with a require or something. The filter itself is not important, its the access to the other global data I want to understand.

So to get round this, we can throw the data 'sets' into a node module to inter-relate the data:

mymodules/football.js

const fetch = require('node-fetch');
const getdivisions = async () => {
    let resp = await fetch('/_api.asp?a=divisionsget');
    let divs = await resp.json();
    return divs ;
};
const getteams = async () => {
    let resp = await fetch('/_api.asp?a=teamsget');
    let teams = await resp.json();
    return teams;
};
exports.getdivisions = getdivisions;
exports.getteams = getteams;

_data/divisions.js -

// each division now includes the teams that belong to it
const football = require('../mymodules/football.js');
module.exports = async function() {
    var divisions = await football.getdivisions();
    var teams = await football.getteams();
    divisions.map(div => {
        div.teams = teams.reduce(team=>team.divid=div.id);
        return div;
    });
    return divisions;
}

_data/teams.js

// each team now includes its division
const football = require('../mymodules/football.js');
module.exports = async function() {
    var teams = await football.getteams();
    teams.map(team => {
        team.division = findinarray(divisions, "divid", team.divid);
        return team;
    });
    return teams;
}

Instead of modules, could we place all data into one global data file that every layout refers to, which might mean fewer calls in a pagination build perhaps?:

_data/footballdata.js

const fetch = require('node-fetch');
module.exports = async function() {
    let resp = await fetch('/_api.asp?a=teamsget');
    let teamdata = await resp.json();
    let resp2 = await fetch('/_api.asp?a=divisionsget');
    let divdata = await resp2.json();
    return { "teams": teamdata, "divisions: divdata }
}

What method(s) would you recommend to inter-link global data, principally for more complex models than this example? For example, is the api called unnecessarily and repeatedly when building multiple html files using pagination? I wonder if there is some caching solution. Any thoughts, alternatives, recommendations or feedback would be appreciated.

[NVolcz]

Did you find a solution for your question? I am facing the same challenge

[stevenk99]

NVolz In the end I spent a lot of time preparing all the data from mongoDB to create a single master data store, before feeding it into 11ty. This store consisted of, say, a dozen main object arrays (Seasons, Divisions, Matches, Players etc) with helper/pointer arrays into these, for example, 'DivisionMatches' (with better DB practise, some of these could have been in mongoDB, but this can make the admin side more detailed because of the mongoDB (nosql) requirement to update an inter-table relationship in more than one table. If one knows the end result is only for SSG, it can make sense to keep it simpler in mongoDB). I had to keep on top of js object array principles and functions to avoid assignments that replicated data (increasing memory and duplication of data) and stick to reference, preferable, because the templates are then pretty much ready to go for handling pagination and filtering. Then within the templates, if a helper wasn't available, I always had access to all data, 'alldata.Seasons' for example, for filtering or sort. I suppose a good principle is that your templates read very simply and cleanly, indicating clearly your entity structure. These are familiar in your mind (and documented or on your wall) with the right mapping ready to go. You do sometimes need filtering, so I wrote some generic 11ty filters in /lib/filters, 'filterarray' for example). I've not bothered with subdir data files or any of that, just a few filters used occasionally. Its working well, I use a low cost node provider for the admin side, photos on s3, then netlify runs through the 11ty app data. One league has seasons back to 2008, about 3 divisions/season and 6 teams/division playing home and away, with variable and complicated scoring systems. Netlify crunches out pages for each season, team, player stats tables, in 90s or so. During the covid years, I moved away from monolith (sql server, .net, 'live' dedicated servers) to node, js, mongoDB and I'm so glad I did, as it has given a fool-proof, rock solid, very low cost, secure system that works fantastically for clients who require data/informational websites. It took a long time though.