Add access to StaticHttp data in more places

[dillonkearns]

One thing I'm thinking about right now is changing generateFiles to have access to StaticHttp.Request data.

I'm feeling pretty good about that direction because you can share StaticHttp.Requests by just extracting them in your Elm code. For example, you might have:

type alias Event = { title : String, {- ... -} }
Event.staticRequest : StaticHttp.Request (List Event)

Then re-use the same one in generateFiles, or in your view function for specific pages (or all pages).

One interesting thing about StaticHttp.Requests in the context of generateFiles is that the raw responses don't end up in your final bundle, only the resulting files that are generated. So there's no need to make optimized requests there (so if you wanted to do StaticHttp.unoptimizedRequest in that context, it wouldn't be any different).

Access to StaticHttp.Request data for static results

There are currently 2 other places where we can just use StaticHttp.Requests as a helper to fetch data to get your final results, but the responses are not bundled into your data, only the final result is.

Pages.Platform.application {
  - ...
    , canonicalSiteUrl : String
    , manifest : Pages.Manifest.Config pathKey
    , generateFiles :
        List
            { path : PagePath pathKey
            , frontmatter : metadata
            , body : String
            }
        -> List (Result String { path : List String, content : String })
}

So giving these access to StaticHttp data could look like this.

Pages.Platform.application {
  - ...
    , canonicalSiteUrl : StaticHttp.Request String
    , manifest : StaticHttp.Request (Pages.Manifest.Config pathKey)
    , generateFiles :
        List
            { path : PagePath pathKey
            , frontmatter : metadata
            , body : String
            }
        -> 
        StaticHttp.Request (List
             (Result String { path : List String, content : String })
        )
}

Access to StaticHttp data in more dynamic places

I think it makes sense to have access to StaticHttp data in dynamic places.

It could be in init, or in update, or in onPageChange (or some combination). A few things to consider:

• init is called for all pages, so any StaticHttp.Requests in that context would be global data (need to download for any page you visit, whether it's used there or not)

Giving onPageChange access to StaticHttp.Request data

If we give onPageChange access to StaticHttp data, then you can access data for a specific page (and potentially fetch different data based on the PagePath and the page's metadata)

We would also need to change the arguments so that you don't have access to the query and fragment when you make the StaticHttp.Request, since that's dynamic and not known at build-time.

Currently, this is the function signature:

    , onPageChange :
        { path : PagePath pathKey
        , query : Maybe String
        , fragment : Maybe String
        }

Giving input

This is an early stage idea and I'm still doing some thinking on the design. Ideas and use cases are welcome!

I'm also going to be working on a builder-style way to add to your Pages.Platform.application config to build it, adding one piece at a time. So that design could work nicely with adding StaticHttp data in more places because we could have helpers for building that have StaticHttp data, or that don't.

[asianfilm]

The onPageChange access would be important to me.

My film festival website is built as an Elm SPA with Airtable as a backend. It's hosted on Netlify.

As part of the "build" step, a bash script downloads Airtable data into multiple JSON files, one for each edition of the festival. The bash script uses the command-line jq processor to hide secret data and merge exported data. The result is a 2019.json file, a 2018.json file, etc.

I'd like to drop the bash script, and move to elm-pages which could download the Airtable data at build stage, merge it, and strip out the JSON that my decoders ignore. And, most importantly, it won't push a broken version of the website to Netlify if my decoders fail on bad or incomplete data.

I've also written robust scripts in Elixir to export the JSON data from Airtable, which Netlify should be capable of running at build stage. These scripts do a lot more post-processing. Perhaps an Airtable plug-in for elm-pages (in Elm) would be useful for throttling, handling data that exceeds the 100 record batch limit, with helpers for its image format, and merging data from related tables.

Returning to onPageChange access.

Currently, my SPA only loads and decodes the current edition's data (from 2018.json, etc) when it visits the relevant route. Each edition's data is cached in the model to ensure that each edition's data is loaded and decoded once per browsing session. So, if one visits the 2018 guest page and then the 2018 film page (which has a different route), it will only download the 2018 data once.

Each edition's data averages about 200KB in size. It's currently about 1.5MB in total, which I don't want to spend time or memory loading and decoding when the application starts. However, I may want to load and decode a stripped down multi-year database at application start for site-wide search, and elm-pages could filter the data to the minimal fields by using paired-down decoders.

Here are examples of URLS that loads and decode different edition data from different JSON files: https://www.nyaff.org/nyaff19/films https://www.nyaff.org/nyaff18/films

Here, only the first visit to a 2017 link loads and decodes the JSON data: https://www.nyaff.org/nyaff17/films https://www.nyaff.org/nyaff17/guests https://www.nyaff.org/nyaff17/schedule

(Although you may need to load the URLs in the same browser tab for the caching to work!)

[dillonkearns]

Hello @asianfilm! That's very cool, thanks so much for sharing all these details from your use case. It's nice to hear about it, and it makes discussing these types of things a lot easier! Also, I've been thinking Film Festivals would be a really cool use case for elm-pages, so I would love to see that happen!

I believe I have some good news for you. If I understood your situation correctly, you only need access to that data in your views. So you can already do that! And you don't need to use onPageChange to manually load the data. elm-pages already takes care of exactly what you described for you! It will only run the decoder for exactly the data you use to build the page you're on.

Give it a try and let me know how it goes! You can do a simple demo by doing something like this in your view function:

view siteMetadata page =
    if page.path == Pages.pages.page1 then
        -- StaticHttp request for the data for one page... you can just log call `Debug.toString` on it in your view
    else
     -- Statichttp request for the other page

You could have a content folder like this:

content
- page1.md
- page2.md

If you have any questions about that, I'd be happy to help in the #elm-pages channel!

[asianfilm]

Thanks. I'll give it a try and reach out on Slack if I have any issues.

I agree that elm-pages would be a good fit for film festivals, especially as many use third-party sites (or venue partners) for ticket sales. On the data side, Airtable has proven a good fit for us, since it's one of the few backends that non-developers seem to have the confidence to edit.

[stefanwouldgo]

I really think having access to StaticHttp data in init is crucial. I need to initialize my model with some of that data. Is there another good way of doing that apart from init? It might be a page/template specific init.

[gregziegan]

hi! I'm trying to use environment-specific config (DataSource.Port.get "environmentVariable") in my Shared.elm#init() function. I assume this would be a breaking change, but would it be possible to expose an alternative init() with Data as a first argument like Shared#view()?

[dillonkearns]

Ah yes, indeed, that makes a lot of sense to expose the Shared.Data as part of Shared.init (analogous to how Page Modules do that) 👍 Thanks for the suggestion! I created a separate issue to track that here: https://github.com/dillonkearns/elm-pages/issues/242

[dillonkearns]

I think this issue has been addressed with the v3 release. If there are any missing places where it needs to be exposed lets open separate issues to track individual requests for that. Thanks all!