Environment Variables

[roobyz]

Hi @jimafisk ,

I just pushed a few commits with some initial changes intended to wire up the "contact me" page page_contact.svelte.

• What: provide secure message delivery
• Why: enable page visitor to send a message via a contact request
• How: Use "environment variables" to security provide API_KEYs and trigger sendgrid API function
• Expected outcome: function will securely load environment variables and execute API call

I feel like this is my remaining step to completing a full JAMStack site, but I am missing something. Currently I am using Cloudflare pages to host my site. I setup some "environment variables" on my Coudflare Pages settings, which I attempt to access in my sendGrid.svelte script, but I get an undefined error.

What would you recommend? Is there a best practice approach I should be taking? Is there a generic approach that would work regardless of hosting provider?

Thanks!

[jimafisk]

Hey @roobyz,

This is a great idea, just to make sure I fully understand though, which site is currently deployed to Cloudflare pages, is it https://www.quantifiedleap.com/pages/contact or a different URL? I'd imagine that the version on GitHub pages (https://plenti-themes.github.io/compendium/pages/contact) would not have access to those environment variable, right?

I know Netlify and other hosting providers have similar features for env variables and proxying requests to keep keys secure, but I've always thought it would be nice if there was a cheap, easy, (open source) platform for doing this on generic hosting environments like GitHub / GitLab Pages, S3, or whatever...

[roobyz]

Hi @jimafisk, Yes, sorry I missed that detail... my site currently deployed to Cloudflare pages is https://www.quantifiedleap.com/pages/contact. I imagine that the local environment variables used for development would be the same and would work similarly in production on Cloudflare. I'm just not sure how to connect the dots. Seems like there is a load or API step that is missing?

[jimafisk]

Yes, you're totally right. Funny enough, I was thinking about this when shoveling my driveway earlier today. There are certain exposed values like baseurl in the sitewide plenti.json configuration file that come through a "magic" prop (env.baseurl). However, there's no mechanism for user defined environment variables to aid with local development.

Would writing simple key/value pairs in plenti.json work, or you hoping for a separate .env file? We already have a few predefined values in the env prop so we could set this up so you access it in your templates like env.custom.MY_KEY, but maybe I should rethink how I'm using env currently so it's just env.MY_KEY.

[jimafisk]

Well I guess they'd just have to be MY_KEY in order to work in your deployed environment as well, right?

[roobyz]

I've been doing a lot of reading and found a few articles on the topic. Seems that the features are rather new and haven't been well documented. I have a sneaking suspicion that "environment variables" are only available for "build steps."

There is a svelte adapter, which indicates that Cloudflare has something called "durable objects" which can be used to setup "environment variables" that would be referenced by platform.env. https://github.com/sveltejs/kit/tree/master/packages/adapter-cloudflare

There is a lessons learned blog post which indicates that KV and Durable Objects bindings are not easily accessible from serverless functions, and must be declared "in the functions directory which is managed and built by Pages, outside of your framework project." https://dancowell.com/blog/2022-01-09-building-this-site-with-cloudflare-pages

A third post indicates that I can trigger a Cloudflare Worker to run the Sendmail function, but there are some details missing. https://blog.amanbhargava.com/send-email-using-cloudflare-worker ... I was thinking that I might need to develop and API endpoint to trigger the cloudflare worker somehow. :-)

[roobyz]

Hi @jimafisk, I found a breadcrumb. A Cloudflare project that I found on Github includes some sendmail functionality, It contains a Javascript file cfw.js in the root path, which lays out global variables in the following way. Apparently Environment Variables are called binding, and there are secret binding as well (i.e SENDGRID_TOKEN).

// @ts-ignore - @types/node
const ENV = process.env;

/** @type {import('cfw').Config} **/
module.exports = {
    entry: 'index.ts',
    profile: 'workers.demo',
    name: 'cms-api',
    routes: [
        'api.ley.dev/*'
    ],
    globals: {
        SENDGRID_TOKEN: `SECRET:${ENV.SENDGRID_TOKEN}`,
        SENDGRID_EMAIL: 'ENV:email@addr.com',
        SENDGRID_NAME: 'ENV:CONTACT_FORM',
    }
};

Where would I include this into my project? Also, I'm thinking that I only need to globals section. Thoughts?

[jimafisk]

Thanks for researching this @roobyz! Are you able to share a link to the repo this snippet was pulled from so I can poke around a bit? Just want to make sure I'm setting this up right.

[roobyz]

hi @jimafisk,

I checked the Cloudflare discord and asked for clarification. The discord response indicated that, rather than exporting the environment variables, I should use an onRequest call, which leverages Cloudflare Functions, where:

export async function onRequest(context) {
  // Contents of context object
  const {
    request,    // same as existing Worker API
    env,        // same as existing Worker API
    params,     // if filename includes [id] or [[path]]
    waitUntil,  // same as ctx.waitUntil in existing Worker API
    next,       // used for middleware or to fetch assets
    data,       // arbitrary space for passing data between middlewares
  } = context;

  return new Response("Hello, world!");
}

Based on this example it looks like my plenti output needs to look something like:

├── functions
│   └── api
│       └── submit.js
└── public
    └── index.html

Where the api endpoint (submit.js) would become /api/submit, with logic that looks something like:

export async function onRequestPost({request, env, data}) {
  try {
    let input = await context.request.formData();
    const ADDR = {
      email: env.SENDGRID_EMAIL,
      name: env.SENDGRID_NAME,
    };

    // maybe use the data object?
    // logic from sendGrid.svelte would live here

  } catch (err) {
    return new Response('Error parsing JSON content', { status: 400 });
  }
}

So basically this would be an API Proxy, where instead of my existing svelte logic sending the POST request directly to sendGrid, it would send it to /api/submit. The request handler (submit.js) would collect and parse the submitted form data and deploy a Worker, which then triggers the POST call to sendGrid using the env variables. This is where my brain explodes. :-)

[roobyz]

hi @jimafisk ,

I altered my plenti.json build to: "build": "deploy/public",. Then I added a submit.js file to deploy/functions/api/ so that my output folder (deploy) looks like:

├── functions
│   └── api
│       └── submit.js
└── public
    └── index.html

Then on Cloudflare I changed the "Build output directory:" to /public and on the next commit, the build process picked up the changes and automatically created the "Cloudflare Function" with this configuration:

{
  "routes": {
    "POST /api/submit": {
      "module": [
        "api/submit.js:onRequestPost"
      ]
    }
  },
  "baseURL": "/"

So now, I need to write a function that actually works and fingers-crossed I should have access to the environment variables via the "Cloudflare Worker" that is triggered by the "Cloudflare Function". BTW, unlike other serverless function services, Cloudfare uses sandboxed "Chrome V8" instances rather than Node instances, which makes them lighter, faster and "more secure". This also means that we can't use Node inside them.

[roobyz]

hi @jimafisk

I figured it out! Definitely need to write an article describing the journey. In summary:

1. My last comment was only the first step (i.e. getting the functions and public folders to be sibling folders in Cloudflare and then adjusting the "Build Output Directory" setting)
2. After my handleOnSubmit function passes the form details object to the send_contact function, the object need to be urlencoded for compatibility with the Cloudflare Functions endpoint.
3. After the send_contact function runs the fetch POST of the encoded data to the API endpoint, we need to convert the encoded data back into an object and then transform it to a SendGrid-compatible JSON, which includes the API environment variables.

Basically it was a bit "too magic"... there were these little configuration-related issues that made it difficult to know whether the code was correct or not. Also, the wrangler 2.0 command works with Cloudflare Functions, but 1.0 does not. Finally getting the environment variables to work locally was a pain.

I'll push the compendium code out soon. This gets the theme to be fully functional. Still need some clear documentation, ARIA attributes, and maybe some more SEO tweaks.

Cheers, Roberto

[jimafisk]

Awesome job figuring this out @roobyz! I'm curious how you were able to get environment variables working since Plenti doesn't have support for that. If there are ways to make this easier going forward I'd be interested in your thoughts. Congrats on all the progress on Compendium, it's definitely the most comprehensive Plenti theme so this is all super exciting to hear!

[roobyz]

@jimafisk ... thanks and good question. I've basically integrated Plenti, Node, and Wrangler together as a "unified" toolchain. ;-)

My Process:

1. My .env file contains the same "environment variables" that I have in my Cloudflare Pages settings
2. My plenti.json sets the plenti output to a deploy folder "build": "deploy/public"
   • I use the standard plenti commands to develop, build and design my site
   • plenti build is always the last step before I publish
3. My Node package.json has a few scripts to handle all the repetitive stuff
   • npm run build: handles the production-ready "plenti build" step and updates my functions to "deploy/functions"
   • npm run preview: executes... npx wrangler pages dev deploy/public --binding $(cat ./.env), which
     • reads the wrangler.toml file to access the Cloudflare API and settings
     • binds the local environment variables to the local Cloudflare context
     • creates a local preview of the site using the content of both the public and functions folders from the deploy folder
     • Note: I am running plenti serve simultaneously with npm run preview on different terminals; this way Svelte changes will auto-compile and my functions changes will autocompile; so I only need to hit refresh on the browser when my Svelte code changes
4. My deploy folder is actually another git repo that I push to "Cloudflare Pages"
   • once all my changes are tested and validated I run another npm script
   • npm run publish: commits and pushes my deploy repo to github
   • Cloudflare triggers the "build" process on changes to main, which basically just publishes the changes of the deploy repo since all the compile steps happen locally

Thoughts/Recommendations:

• it would be ideal to have Plenti compile Svelte code into an api endpoint; I wrote the JS by hand
  • in other words, the output would be a JS file that is compiled to the deploy/functions folder; maybe have a setting in plenti.json like "endpoint": "deploy/functions" to work with different providers
  • I'm thinking this might work similar to how I did "tags", where the content of "{}" would be in an api folder and the svelte file would live in an api sub-folder in layout
• could be good to get Plenti integrated with Cloudflare Pages... they have a drop down for selecting the build tool (i.e. site generator) like Hugo, Gatsby, etc.
• might be nice to enhance the pagination functionality to optionally associate a name to the page index; this way the page number could show up for standard pages and names could show for tag and category pages; I'm thinking you would need a component key too, so that depending on the component type, it would associate the correct indexes
• could be good to look at what Cloudflare is doing with wrangler... it is pretty slick and includes the --binding parameter

Open to feedback on ways to improve as well. ;-)

Cheers, Roberto

[roobyz]

hi @jimafisk

I've updated the theme, which is now actually a complete MVP with a README file that touches on how I got everything to work. Please have a look and let me know if you have any feedback. After implementing some basic WAI-ARIA roles, I ran it through the W3C HTML Validator and completed some refactoring/debugging. The 0.5.1 version of Plenti took care of the "Quirks Mode" issue, so now the validator output is clean.

Cheers, Roberto

[jimafisk]

That's amazing @roobyz congrats! I just took a like at https://plenti-themes.github.io/compendium/ because the automatic deploys seem to be working and it looks great! Did notice one small issue with pagination on the homepage:

Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'citation')

See screenshot


By the looks of it, the pagination is going to https://plenti-themes.github.io/2 when it should be https://plenti-themes.github.io/compendium/2, so the pager probably just needs to be updated to use a relative link so it respects the baseurl. Let me know if you need any help looking into that!

[roobyz]

Great catch @jimafisk ...I know how to fix it. Working on it. ;-)

[roobyz]

@jimafisk ... fixed the pagination issue. Forgot to account for the baseurl. ;-)

Any other feedback?