💡 RFC: Non-HTML dynamic files

Tc-001 commented 3 years ago

Background & Motivation

There are many reasons to want custom, dynamic files. One of the primary contenders are JSON feeds and other read-only APIs, because currently there is no clean way to make them. But there is also config files like .htaccess, vercel.config.json and others to for example set redirects!

Things like image optimization may also play a role (more so in #965, but there are some use cases here too), with something like a frequently changing logo (think google doodles) fetched and processed seperately from the page itself.

Proposed Solution

Possible solutions

Sveltekit endpoints. To generate a dynamic file example.com/api/feed.json In sveltekit you would have:

//   ./src/pages/api/feed.json.ts
export async function get() {
  var articles = Astro.fetchcontent("/blog/**/*.md").map(article => /* */)
  return {
    body: {
      JSON.stringify(articles)
    }
  };
}

Can be changed to a more astro-specific API, because headers in SSR... (they could be done with a custom config file though!)

export async function get() {
  return "hello world"
}

export async function get() {
  const image = await fetch("example2.com/images/dynamic-logo.php").then(x => x.buffer())
  return image //also does buffers
}

By adding a .js or .ts extension, you make it possible to create any file type with the desired content

Alternatives considered

Discussed in #965, but those would rely on low level components (cough snowpack cough) to make them work correctly

Risks, downsides, and/or tradeoffs

User-controlled php file (or just injection attacks in general)
May be confusing to someone not familiar with sveltekit
Open Questions
Would it theoretically be possible to use getstaticpaths to generate these files? Because Astro.props would be exposed and there should be no top-level code, and it should solve most of the usecases presented in #965 and discussed in the RFC call

Detailed Design

Go back in the git history, right click on the commit that removes the endpoint support, revert commit

-jasikpark

Some discussion about this in the last (as of writing) RFC meeting: https://youtu.be/hhsKS2et8Jk?t=1237

Help make it happen!

[ ] I am willing to submit a PR to implement this change.
[X] I am willing to submit a PR to implement this change, but would need some guidance.
[ ] I am not willing to submit a PR to implement this change.

Tc-001 commented 3 years ago

cc: @tony-sull Happy {timeofday}! I think I included everything important, feel free to discuss and/or correct

tony-sull commented 3 years ago

Nice, can't wait to see this one come together!

Can be changed to a more astro-specific API, because headers in SSR

I'm a little hesitant to change the API surface here. We don't need headers today, but they'd come in handy if we want to add support for on-demand builders or similar in the future. That said, I'm not a hard pass on the idea of simplifying it if that's the direction we go - backwards compatibility shouldn't be a big deal if we later need to change it from returning a string to an { body } object

Would it theoretically be possible to use getstaticpaths to generate these files?

That's definitely a feature I had in mind for dynamic file generation. It may not be the common case, but I could see it being very handy to be able to have one src/pages/data/[slug].json.ts that spits out a bunch of JSON files

Tc-001 commented 3 years ago

We don't need headers today, but they'd come in handy if we want to add support for on-demand builders or similar in the future.

Yeah fair, especially with the go compiler.

It may not be the common case, but I could see it being very handy to be able to have one src/pages/data/[slug].json.ts that spits out a bunch of JSON files

I actually had to get a glob of files like that to work (wiki-type article to a seperate md editor), finally gave up and had the client fetch the raw html and extract a meta tag :sweat_smile:

matthewp commented 2 years ago

I think this is a great idea! What I'd like to see before accepting is a breakdown of how other static site builders provide this functionality. I'd do some research on things other than SvelteKit, like Gatsby, Hugo, 11ty, and write up how they solve this problem, even if you don't like their solutions, just so we can discuss it together and go over the various strengths and weaknesses of this approach compared to those others.

Tc-001 commented 2 years ago

I do not have much experience with other frameworks (save for sveltekit coincidentally), but I think I can gather a quick example for each.

But if someone with more know-how about the differences between them can get one together I'm all for it!

Princesseuh commented 2 years ago

I really like how Eleventy does it, it might not be necessarily possible for Astro due to how different the architecture is but the gist of it is that Eleventy never care about what you're outputting, the way you create a json file from a template is really just putting .json at the end of the permalink

What this mean is that you can create literally anything you want, you can create css files, _headers file for Netlify, config files using Eleventy templates. You're just outputting text at the end of the day. I'd love to see this kind of flexibility in Astro

Tc-001 commented 2 years ago

So in astro land it would be similar to

---
// ./pages/foo.json.astro
const json = JSON.stringify({foo: "bar"})
---
{json}

I think that might be a feasible solution, but it will not work for non-plaintext files like images

Although there could be just a special rule to not apply anything if there is only one embed and nothing else

jasikpark commented 2 years ago

---
export default = <raw image buffer bytes>;
---

😅

Tc-001 commented 2 years ago

That... could actually work But at that point maybe it is better to have it as a js/ts file?

But I am not against just default exporting the resut

//For a simple json feed
export default JSON.stringify({url: Astro.request.url.toString()})

//SSR expansion in the future?
export default {
    headers: {foo: "bar"},
    body: JSON.stringify({url: Astro.request.url.toString()})
}

//should work with getstaticpaths too, but maybe not as nicely? Can someone with code knowhow confirm?
export async function getStaticPaths() {
   //...
}
export default JSON.stringify({Page: Astro.props.page})

That also seems to be what eleventy uses, although they seem to have a global config of sorts for this


module.exports = eleventyConfig => {
  eleventyConfig.addDataExtension("customfile", contents => "customcontentfunction");
};

Tc-001 commented 2 years ago

If getstaticpaths becomes a problem we could also have it be a function

//Astro is not a global but gets passed as a function param
export default (Astro) => JSON.stringify({url: Astro.request.url.toString()})


export default (Astro) => {
  headers: {foo: "bar"},
  body: JSON.stringify({url: Astro.request.url.toString()})
}

rebelchris commented 2 years ago

I'm using a page called search.json.astro and have a content like so:

---
const allPosts = Astro.fetchContent("./posts/*.md");
const json = JSON.stringify(
  allPosts.map((p) => {
    return {
      title: p.title,
      description: p.description,
      slug: p.url,
    };
  })
);
---
{json}

Renders a nice json file as such: https://loving-wilson-87e511.netlify.app/search.json/

Works well for my purpose of generating a SSG search (more on that in another topic)

matthewp commented 2 years ago

RFC Call September 28th

Tabling the dynamic [...files].json.ts form, not supported up front.
Astro global is not available in js/ts files for now, will work on making them available some how.
Keep { body: ... } form for now.

Tc-001 commented 2 years ago

I would love to get a working proof of concept together soon-ish.

Would this be something to tackle in the current compiler, or is it better to sort it out in the new one? Is there a difference in the ease of implementation between the two?

matthewp commented 2 years ago

@Tc-001 This should be tackled in the next branch. However, I don't think this would involve changing the compiler, but more likely just changing the routing logic.

Tc-001 commented 2 years ago

Alright, I have gotten a hacky version to work(ish). I was really hoping to show more but I do not think I can.

The main part that I did is fool ssr.ts to just echo whatever get().body returns as the content, because vite just imports any .js file in pages/ as an astro component https://github.com/Tc-001/astro/blob/b3ff7b3752b70c5ef1470b82e120f3e6759fcd17/packages/astro/src/runtime/ssr.ts#L143

I also bypassed adding foo/index.html at buildtime if it is a .js file https://github.com/Tc-001/astro/blob/b3ff7b3752b70c5ef1470b82e120f3e6759fcd17/packages/astro/src/build/index.ts#L83

While it works* on dev, there is still <head></head><body>...</body> on build. The reason for that I believe is that the pre-build system is almost entirely centred around rollup-plugin-html, and I have not found (or I lack the knowledge to find) an easy way to utilise their input format (allPages) without creating a custom plugin

*On dev, because I just bypass content rendering, it is still served with an HTML content-type, so image tags, JS, and almost anything else will not work regardless.

From my (quite inexperienced) perspective, quite a lot would have to change to account for endpoints. Most of which is going way over my head.

P.S. the branch is quite roughed up after the rebase, may sort out later if needed

Dev server:

me@me /# curl http://localhost:3000/test.txt
Foo⏎
me@me /# curl -v --silent http://localhost:3000/test.txt 2>&1 | grep "Content-Type"
< Content-Type: text/html

Build:

gabrielgrant commented 2 years ago

@rebelchris looks like your example has this same issue of being HTML-wrapped. Do you have a workaround for that that you're using on a live site somewhere?

rebelchris commented 2 years ago

I'm using it on the search site: https://loving-wilson-87e511.netlify.app/search.json/

https://daily-dev-tips.com/posts/render-a-json-page-in-astro/

You can see how it works, the JSON query doesn't really care if it's wrapped. Let me know if that helps you.

jonathantneal commented 2 years ago

Hey everyone! This RFC is still accepted, and it has been moved to https://github.com/withastro/rfcs/blob/main/active-rfcs/support-html-files.md

In an effort to improve our RFC process, we made some changes to better organize things, which include a dedicated RFC repository.

tansanDOTeth commented 1 year ago

@Tc-001 Do you have a link to where this issue is tracked?

Tc-001 commented 1 year ago

@Tc-001 Do you have a link to where this issue is tracked?

This is already implemented as "Endpoints", have fun! https://docs.astro.build/en/core-concepts/endpoints/

tansanDOTeth commented 1 year ago

@Tc-001 Do you have a link to where this issue is tracked?

This is already implemented as "Endpoints", have fun! https://docs.astro.build/en/core-concepts/endpoints/

This is awesome! Is there a way to use a component to render the body in this? Ex,

// Outputs: /builtwith.json
export async function get({params, request, props}) {
  return {
    body: <SomeComponent {...props} />
  };
}

Tc-001 commented 1 year ago

No, for that you can follow https://github.com/withastro/roadmap/issues/533

tansanDOTeth commented 1 year ago

No, for that you can follow withastro/roadmap#533

Thank you!

withastro / astro