search

dai-shi / waku

⛩️ The minimal React framework
https://waku.gg
MIT License
4.08k stars 108 forks source link

feat: private dir #514

Closed dai-shi closed 3 months ago

dai-shi commented 3 months ago

"./private" for files that can only be read on the server.

vercel[bot] commented 3 months ago

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Updated (UTC)
waku ✅ Ready (Inspect) Visit Preview Feb 22, 2024 1:42am
codesandbox-ci[bot] commented 3 months ago

This pull request is automatically built and testable in CodeSandbox.

To see build info of the built libraries, click here or the icon next to each commit SHA.

aheissenberger commented 3 months ago

@dai-shi no problem to implement this for aws lambda but :

dai-shi commented 3 months ago

This feature is about "filesystem access" and not meant for a cloud deployment without filesystem. This is a bit of regret for me, because the capability isn't consistent, but the use case with fs.readFile is fairly acceptable for Node.js env, and without this feature, it's not possible.

For cloud services without filesystem, we need bundling as you suggest, and I think it's already possible as of now, optionally using such plugins.

aheissenberger commented 3 months ago

have a look at unstorage - with this abstraction you can cover all your use cases, get a consistent api. There is already a huge list of plugins. If you miss something I can provide the code for the missing e.g. filesystem adapter as I have done for AWS S3 and dynamodb.

dai-shi commented 3 months ago

I mean, we don't prevent using such libraries. My understanding is it's already supported. Let me know if it doesn't work. This PR isn't trying to provide a consistent api. It's limitedly for node:fs.

aheissenberger commented 3 months ago

I only want to help and not to extend this discussion, but you mention somewhere you would like to follow web standards. node:fs is not covert by web standards and not by the new WinterCG Standards for JS Server Side Runtimes.

I know unstorage is a product of the unjs universe where h3 is a similar product as Hono chooses for this project. What I found is Hono Storage but I think unstorage is currently the winner in this segment :-).

There is no problem to implement this for AWS Lambda as it uses Node. But AWS Lambda Edge is similar to Cloudflare and some other Serverless provider only a browser runtime without any support for node:fs.

dai-shi commented 3 months ago

but you mention somewhere you would like to follow web standards.

Yeah, that's correct. So, this seems kind of exceptional. More precisely, we follow web standards but also allow using Node.js, if users want. It means, if this PR breaks cloudflare usage, it's not acceptable.

There is no problem to implement this for AWS Lambda as it uses Node.

Oh, I see. Yeah, if it's possible in an easy way, let's do that.

AWS Lambda Edge is similar to Cloudflare

I think it's the same situation with Vercel and Netlify. Vercel serverless functions can be Node based, but Vercel edge functions are not. Netlify functions are Node based, but Netlify edge functions are not. So, if we use AWS Lambda, not AWSLambda Edge, which is Node based, we can support node:fs usage.

I only want to help and not to extend this discussion

So, it was our miscommunication and does "we follow web standard, but also support Nodejs if users want, as long as it doesn't break web standard usage" sound okay?

dai-shi commented 3 months ago

https://github.com/dai-shi/waku/pull/514#discussion_r1500159984

Maybe an alternative approach is to provide getPrivateDir from waku/server to abstract the actual path. It may solve #530 issue.

aheissenberger commented 3 months ago

#514 (comment)

Maybe an alternative approach is to provide getPrivateDir from waku/server to abstract the actual path. It may solve #530 issue.

Yes, this is an option but I would remove this feature - people who need a private read and writable data store, can implement it by them self and know the limits of their deployment environments. Nearly all cloud provide suggest to use a block storage (e.g. s3) or a shared KV store (Cloudflare, Deno deploy, ..) for this use case. The existing examples, with single files, could use simple imports with the ?url to get the file path, avoid embedding the json into the bundled code and have the data copied to the dist folder.
If someone need access to many static files, the solution depends on the overall size of the folder to choose the right type of deployment. On AWS there is a limit on the deployment bundle size so I would copy the files to a private s3 block storage.

dai-shi commented 3 months ago

I would remove this feature

That's one option. But that means to give up fs.readFile, right?

The existing examples, with single files, could use simple imports with the ?url to get the file path

Or, does it work with fs.readFile? Do you know if it works with markdown files?

have the data copied to the dist folder.

Do you mean to manually copy files? That's exactly the feature this PR is trying to automate.

aheissenberger commented 3 months ago

Ok - the use case is a blog with a folder of markdown files which are not static rendered.

To be clear - the current release of WAKU with --with-aws-lamba and the [serverless framework]() can handle the private folder and access to this folder with fs.readFile(./private/myfile.md) without any changes in theoutput-aws-lambda.js. The only handicap is that the private folder needs to be added to theserverless.ymlin thepackage` section to included into the final zip. For people using other deployment tools it will be similar. There is no manual copying required.

535 is a minimal documentation on how to deploy to AWS Lambda with an example serverless.yml config file including support for ./private.

dai-shi commented 3 months ago

Thanks for the clarification. So, it seems like the situation is similar to netlify functions.

We create netlify.toml file like this: https://github.com/dai-shi/waku/blob/ff20bfe538e32d9315f679a5d4d40e0b377aee18/packages/waku/src/lib/builder/output-netlify.ts#L38-L48

aheissenberger commented 3 months ago

The problem with AWS is, that there are 3 official ways (AWS CLI, SAM, CDK) to deploy to the infrastructure and a whole bunch of other options (e.g. Serverless Framework, SST, Pulumi, Terraform,...).

Currently the builder for AWS creates an output which can be consumed by all of them. To support them is too much work and the user needs to adapt the config files to their environment on AWS and to the other infrastructure they want to create (e.g. DNS records, databases,...). This will only allow us to create a config once and does not allow any changes later.

I hope you agree, that the current documentation is a good start to get a project up and running including support for the private dir.

dai-shi commented 3 months ago

To support them is too much work

Can we pick one that best matches to Waku? Like, as for Netlify, they have functions and edge functions, but we only pick "functions". They also support UI deploy and CLI deploy, and in this repo, we only support CLI deploy.

the current documentation is a good

Let me check the PR again, but in general, I just count on you, based on our conversation so far.

aheissenberger commented 3 months ago

People which look for a simple deployment platform will choose Vercel or netlify - this is their market position against AWS and Azur. Choosing AWS requires a deeper background knowledge of their services or one of the deployment options to start.

The main reason I support this project is, that I am looking for a valid react rsc framework which supports AWS Lambda directly. Running NextJS in the AWS environment is not supported by Vercel and the existing external solutions break with small and big releases.