ManasMadrecha commented 7 months ago

Bug Report

Describe the bug

I was using this amazing lib in Firebase Functions, but I have recently moved to Cloudflare Workers. But, I am getting Code generation from strings disallowed for this context error when I use in lib in production api.

Note: I have enabled nodejs in cloudflare. nodejs_compat flag in the worker but still this error comes with this lib.

Minimal Reproduction

latest Nitro app (unjs/nitro) with preset cloudflare-module

Environment

Browser or Node: Cloudflare (with nodejs_compat enabled) Node version (if applicable): on dev (windows 11), node version 20+ Npm version: 10+

Additional Context

Docs

Cloudflare docs (https://developers.cloudflare.com/workers/runtime-apis/web-standards/#javascript-standards) say:

eval() is not allowed for security reasons.
new Function is not allowed for security reasons.

Is yf lib using these? If yes, is there a way to refactor them?

[EDIT] Error

I think the cause of error might be a particular package tough-cookie, because when I build my nitro app with yf lib, this non-fatal error shown in terminal (though the app still builds, but api routes using yf lib dont work):

(node-resolve plugin) Could not resolve import "unenv/runtime/mock/proxy-cjs/" in C:\Users\<user/nitroapp>\api\node_modules\tough-cookie\lib\cookie.js using exports defined in C:\Users\<user/nitroapp>\api\node_modules\unenv\package.json.

(node:13008) [DEP0155] DeprecationWarning: Use of deprecated trailing slash pattern mapping "./runtime/mock/proxy-cjs/" in the "exports" field module resolution of the package at C:\Users\<user/nitroapp>\api\node_modules\unenv\package.json imported from C:\Users\<user/nitroapp>\api\node_modules. Mapping specifiers ending in "/" is no longer supported.

(Use `node --trace-deprecation ...` to show where the warning was created)

eddie-atkinson commented 2 months ago

Just to add some colour to this issue I think the problem (or at least one of them) is coming from the dependency on ajv for validation.

Running the repro env using nitro and wrangler I get the same error message as in the AJV issue:

Code generation from strings disallowed for this context

There are also possibly more issues with getting this to run in a non-node runtime environment.

Reading through the source code ajv is clearly a core part of this library, so ideally we'd fix the issue at source rather than here. That being said, I've read through the source for ajv where this issue appears to be coming from and it's hairy as hell.

I'd happily contribute PRs to rewrite the validation logic in this library to another strategy like zod, but am not sure I personally could solve this problem upstream.

I also appreciate that it's not a stated goal of this library to run on every JS runtime, so if the answer is "too much complexity" or "give the AJV people a chance" I'd also understand.

Any thoughts @gadicc?

gadicc commented 2 months ago

Hey @eddie-atkinson, thanks for this report too.

I would love for the library to run in non-node environments. I personally use edge computing on a number of other projects so I'm totally behind this in principle.
However, you're right, ajv is very central to the library and won't be easy to replace. We don't actually run any complex validation of the kind zod does so well. The flow is actually: a) typescript interfaces for all the APIs, b) typescript interfaces -> json schema, c) ajv to validate input using the json schema. But to further complicate matters, we use a number of ajv features / API for a bunch of more complicated cases specific to the yahoo API responses. Lastly, AJV was actually selected (at the time?) due to being significantly faster (by a large margin) than all other solutions, which is handy considering that we have 1000's of tests.

Having said that, I've also had issues debugging ajv before, and I think actually we're still stuck on an old release (at least for a while, I haven't checked recently) because something broke and it was so hard to work out what was happening. I also think Zod has done a great job at establishing itself, and of course, we can infer the typescript interfaces from zod schemas, it could even uncomplicate our release logic to not need to always rebuild the schema. But this would be a huuuge job, and I also think, probably worthy of a communal debate with enough time to elicit responses.

If after all that, it's still something you'd like to investigate, you have my support :) I would probably start off by looking for something that converts typescript interfaces to zod code, as a starting point... as there are a loooot of them (see e.g. quoteSummary-iface.ts).

eddie-atkinson commented 2 months ago

Hi @gadicc,

Thank you for taking the time to give a detailed response.

The flow is actually: a) typescript interfaces for all the APIs, b) typescript interfaces -> json schema, c) ajv to validate input using the json schema.

Given that this is the current workflow it feels like cutting out JSON Schema could make sense as a way of simplifying development as the source of truth is Typescript.

Lastly, AJV was actually selected (at the time?) due to being significantly faster (by a large margin) than all other solutions, which is handy considering that we have 1000's of tests

That's a fair call out that I didn't consider. Looking at the benchmarking done here it seems like zod has pretty poor performance characteristics compared to ajv. Interestingly the top 3 in the rankings all use Typescript types as their source of truth, though admittedly not "pure" Typescript (they have annotations you need to add). I did a little survey of the tools that ranked higher than ajv on that benchmark and I'm interested in your view. From the perspective of bundle size, maturity, and weekly downloads @sinclair/typebox seems like a promising option. Especially given that its perf in AOT and JIT mode exceed ajv. It also appears to support CF workers.

(source)

Name	Stars	Weekly Downloads	Date of first commit	Minified Bundle Size (as reported by bundlephobia)
ts-runtime-checks	228	146	2022-07-22	58 kB
@sinclair/typebox	4200	24,239,331	2017-04-07	50 kB
typia	4045	24,148	2022-04-19	65.4 kB
spectypes	91	31	2022-04-29	1.7 kB
suretype	490	17,553	2023-02-26	139 kB
ts-json-validator	341	29,848	2019-10-31	118.8 kB
ajv	13,378	81,611,055	2015-05-31	119.6 kB

But this would be a huuuge job, and I also think, probably worthy of a communal debate with enough time to elicit responses.

Is there a particular venue where you'd like to have that discussion? Here in this thread? GitHub discussions? I appreciate that this thread is, at the moment at least, a bit too hidden a way to have a discussion of this magnitude.

I would probably start off by looking for something that converts typescript interfaces to zod code, as a starting point... as there are a loooot of them

Yes and no, as shown by my highly scientific survey here:

The real questions for me would be less the leg work of converting all the interfaces and more so how we review the changes.

Keen to hear your thoughts and happy to put together a micro benchmark of some of the more gnarly interfaces to test feasibility and performance on any libraries of interest.