Requirements for module caching

jldec commented 2 months ago

Context

While pulling plugin modules e.g. for lint reports from a cdn is useful in environments which don't have a package manager (e.g. in browsers), the current behavior is surprising in other environments where installed packages are supported (e.g. in locally installed npm projects or in a vs code extension)

Please use this issue to report requirements.

performance: network requests are expensive e.g. on each paraglide compile
offline usage: could be mitigated by preloading and caching
security checks: would be easier with a trusted/enterprise package registry
plugin user DX: configuring and knowing which plugin is going to run should be simple
plugin developer DX: how to dev-test-build-ship new plugins and plugin-updates should be straightforward
reliability: depending on a 3rd party cdn which is not typically used for app hosting, adds operational risk
observability: it should be possible to trace plugin calls to debug when something is not right

jldec commented 2 months ago

Proposal

Latest versions of inlang-maintained modules should be distributed with the sdk so that they can be loaded without additional network requests.
old versions of inlang modules TBD
3rd party user-maintained modules should (optionally) be dynamically loadable, with improved security warning.

samuelstroschein commented 2 months ago

Disagree strongly with the proposal.

Just load JS files from a remote, cache it, postpone everything security until Shadow Realm. Done. The user requirements can be fulfilled by just caching the plugins. No need for anything security or performance just yet.

jldec commented 2 months ago

@jan.johannes do you have thoughts on where to cache modules depending on the environment?
Are there any plans for caching in Lix?

samuelstroschein commented 2 months ago

@jldec that's the right question ;) Ideally, you can just fs.write file. i think jan refers to files that are not synced to as "Artifacts"

samuelstroschein commented 2 months ago

classic dynamic esm import.

That is what we are doing.

modules are published to NPM to use JSDelivr as CDN (over NPM) and avoid building our own registry
JSDelivr caches stuff for 1 week. If no manual pruning happens, plugins might take a week to update.

There is nothing wrong with module resolution except a missing if statement that checks if a module is cached (which requires lix to support artifacts):

const moduleIsCached = await repo.fs.fileExists("blabalbalba@0.24.0.js")

if (moduleIsCached){
  const module = await repo.fs.readFile("blabalbalba@0.24.0.js")
  await import(module)
} else {
  await import(remoteUri)
}

thenbe commented 2 weeks ago

Is there a way to setup paraglide without having to fetch from jsdelivr like this? jsdelivr has been been fickle in my experience.

If I understand correctly, this constraint exist because certain use cases require fetching from a cdn at runtime in the browser. Would it be possible to avoid funneling the rest of the use cases through that constraint? For instance, a majority of the time I just want to run pnpm install && pnpm run check in a project. And occasionally, I see error messages like these:

ℹ [paraglide] Compiling inlang project at "./project.inlang".
Using existing cloned repo

 WARN  The project has errors:

 ERROR  no loadMessages in resolved Modules found

  at Object.fn (/home/nbe/projects/sail/node_modules/@inlang/paraglide-js/dist/index.js:9221:34)
  at runComputation (/home/nbe/projects/sail/node_modules/@inlang/paraglide-js/dist/index.js:7309:23)
  at updateComputation (/home/nbe/projects/sail/node_modules/@inlang/paraglide-js/dist/index.js:7302:3)
  at runTop (/home/nbe/projects/sail/node_modules/@inlang/paraglide-js/dist/index.js:7371:7)
  at runUserEffects (/home/nbe/projects/sail/node_modules/@inlang/paraglide-js/dist/index.js:7428:5)
  at /home/nbe/projects/sail/node_modules/@inlang/paraglide-js/dist/index.js:7412:22
  at runUpdates (/home/nbe/projects/sail/node_modules/@inlang/paraglide-js/dist/index.js:7392:17)
  at completeUpdates (/home/nbe/projects/sail/node_modules/@inlang/paraglide-js/dist/index.js:7412:5)
  at runUpdates (/home/nbe/projects/sail/node_modules/@inlang/paraglide-js/dist/index.js:7393:5)
  at writeSignal (/home/nbe/projects/sail/node_modules/@inlang/paraglide-js/dist/index.js:7266:7)

 ERROR  Couldn't import the plugin "https://cdn.jsdelivr.net/npm/@inlang/message-lint-rule-empty-pattern@latest/dist/index.js":

TypeError: fetch failed

  TypeError: fetch failed
  at resolveModules (/home/nbe/projects/sail/node_modules/@inlang/paraglide-js/dist/index.js:7010:25)
  at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

 ERROR  Couldn't import the plugin "https://cdn.jsdelivr.net/npm/@inlang/message-lint-rule-identical-pattern@latest/dist/index.js":

TypeError: fetch failed

  TypeError: fetch failed
  at resolveModules (/home/nbe/projects/sail/node_modules/@inlang/paraglide-js/dist/index.js:7010:25)
  at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

 ERROR  Couldn't import the plugin "https://cdn.jsdelivr.net/npm/@inlang/message-lint-rule-missing-translation@latest/dist/index.js":

TypeError: fetch failed

  TypeError: fetch failed
  at resolveModules (/home/nbe/projects/sail/node_modules/@inlang/paraglide-js/dist/index.js:7010:25)
  at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

 ERROR  Couldn't import the plugin "https://cdn.jsdelivr.net/npm/@inlang/message-lint-rule-without-source@latest/dist/index.js":

TypeError: fetch failed

  TypeError: fetch failed
  at resolveModules (/home/nbe/projects/sail/node_modules/@inlang/paraglide-js/dist/index.js:7010:25)
  at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

 ERROR  Couldn't import the plugin "https://cdn.jsdelivr.net/npm/@inlang/message-lint-rule-valid-js-identifier@latest/dist/index.js":

TypeError: fetch failed

  TypeError: fetch failed
  at resolveModules (/home/nbe/projects/sail/node_modules/@inlang/paraglide-js/dist/index.js:7010:25)
  at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

 ERROR  Couldn't import the plugin "https://cdn.jsdelivr.net/npm/@inlang/plugin-message-format@latest/dist/index.js":

TypeError: fetch failed

  TypeError: fetch failed
  at resolveModules (/home/nbe/projects/sail/node_modules/@inlang/paraglide-js/dist/index.js:7010:25)
  at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

 ERROR  Couldn't import the plugin "https://cdn.jsdelivr.net/npm/@inlang/plugin-m-function-matcher@latest/dist/index.js":

TypeError: fetch failed

  TypeError: fetch failed
  at resolveModules (/home/nbe/projects/sail/node_modules/@inlang/paraglide-js/dist/index.js:7010:25)
  at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

 ERROR  No plugin provides a loadMessages() or saveMessages() function

In case no plugin threw an error, you likely forgot to add a plugin that handles the loading and saving of messages. Refer to the marketplace for available plugins https://inlang.com/marketplace.

  In case no plugin threw an error, you likely forgot to add a plugin that handles the loading and saving of messages. Refer to the marketplace for available plugins https://inlang.com/marketplace.
  at resolvePlugins (/home/nbe/projects/sail/node_modules/@inlang/paraglide-js/dist/index.js:6985:24)
  at resolveModules (/home/nbe/projects/sail/node_modules/@inlang/paraglide-js/dist/index.js:7051:33)
  at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Sometimes I need to try 3 or 4 times to make it go through. Other times, I need to wait a minute. Generally, it's not that bad. But consider a use case like a monorepo where multiple people are working on a project. Some developers might only be working on one part of the codebase and might not even be aware that inlang/paraglide is even installed. All they see is this error message when they're just trying to run the lint job.

tldr; Is there a way to opt out of this behavior?

UPDATE: Looks like this works. It's not ideal as it looks like I might lose some functionality. But I really need to get past this jsdelivr issue. If anyone can share a cleaner way to set it up that'd be appreciated.

pnpm add @inlang/plugin-message-format -D

{
    "$schema": "https://inlang.com/schema/project-settings",
    "sourceLanguageTag": "en",
    "languageTags": ["en", "de"],
    "modules": [
+       "../../node_modules/@inlang/plugin-message-format/dist/index.js"
-       "https://cdn.jsdelivr.net/npm/@inlang/message-lint-rule-empty-pattern@latest/dist/index.js",
-       "https://cdn.jsdelivr.net/npm/@inlang/message-lint-rule-identical-pattern@latest/dist/index.js",
-       "https://cdn.jsdelivr.net/npm/@inlang/message-lint-rule-missing-translation@latest/dist/index.js",
-       "https://cdn.jsdelivr.net/npm/@inlang/message-lint-rule-without-source@latest/dist/index.js",
-       "https://cdn.jsdelivr.net/npm/@inlang/message-lint-rule-valid-js-identifier@latest/dist/index.js",
-       "https://cdn.jsdelivr.net/npm/@inlang/plugin-message-format@latest/dist/index.js",
-       "https://cdn.jsdelivr.net/npm/@inlang/plugin-m-function-matcher@latest/dist/index.js"
    ],
    "plugin.inlang.messageFormat": {
        "pathPattern": "./messages/{languageTag}.json"
    }
}

Note: the paths are relative to the parent of the project.inlang directory.

samuelstroschein commented 1 week ago

fyi everyone: @loris.sigrist will implement module caching ASAP.

Given the incident I discussed this with @samuel.stroschein on discord.

I'm going to be building a version of offline-module caching with the following behavior

Modules are still served via JSDelivr but cached locally in project.inlang/cache/*.

Executions use the locally cached versions if available

Cached modules are not persisted via version control.

This enables developers to work offline. Caching is an implementation detail of the SDK, we can change the caching behavior later

opral / inlang-message-sdk