sdk v2 release on sqlite

[samuelstroschein]

Context

• We agreed on building inlang on sqlite MESDK-154.
• We need to define the minimal requirements to release sdk v2 on sqlite now.

Proposal

Release on August 5 to have 1 more week before Martin goes OOO to fix bugs.

What we need:

• [ ] CRUD ops (via kysely)
• [ ] Importer/exporter
• [ ] Lint reports
• [ ] Agree on MESDK-152
• [ ] loadProjectfromOPFS and loadProjectFromNodeFS

What we don't need:

• Validation (inlang projects use import/export at first. corrupted inlang files is no issue)
• search (apps can query what they need already)
• Reactivity (apps can workaround with polling https://github.com/opral/sqlite-prototype/blob/bdb9cabc0d952ed73455e1642c0d855b07b57932/csv-app/src/reactivity.ts#L1-L9)

[samuelstroschein]

Potential minimal inlang sdk API

• expose kysely, let apps query whatever is needed
• let apps do workaorunds with reactivity and search

type API = {
  db: Kyseley
  import: ()
  export: ()
  // fixing lint reports
  fix: ()
}

How import/export would work in dev apps

1. dev apps read the settings file in git
2. create new inlang project
3. execute import()

import { fs } from "node:fs/promises"
import { newInlangFile, loadProjectFromNodeFs } from "@inlang/sdk"

async function loadInlangProject(){
  // the settings file is still stored in git 
  const settings = await fs.readFile("source-code/website/project.inlang/settings.json")

  // new inlang binary file
  const inlangFile = await newInlangFile()

  // write to temp dir
  fs.writeFile("temp/project.inlang", inlangFile)

  // loaded the inlang project
  const project = await loadProjectFromNodeFs("temp/project.inlang")

  // "importing" the settings
  await project.settings.set(settings)

  // importing the messages based on the settings file
  await project.import()
  return project
}

async function saveInlangProject(){
  await project.export()
  const settings = await project.settings.get()
  await fs.writeFile("source-code/website/project.inlang/settings.json", settings)
  await fs.delete("temp/project.inlang")
}

@martin.lysk1 do you have a clear picture on what i have in mind?

[samuelstroschein]

cc @nils.jacobsen this is the roadmap of sdk v2. we don't need more than importer/exporter and variants

[samuelstroschein]

I propose a toBeImportedFiles() api to keep fs out of import/export()

Problem

Web apps like Parrot or Fink will need the capability to import and export translation files.

• If nodeFs is leaked into import() or export() web apps will not be able to import and export. No node fs exists in the browser.
• Keeping nodeish fs out of the inlang sdk will reduce maintenance burden.
• Enterprises asked for importing and exporting in parrot already.

Proposal

Give plugins a toBeImportedFiles() API to keep fs out of import/export().

1. A plugin returns what files should be passed to importFiles()

type TranslationFile = {
  path: string
  content: string
  pluginKey: string
}

type Plugin = {
  key: "i18next",
+  toBeImportedFiles: ({ settings, nodeFs }) => Array<TranslationFile>
  importFiles: ({ files: Array<TranslationFile> }) => Array<NestedBundle>
  exportFiles: ({ bundles: Array<NestedBundle>, settings }) => Array<TranslationFile>
}

2. The SDK exposes a project.toBeImportedFiles({ nodeishFs })

• Keeps nodeishFs stuff out of SDK internals because its only forwarded to the plugin api call
• importFiles and exportFiles can operate in web apps

// either via toBeImportedFiles if nodeFs is available
const files = project.toBeImportedFiles({ nodeFs })
await project.importFiles({ files })

Pseudocode example in Sherlock or Paraglide

import { fs } from "node:fs/promises"
import { newInlangFile, loadProjectFromNodeFs } from "@inlang/sdk"

let project

// set up new project if not exists
if (fs.exists("temp/project.inlang") === false){
   const inlangFile = await newInlangFile()
   const settings = await fs.readFile("source-code/website/project.inlang/settings.json")
   fs.writeFile("temp/project.inlang", inlangFile)
   project = await loadProjectFromNodeFs("temp/project.inlang")
   project.settings.set(settings) 
}

// import files on boot up
+const files = await project.toBeImportedFiles({ nodeFs: fs })
await project.importFiles({ files })

// export on close
const toBeExported = await project.exportFiles()
for (const file of toBeExported){
  await fs.writeFile(file.path, file.content)
}

3. Web apps can import files via the UI

• insight: file paths contain all information (locale and optionally namespace) that a plugin needs
• we likely want a import/export ui component cc @nils.jacobsen

Web UI Import and Export Simplified

// or via a UI where the user could upload a zip,
// select a plugin, and let the app unzip and pass it to import 
const filesUploadedByUser = [{
  path: "i18n/en/common.json",
  pluginKey: "i18next",
  content: "<content>"
}]
await project.importFiles({ files })

Additional information

• nodeishFs should be completely gone and replaced by regular nodeFs regardless of accepting this proposal
• plugins should have key instead of id as the key can change

[martin-lysk]

To understand this better and verify this approach I need further information about the lifecycle of an inlang file:

When will loadInlangProject and saveInlangProject be called - whats the lifecycle of the inlang file?

// export on close

From this comment in the pseudo code i assume that one would open sherlock - files are imported, one works in sherlock changes messages settings etc. Then sherlock is closed and we save the settings and bundles back via export?

Lifecycle Export:

Do we plan to export only on close of sherlock or also on other events?

If we export between an open and close of sherlock - which events should trigger an export?

Lifecycle Import:

How do we plan to deal with changes on the file system of the user during the lifecycle of an inlang file / a sherlock session?

Shall changes on the files we imported lead to a reimport - do we need to watch on changes on those files within sherlock - and trigger import again?

Scenarios:

User edits the json file while sherlock is open,

User switches branches - which lead to changes in the json file while sherlock is open.

[samuelstroschein]

Is the lifecycle of an inlang file in dev tools a concern for the inlang SDK?

I don't think so. When sherlock imports or exports is up to sherlock (cc @felix.haeberle). Keeping the (edge case) logic out the inlang SDK seems good because most (web) apps won't need it. And, dev tools in the future also not.

@martin.lysk1 Agree to keep lifecycle stuff out of the inlang SDK and let Sherlock/Paraglide deal with it?

Example dev tool lifecycle flow

If we export between an open and close of sherlock - which events should trigger an export?

Up to dev tools, and exposing DB makes everything possible ;) That's why I am so much in favor of exposing db. let apps do what they need without having to think everything through in the inlang sdk.

// export on every change of a nested bundle
project.db.selectFrom("bundle")
  .innerJoin("message")
  .innerJoin("variant")
  // pretending here that subscribe already exists
  // can be poll based at the beginning ofc
  .subscribe((nestedBundles => {
    const files = project.export(nestedBundles)
    await fs.writeFile(files)
  }) 

// export on close

const files = project.export(nestedBundles)
await fs.writeFile(files)

[felixhaeberle]

@martin.lysk1 @samuel.stroschein do we have an agreement here that the plugin API should look like this for now? so I'll start transforming the i18next plugin & plugin interface to it.

[martin-lysk]

@felix.haeberle and I had a short exchange about this the api.

One issue we see here is that that sherlock will need to watch on the files the importer needs.

While we could take the list returned by toBeImportedFiles and the included paths to setup listeners - sherlock would not be able to detect added files. This is a problem is currently handled (partialy) by the fsWithWatcher that detects reads by the Plugin (which eventually detect new files files on the next load message call).

One option would be to use a polling approach in sherlock that calls toBeImportedFiles and loads the files in intervals - might be something we need to consider anyway since the fs watch api has shown to be blind on renames of files…

Any take on this @samuel.stroschein?

[samuelstroschein]

Take the polling approach. (as discussed with @martin.lysk1 on a call)

[felixhaeberle]

One option would be to use a polling approach in sherlock that calls toBeImportedFiles and loads the files in intervals

its a bit hard to follow here, do apps now directly use plugin behavior like plugin.toBeImportedFiles (in comparison to only use inlang sdk project apis such as project.import / plugin.export) ?

Edit: Ah wait, re-reading the proposal you stated that the functions get reexported by the project interface.