fix: packages being installed without lock file

[toomuchdesign]

Given that:

• We are currently using npm workspaces to handle the monorepo, keeping a single root lock file
• gcloud builds in isolation from the specific packages/package-name folder since it is ONLY able to deploy from the same folder where the function entry point lives

...gcloud is building the packages out of the monorepo context (since the root folder in the package itself), and npm dependencies are re-built from scratch without a lock file (since it lives in the root monorepo folder which is not available in gcloud environment).

In order to work around this limitation, we are considering the following options:

Get rid of npm workspaces and go for separate packages

Pros

• Fits perfectly the gcloud way, since each package has 0 externals dependencies (1 lock file for package)

Cons

• No monorepo to handle dependencies and commands
• No abstraction of common functionalities into local shared packages

Expose a root entry point

Root monorepo would expose an index file exporting the 3 packages entry points.

Pros

• All monorepo's pros

Cons

• Each package CI would build the whole monorepo dependencies
• Each gcloud instance would import the dependency tree of all other packages (since we only have 1 entry point file)

Challenges

• slack-bot should be reviewed to not run side effects (initialization) on import
• embeddings-creation and crawler do not expose a plain function, therefore we should find a proper way to re-expose them from the root entry point file

[simoneb]

As discussed, the only things I would look into at the moment are:

1. does gcloud allow us to do this, e.g. by getting the whole repo then specifying which function to run in which folder?
2. would a different package manager help?

unless one of the above is a simple and viable solution with little drawbacks, I would accept the current situation as a good enough compromise, with only these downsides:

1. we can't use a package in the monorepo to hold shared code
2. we can't use a lock file

Both of the above limitations are acceptable in this project in my opinion

[toomuchdesign]

gcloud seems to be quite restrictive in this regard.

gcloud beta functions deploy documentation states:

Note that, depending on your runtime type, Cloud Functions will look for files with specific names for deployable functions. For Node.js, these filenames are index.js or function.js. For Python, this is main.py.

gcloud seems to be stricly designed around the idea that the source folder is the same place where function entrypoints live.

I could find documented attempts to workaround this limitation consisting of pre-building/publishing shared dependencies:

• https://stackoverflow.com/questions/61015842/deploying-to-firebase-functions-with-a-monorepo
• https://stackoverflow.com/questions/75323548/google-cloud-functions-nodejs-monorepo-and-local-dependencies

monorepo or workspaces seem not to be mentioned among Node.js gcloud GH issues.

[toomuchdesign]

The discussion about having dedicated package lockfiles in Yarn seems a lang debated one, instead:

• https://github.com/yarnpkg/yarn/issues/5428
• https://github.com/yarnpkg/berry/issues/1223

...and a few yarn plugins have been put together to workaround it:

• https://github.com/JanVoracek/yarn-plugin-entrypoint-lockfiles
• https://www.npmjs.com/package/generate-lockfile

Turborepo implemented a very promising experimental Pruned Workspaces feature, which should be able to generate pruned lock files for a specific package target. Very worth investigating even beside this specific feature:

https://turbo.build/repo/docs/reference/command-line-reference#turbo-prune---scopetarget

[toomuchdesign]

Given the limited scope of the project we are keeping the current deploy strategy untouched for now. Leaving the the issue open since still relevant.