Open fricklerhandwerk opened 1 year ago
Hi! Awesome that you want to help out :)
I'll first make sure both Atomic-Server and Atomic-Data-Browser codebases are combined into one monorepo.
We can start with packaging the Rust backend, and then continue with the Typescript frontend when it's merged. Then testing both components in integration as well as building a module to run the service on NixOS would be possible.
It's merged! It's now one monorepo. Check the dockerfile
for how it's currently built. Let me know if I can help in any way!
Thank you :)
@ngi-nix/moss is claiming this. We will ask you about any questions we have about the project as we learn more about it @joepio !
Brainstorming a plan of action (not necessarily in order):
Summarizing where we are right now: https://github.com/atomicdata-dev/atomic-server has 5 components
lib
is a Rust library so will be compiled from source by components that need it, so it doesn't need to be packaged separatelyserver
and cli
have been packaged in https://github.com/ngi-nix/ngipkgs/pull/62browser
(and desktop
depends on this via tauri).The current challenge with browser
is that dependencies are packaged via pnpm
, which has poor support for Nix at the moment (https://github.com/nix-community/pnpm2nix is old and outdated, at minimum we need to use this fork for compatibility with the latest version of pnpm
https://github.com/nix-community/pnpm2nix/pull/26). One of the issues with pnpm2nix
is that it expects a version
key in package.json
. We have had trouble trying to patch this while building in Nix, so we have submitted a PR to add this in https://github.com/atomicdata-dev/atomic-server/pull/673
Another approach we are exploring is converting pnpm-lock.yaml
into an equivalent package-lock.json
(npm) or yarn.lock
.
node_modules
directory structures. So one approach was to use pnpm to install a valid set of packages, then reverse engineer a yarn or npm lock file out of that.
npm i --package-lock-only
, but apparently this is incompatible with pnpm workspacesnpm ERR! code EUNSUPPORTEDPROTOCOL
npm ERR! Unsupported URL Type "workspace:": workspace:*
yarn import
gives us
yarn import v1.22.19
info creating yarn.lock from local node_modules folder
error "eslint-plugin-import#array-includes" not installed
... several similar errors
pnpm-lock.yaml
to either package-lock.json
or yarn.lock
directly without actually installing the packages into node_modules
first.
nix run nixpkgs#pnpm-lock-export -- --schema yarn.lock@v1
in the browser
directory of atomic-server to get a yarn.lock
. We are still learning how to get this to work with the mkYarnPackage
builder in nixpkgs, but this seems promising so far.@joepio we also noticed there is a bun.lockb checked in. Is bun
an "officially" supported way to install dependencies for browser
too and is it kept in sync with pnpm-lock.yaml
? Because according to https://bun.sh/docs/install/lockfile, we can have bun additionally export a yarn.lock
using bun install --yarn
, which could be easier than using the pnpm-lock-export
tool.
@joepio another question, for running browser, do we only need the artifacts in data-browser/dist
or do we need to copy over various node_modules
as well? I couldn't tell if dependencies were getting bundled into the artifacts of data-browser/dist
, but based on the Dockerfile
of the current monorepo AND the build script from the old repo, it looks like that's all we need.
@joepio another question, for running browser, do we only need the artifacts in
data-browser/dist
or do we need to copy over variousnode_modules
as well?
yes! Only the dist folder is needed to compile atomic-server
Is bun an "officially" supported way to install dependencies for browser too and is it kept in sync with pnpm-lock.yaml?
Hmm... No, not yet, but as soon as Bun Test supports the features I need (e.g .toHaveBeenCalledWith
) I'll probably migrate. I don't expect this to take very long. So I don't know if it makes more sense for you to assume the bun scenario.
Also, perhaps interesting - I spent some time last weekend setting up an earthfile
. Could be useful inspiration?
Summarizing some recent progress:
yarn.lock
we have committed.lib
> react
> data-browser
), we've noticed that the yarn.lock
we've generated is not the union of all required packages.lib
package.json (e.g. whatwg-fetch
, cross-fetch
).tsc
binary not being in scope, which means we're successfully working around not having a pnpm
builder.package.json
.@joepio is migrating to Bun still an option for you? This would likely make it easier on our end to finish packaging, otherwise we'll have to make a detour of creating tooling to deal with pnpm
.
@fricklerhandwerk Hmm looks like bun
now has support for the testing methods so maybe it can replce pnpm
! I'll work on it today.
Reviving this issue!
@ngi-nix/moss picked this up towards the end of Summer of Nix 2023 and we were not able to finish packaging it by the end of the program. Sorry for not posting a final status update at the end.
We were still stuck on figuring out how to get pnpm + nix working. We wanted to consider bun as an alternative but it looks like it was rejected in the upstream project https://github.com/atomicdata-dev/atomic-server/pull/806
@Chickensoupwithrice and I picked this issue back up as part of our extension work.
First we updated the existing packaging to the latest releases from the upstream project https://github.com/ngi-nix/ngipkgs/pull/162 https://github.com/ngi-nix/ngipkgs/pull/163
There seems to have been more activity with getting pnpm to work with nix and we should be able to use it directly instead of transforming it to npm or yarn as we were investigating before .
https://github.com/nix-community/pnpm2nix/pull/26#issuecomment-1833093501 leads us to
We decided to explore using https://github.com/nzbr/pnpm2nix-nzbr. We are making good progress so far and hope to report more soon.
Atomic Data is a modular specification for sharing, modifying and modeling graph data. It combines the ease of use of JSON, the connectivity of RDF (linked data) and the reliability of type-safety.
This project is actively developed, and the lead is interested in collaboration. We will figure out the best way to approach this, and it may involve some sort of onboarding if an upstream Nix setup is desired.