Worker: support @local adaptors

josephjclark commented 4 weeks ago

Short Description

This PR allows the worker to load adaptors with the @local version number, which will be loaded from the monorepo

Fixes #806

This later allows Lightning to be compatible with the monorepo

Implementation Details

Basically what we've done is detected @local version numbers and given them an explicit path. This works for collections too, if the server's collection version is set to local.

OPENFN_ADAPTORS_REPO must be set in the env for this to be supported

I've had to make some changes here in the runtime and I have a feeling they're ill-advised. I want to check with a fresh brain later.

AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to know!):

[ ] Code generation (copilot but not intellisense)
[ ] Learning or fact checking
[ ] Strategy / design
[ ] Optimisation / refactoring
[ ] Translation / spellchecking / doc gen
[ ] Other
[x] I have not used AI

You can read more details in our Responsible AI Policy

josephjclark commented 4 weeks ago

TODO: the version readout in the worker says "unknown", not "local"

josephjclark commented 4 weeks ago

Some of this log out put is really weird

    ▸ @openfn/language-collections=/home/joe/repo/openfn/adaptors/packages/collections    unknown
    ▸ @openfn/language-common                                                             2.1.1
R/T Executing ea6d1457-afed-4c08-897b-a3abfc7b27da
R/T Starting step New job
R/T [linker] loading module @openfn/language-collections=/home/joe/repo/openfn/adaptors/packages/collections
R/T [linker] Loading module @openfn/language-collections=/home/joe/repo/openfn/adaptors/packages/collections from /home/joe/repo/openfn/adaptors/packages/collections/dist/index.cjs
R/T Resolved adaptor @openfn/language-collections=/home/joe/repo/openfn/adaptors/packages/collections to version 0.1.0

josephjclark commented 3 weeks ago

Great - I've managed to really simplify this and get a much cleaner solution

OpenFn / kit