document team workflow

[nherment]

Here is our scenario:

• dev1 sets up repo with fuge, pushes to origin
• dev2 pulls repo and installs fuge -g
• dev 2 runs > fuge shell fuge/compose-dev.yml then start all
• dev2 has dependencies issues

➜  fubar git:(master) ✗ fuge shell fuge/compose-dev.yml
compiling...
starting proxy...
  proxy frontend 10000 -> 192.168.1.6:20000
starting shell..
? fuge> start all
running: frontend
running: __proxy
? fuge> [frontend - 12501]: module.js:341
    throw err;
    ^

Error: Cannot find module 'hapi'
    at Function.Module._resolveFilename (module.js:339:15)
    at Function.Module._load (module.js:290:25)
    at Module.require (module.js:367:17)
    at require (internal/module.js:16:19)
    at Object.<anonymous> (/home/nherment/workspace/nearform/fubar/site/api/index.js:3:14)
    at Module._compile (module.js:413:34)
    at Object.Module._extensions..js (module.js:422:10)
    at Module.load (module.js:357:32)
    at Function.Module._load (module.js:314:12)
    at Function.Module.runMain (module.js:447:10)
process exit [12501]: frontend

This was solved by executing:

cd site/api
npm install
cd -
fuge run fuge/compose-dev.yml

It'd be great if fuge could see that it's a new install and run npm install wherever it is needed. It'd be even better if someone updates a package, for fuge to see it and automatically maintain the npm dependencies.

[AdrianRossouw]

would you mind trying this with the latest master? it split up the api and site into separate services, and there shouldn't be any dependency issues.

I am not sure how this relates to the title though (document team workflow), but I will try to answer what I think the question is.

Fuge has a very constrained scope, and part of that is that it doesn't try to manage dependencies or versions, it's up to you to get the files there. I've been using npm itself to manage these dependencies in the nodezoo-system project. I just created a package.json and did npm install on the various repos directly from git. i then pointed the yml file to ../node_modules/service.

My first attempt did end up in an unholy mess, but I discovered that recent versions of npm allow you to configure global-style for packages, which I stuck in a .npmrc.

That setting will install direct dependencies in their own folders and not try to flatten packages outside of the services.

What I like about this technique is that you can get the system up and running by just doing npm install in the root, you can update packages using npm update, it works fully with git repos and published packages and lastly npm link works perfectly for managing symlinks in your dev environment.

I will probably be adding a package.json and .npmrc to the fuge system generator soon, so new systems can run like this out of the box.

[nherment]

I'll try to give it a shot today

[nherment]

Fuge has a very constrained scope, and part of that is that it doesn't try to manage dependencies or versions, it's up to you to get the files there. I've been using npm itself to manage these dependencies in the nodezoo-system project. I just created a package.json and did npm install on the various repos directly from git. i then pointed the yml file to ../node_modules/service.

Doesn't that conflict a bit with what fuge is about ? If I understand correctly, you recommend installing the different services through NPM but fuge is for development environment. Ideally we can use git repos for each of the custom services and benefit from the auto-reload functionality.

An idea is for fuge to tell me where I should put my stuff and manage my dev environment for me. It's especially usefull for newcomers/onboarding. As such, could we envision fuge as being able to:

• pull master/production/ branches from a git repo
• pull npm packages (arguably not necessary if the above is supported)
• auto npm-install all local services
• recognizes services that are developped locally and installing them as symlinks

For example I now want to add a new service. It will be a new repository and I'm probably going to symlink it into our fuge-specific repo. When another dev will pull the origin with my changes, I'd like them to not have anything to do but fuge run ... to get all the services running, including the new one. How would you recommend that I achieve this ?

[mcdonnelldean]

@nherment So that pretty much is how we are doing things with Nodezoo, granted it's a little hairy in places but the ultimate goal is to have it as,

• download the 'system' repo
• run npm install (to bring down the system)
• run fuge build fuge/my-system.yml to build were required
• run fuge shell fuge/my-system.yml to orchestrate

I had to do the symlinking myself but I think this would be a nice feature. @pelger Thoughts on that?

[nherment]

fyi currently looking into git submodules. It looks like it's a good solution but may require some (limited) custom scripting.

[mcdonnelldean]

Do you need them though? Having one or separate repos seems to work fine, any reason to introduce them?

[nherment]

My reason is that I don't want devs to have to manage symlinks, dependencies and whatnot.

[mcdonnelldean]

@nherment Ah! agreed, I would like Fuge to do some magic here, even if it something I had to add to the initial config.

[AdrianRossouw]

i have never had an instance where git submodules didn't eventually make me want to claw out my eyes. (sorry for being dramatic, but i'm being pretty earnest when i say that. I have battle scars ...)

[AdrianRossouw]

Doesn't that conflict a bit with what fuge is about ? If I understand correctly, you recommend installing the different services through NPM but fuge is for development environment.

Fuge doesn't care, and is not responsible for how your files get onto the disk, or where you put them.

Ideally we can use git repos for each of the custom services and benefit from the auto-reload functionality.

If there are files, and they are on a disk, and there is a service that refers to that directory, fuge will reload them.

[AdrianRossouw]

My recommendation

I recommended using a single repo for all your custom services unless you have a very very good reason not to.

If you do this, you are done. and you can ignore just about everything else in this post.

You could still use the npm technique to install externally developed services such as concorda and vidi, but those are 3rd party, and are distinct from managing your own services.

(btw: fuge provides the fuge build command to run npm install in all defined services )

Separate repos / projects with npm

If you do want to use separate repositories, you can either install npm packages, or git repositories from the package.json. (this is just one way i have found that works very well. it's out of scope for fuge)

npm install --save vidi-dashboard
npm install --save org/package-name#branch

To avoid flattening the service deps, i recommend using npm with the global-style toggle.

echo'{"global-style":true}' > .npmrc;

A word of warning ... once you switch to multiple repositories you really should start managing dependencies properly and correctly, whether you are using npm packages or git repositories.

Without that, you lose the ability to have reproducible builds, and it will make debugging incredibly difficult. so no pinning things to #master and hoping for the best...

Using npm link for global checkouts

You could also use npm link to link a separate external checkout of a module to the npm global space, and then link an instance into your service package.

This is on a per-service basis imo, and I will only do it when I know I am going to need to modify a specific service / module.

cd ../;
git checkout <url>/package-name.git
cd package-name;
npm link; # will npm install
cd ../system;
npm link package-name;

BUT! ... Once you start using shared checkouts with symlinks, it is your responsibility to manage them independently and correctly.

One thing to keep in mind that one of the most common use cases for symlinks is to share the code across multiple systems in dev ... it would be pretty reckless for fuge to try and mess with those folders too much.

(btw: fuge provides the helpful fuge pull command that will run git pull on all your services that have .git folders).

Initial setup of symlinks

You could write a Makefile that automatically does that for you.

make dev-setup

You could also probably do that in npm scripts too for windows support, but it would be uglier, and why do that unless you needed to support windows explicitly.

Don't try to automate dependency stuff too much.

Having any of this stuff happening automatically on fuge run, is pretty much the same as setting your start script in npm to be : npm update && npm install && node app.js.

It's guaranteed to be an enormous flustercuck, with nobody having any idea what version of the codebase they are actually running, and things constantly mysteriously breaking when a butterfly flaps it's wings in asia.

So not something I would recommend.

[mcdonnelldean]

As above with Adrians comment plus the Readme from https://github.com/nodezoo/nodezoo-system.

Both should contain enough info to get a team started.