Add useful `npm run` tasks, tidy & simplify .sh and .cmd scripts (was: consider using Makefile or Grunt)

[simonbaird]

I don't like the clutter of having all those .sh files and .cmd files in the root directory, so I was initially going to suggest moving them all into a sub-directory.

But it occurred to me that it might be nicer to put them into a Makefile (with a bunch of .PHONY targets, one per .sh file).

Eg instead of ./qbld.sh you might do make qbld or whatever.

It adds a dependency on make. I don't know how problematic that would be for windows developers, but I assume it's not too hard to get GNU make.

It might be possible to have a single Makefile that work in windows and non-windows.

If the dependency on make is not desirable, perhaps Grunt.js would work. I don't know a lot about Grunt, but it seems to be popular for node projects.

[simonbaird]

I thought I would do some experimenting with a Makefile. I think it's pretty good! I can now do

make test
make bld
make 2bld

and it runs (essentially) as per the equivalent .sh files.

There are lots of advantages versus the current many .sh and .cmd files approach.

See https://github.com/simonbaird/TiddlyWiki5/compare/makefile

See the commit message for notes and possible issues.

(If you think it's worthwhile I could finish it off. Might need some help with testing the windows support though).

[Jermolene]

Hi @simonbaird there's an alternative proposal for sorting out the batch files in #356, where we do the build natively within TW5. The only batch files left would be my deployment scripts, which we could move into a subfolder. I like the idea of keeping external dependencies as minimal as possible, (I think make isn't available by default everywhere).

[simonbaird]

Hmm, okay I did read #356 but thought it was probably orthogonal to this. I imagine #356 would make the long node command lines become shorter, but it still might be worthwhile to have a nice way to document, maintain and run tasks other than via a bunch of .sh and .cmd scripts.

[Jermolene]

Yes, I guess they are somewhat orthogonal; makefiles might still be useful even if #356 were done. But #356 solves the immediate issue of the confusing profusion of batch files, and the need to maintain both *nix and Windows versions of them. The other point is that I'd rather not bring in make as a core dependency. We could ship some standard makefiles as starting points for people who want to customise their own.

Done right, #356 would enable us to do away with the existing batch files. Looking at the existing batch files, if one takes out the deployment stuff then all that's left is straightforward running of TW commands.

[pmario]

@simonbaird. I did an experiment with an extended bash script. So there would be only one script left. see: Extended bld.sh script proposal #355. I found out, that it wouldn't solve our problem and Jeremy said, theres too much "bash foo" in it :)

I did have a look at grunt, and gulp. But both projects introduce a lot of external dependencies and you need to learn one more "config language". :(

npm has some scripting features too, that @natecain introduced in #183 "Package.json rework"

356 wants to enhance the --build process using tiddlywiki as a command / framework.

362 wants to enhance the creation of editions, and plguins with an --init command.

So if TW is installed with npm install -g tiddlywiki everyone used to npm expects tw to behave like a stand alone app. So tiddlywiki could be our make, grunt ... buildsystem. With the advantage of no dependencies and the disadvantage, that we have to code the "framework commands" on our own.

I think all these topics belong together. ... As far as I can see, npm will be a fixed dependency. right?

@Jermolene we dicussed the possibilities to publish plugins with npm in the hangouts. @natecain said that there would be some new possibilities in upcomming npm version. ... but I couldn't find anything. So I hope he can join the converstion here.

So imo we need to define which commands tiddlywiki should do and which commands npm should be able to do.

[simonbaird]

@pmario Oh your bash script resembles my Makefile quite a bit. We seem to have similar motivations to clean that stuff up. :) I agree about Grunt btw.

Thanks for the pointer to #183. Actually I think the npm run blah is a good approach. And if #356 gets done then (as @Jermolene suggests) the majority of the scripts aren't needed much and the npm run commands could probably take the place of most of them.

That pull request is quite chunky though. Maybe it could be chopped up into easier to merge pieces. Eg, if Jeremy isn't sure about the versioning then maybe the package.json could be picked out into a smaller pull request.

Here's substack's notes on npm tasks. Seems like his approach would fit pretty well here.

Ps, re make as a dependency, I think the intersection of [people who can install node] and [people who don't already have make] would be fairly small! :)

[Jermolene]

Ps, re make as a dependency, I think the intersection of [people who can install node] and [people who don't already have make] would be fairly small! :)

One of those people would be @giffmex, who has gingerly been taking his first steps with node. He's installing with npm, and not running the batch files, so maybe he wouldn't see the make dependency.

[davidjade]

As a life long Windows developer with a lot of dev tools installed I can tell you, I probably don't have a make installed that is going to be compatible with a UNIX version. So changing the build to require a UNIX compatible make probably does mean an external dependency for me. And which make? GNU tools for Windows doesn't appear to have been updated since 2006.

That all said, the current scripts leave something to be desired and I really would like some sort of target based system for building, one that can easily be extended to build my custom TW targets, etc...

[simonbaird]

Renamed issue, previous title: "Consider coverting .sh and .cmd scripts into a Makefile (or perhaps Grunt tasks)"

[giffmex]

Nice to have you back, Simon!

On Thu, Jan 30, 2014 at 7:46 PM, Simon Baird notifications@github.comwrote:

Renamed issue, previous title: "Consider coverting .sh and .cmd scripts into a Makefile (or perhaps Grunt tasks)"

Reply to this email directly or view it on GitHubhttps://github.com/Jermolene/TiddlyWiki5/issues/383#issuecomment-33751913 .

David Gifford Christian Reformed World Missions, Mexico City

[simonbaird]

:)

[natecain]

@pmario The previous NodeJS lead just stepped down in order to focus on NPM, so large improvements should be coming (relatively) rapidly now.

For the moment, the only "official" way to pass parameters to scripts is to reference an environment variable like "$npm_package_config_FOO" and to call the script with npm run scriptname --tiddlywiki:FOO or npm run scriptname --tiddlywiki:FOO=something and the npm context will set the environment accordingly. This is nice because it avoids collisions in workflows where scripts in one package call scripts in another.

Initiatives like npm-exec try to offer more, and there is ongoing discussion on several open npm issues about how the scripts functionality should be improved.

I'd personally like to see a solution that avoids make files and bash-isms entirely and just sticks to JS, Node, NPM, and Tiddly itself.

[pmario]

I think we all would like to have a generic way, that works with every plattform. Jeremy opened a discussion at #390 "Should wiki and plugin folders be described by package.json"

[pmario]

just found: http://blog.keithcirkel.co.uk/how-to-use-npm-as-a-build-tool ....

I'll make some experiments again this weekend. ... especially the "watch" approach seems to be interesting.

[ght]

This can probably be closed, since the build process has evolved since this discussion.