Building moOde with Gulp

[bitkeeper]

Gulp is a toolkit to automate and enhance the workflow of webapps. For C-minded see it like Make or Java troops just like ant.

Gulp requires npm to be installed.

At the moment I'm working on two different use-cases with gulp:

• Building the frontend webapp:

  • minify every js and css
  • no need to checkin intermediate files like .min.js|css anymore
  • bundle all the js and css file to one js and one css file (going 27 resources to 2 resources).
  • generate sourcemaps for the browser to still make easy debugging possible
  • automatic generate header or update header of files with the current date and or version info
  • deploy a build (like ~pi/moode/distr) to the final location (like /var/www)
  • automatic apply css browser prefixes, so you don't have to
  • make the use sass for css possible

• Make it possible to do frontend development work on my pc

  • use the development tools of your flavor on your PC
  • perform a build of the webapp
  • start a local test webserver (exludes php)
  • which include a proxy to your moode instance for the backend calls
  • optional automatic reload browser on code change
  • it will be more in the comfort zone of a webdev

If you are interested and or have feedback pleas join the discussion.

[bitkeeper]

/**
 * 
 * Description: gulp buildscript for frontend 
 * 
 * Features:
 * - provide automatic bundling of the javascripts files.
 * - uses a cache for speed
 * - provide sourcemaps for browser side debugging
 * - patches files to use the bundles
 * - easy way to deploy
 * - webserver for frontend development (backup running on moode instance, webserver on desktop computer, uses proxy) 
 * 
 * Usage:
 * 
 * 
 * Install:
 * - install npm (apt-get npm, mac and windows download and running installers)
 * - from the moode git repro directory run npm ci (installs the required npm modules)
 * - call a build target
 * - check if your environment is ok by running gulp -v .
 *   - if says 'command not found', run node_modules/.bin/gulp instead
 * - now you can run commands like:
 * 
 * Main build targets:
 *  gulp clean [--all]          - Empties the build/build and build/distr directory, with the --all also the cache is flushed.
 * 
 *  gulp build [--force]        - Build frontend from {app.src} and put output in {app.dest}
 *  gulp watch [--build]        - Start web server with as root {app.src} with proxy to moode, used for local web development. 
 *             [--force]                 When --build is given, use the build output {app.dest} as web root
 *  gulp deploy [--test]        - Deploys everything needed (inc php etc) {app.deploy}.
 *              [--force]                  With the option --test deploy to build/dist (app.dist).
 *                                When used to real don't forget to sudo first
 * 
 *  Default most task only update/use files that are changed (= is newer).  With force the files are also updated.
 * 
 * Sub build targets (are automaticly called by the main build targets on need):
 *  gulp cache                  - fill cache with files (basedon changed and index.html)
 *  gulp bundle                 - (re)bundle the files
 *  gulp listplugins            - list available gulp plugins
 * 
 * Generated Directory tree:
 * 
 * moode
 *  |- build               - directory used by the build scripts for temporary files, no need to checkin. maybe cache checkin for speed.
 *     |- cache            - temporary cache with all individual minified js/css files
 *          |- css         - stylesheets
 *          |- javascript  - javascript
 *          |- maps        - matching sourcemaps
 *     |- build            - all build bundles, including enough files to run again a local js development server
 *     |- distr            - result of a test deploy
 * 
 */

If the buildin webserver is very slow, your hitting a know issue with local networks that have .local as domain. In that case just add to your hosts file moode.test with the specific ip number.

[moodeaudio]

I'll get to looking at this after I finish the new Radio feature which will take me some number of weeks worth of effort.

[bitkeeper]

@moodeaudio I think the build with gulp is finished enough to give it a spin. Out of the box it does nothing. You can continue the way you are used to build/deploy moOde, but you have the possibility to test/use the new frontend build.

No code is broken or changed. The only noticeable changes to existing files are some additional comment that are used as markers for the build process. The added index.html is at the moment only used for the frontend development, on deploy the index|header|footer.php is used.

I tested the build and proxy server for frontend development of the main app (index.php) on both a Raspberry Pi 4 and my desktop. The deployment is of course only tested on the Pi it self.

[moodeaudio]

Does this include the PHP back-end?

I'm a bit confused.

[monkeypox]

If I understand correctly it uses a proxy to route the php calls to moode running on the Pi.

There are two major advantages. First it’d be faster to test client side changes since you don’t need to copy to the pi first. Second and probably more importantly for you is it seems to handle all the minifying automatically so there’s no need to keep the minified versions in the dev tree as that step gets done automatically when you “deploy” moode. Iirc the minification step additionally conglomerates the css & js files which should improve load performance a bit but I don’t know how measurable that’d be. Modern browsers are all multithreaded so loading lots of smaller files while a bigger transaction happens (aka the library load) as well as any other startup actions might have hidden the cost a bit.

Marcel can probably chip in with more advantages.

The downside is it’s an unfamiliar workflow to learn so you’ll have to weigh whether it ends up being a benefit or not. I think it eventually would be since it should lessen the maintenance load but only you can be the judge of that.

Sent from my iPhone

On Jun 2, 2020, at 7:53 PM, Tim Curtis notifications@github.com wrote:

 Does this include the PHP back-end?

I'm a bit confused.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[bitkeeper]

Sorry if below is a little bit to long to read.

@monkeypox your are right, when used in development mode with the watch command(on the pi or desktop) it runs proxy to relay the calls to the php backend (the moOde instance).

For the development mode it only concerns the webfrontend excluding the php, however with the deploy function everything in moode/www (including the php etc ) and moode/var/local/www/header.php is deployed.

Devlopment

example development session:

• Open Visual Studio
• Run in the terminal 'gulp watch' (Browser is automatically opened or open your own)
• This will do a minimal build and start the webserver with proxy
• Edit a css setting
• The moment you press save, the change will be seen and the browser will reload automatically.
• When you done making all the changes, you just commit the changed sources with out the min files.

Till so far you still can use the same way as always for create a moOde release, test run or what ever. It only speeds up the way you develop the frontend.

Deploy

The next big change comes when using build and deploy commands. Here the minified and bundled files are used. To make sure you can still debug stuff also sourcemaps are created. This time it will influence the way how a moOde release is created. The bundling reduces the number of loaded files. And from my personal and work experience it will reduce the load time and only have to insert the script in document once.

In the development mode you can already try to used the build files by adding --build to the wach command like gulp watch --build. Now the webserver will serves the minified and bundled sources. When you make a change to the source code it will automatically rebuild you code and reload the browser.

To use it on the backend or build a release:

• checkout the correct source tree on the pi
• call the gulp build command to create a build
• call the gulp deploy command (or gulp deploy --test so you can inspect the output before deploy) to place the code to /var/www

During the deploy files are selective copied. Also some files are patched, like header.php and footer.php to use the bundled files. Optional also headers can be added or updated with for example version number, release date etc.

You can even run the watch command if you like.

offcourse I assumed hereby that npm is already installed and the package deps are also installed as stated in the gulpfile.js.

Gulp

npm with gulp is tool that is often used for webdevelopment. There are many more flavour to choose from. There is no magic the gulpfile.js contains all commands and can be easly customized. The npm package file package.json contains the settings that are use in the gulp commands. It contains also the dependencies for development tooling (a bunch now) and the app it self (none at the moment). You could reinstall the deps from the package.json with npm install, but a better pratice is to include the package-lock.json with npm ci. Which also list the deps of the deps etc. This will make sure you always get exactly the same versions of the deps.

The current PR is a git squash of a number of changes. I'm already using it for the development work and to deploy it. It simplified my workflow and I don't forget to copy or min any file any more. Off course there is room for improvement, but I think it will make a good starting point.

That having said I tested on a Pi2, Pi3 and Pi4. Installing the first time npm and the depencyies takes some time. Using the watch, build and deploy are usable on the Pi3 and Pi4. I found the Pi2 to slow, but I am not known for my patience ;-)

If you any further questions don't hesitate to ask. If you want an online demonstration with teamviewer or so just let me known.

[moodeaudio]

Currently I'm using Atom / GitHub Desktop combo plus a build script that runs on the Pi and some online services to minify JS/CSS/HTML. Minifying is a bit of a pain but its mainly due to not having the time to install and configure local minifiers on my iMac. They all seem to require installing and configuring other stuff like node.js or Java.

[bitkeeper]

More and more webdevelopment tools are writen in javascript and use the package manager npm. Which seems logic; it stays then in the same environment, so that webdevs can develop their own tools.

[bitkeeper]

@moodeaudio if you don't mind,to prevent polluting the forum thread I answer your question here: "Btw, what minifiers are u using in the gulp builder?"

For ugilifying the gulp plugin gulp-uglify-es is used. This is just a thin wrapper for the actual uglifier very popular terser, which is also a npm package.

Fortunately you don't have to install the uglifier yourself, the only requirement is that npm is installed. If you run the command npm ci from within the moode directory all deps are automaticly installed (even the deps of the deps and so on, thanks to the package-lock.json file).

Btw you can find the used npm packages in the package.json file under the section devDependencies.

[bitkeeper]

While still trying to improve the deployment target, two questions surfaced:

1. The location of the header.php. Is there a special reason why it it located in /var/local/www as opposed to where the rest is in /var/www?

2. Different way around for the mpd binaries. Those are located in /var/www/inc, but clearly no part of the source tree. Those are pretty static and with a deployment cycle the bins will take 70% of the time. Should those better be moved to /var/local/www/mpd_binaries ?

If one or both are allowed to be moved, I will create separate PRs for it. Offcourse for the release after 6.6.x ;-)

To get the source tree as clean as possible, makes the source tree easier to understand and makes a possible buildsystem more efficient.

[moodeaudio]

1 - header.php contains the Browser Title which gets a SED edit. In the past the entire /var/www dir was a squashfs read-only file system. Anything that was editable was put in var/local/www/... I don't use squashfs anymore so maybe there is a better way to handle the Browser Title edit.

2 - Yes they should be moved to /var/local/www/mpd_binaries

[bitkeeper]

Because of merge of PR.