issues with build (reproducability, architecture)

hohwille commented 8 years ago

First Problem: build is not reproducable node.js and gulp cause a lot of headaches and irreproducible builds.

People that come new into OASP check out the code and something that used to work then simply does not work anymore. The same code-base is leading to different build results for different moments in time. From an software-configuration-point of view this is more or less a catastrophy. This is really an extremely painful situation that we ran into many many times now and that we should get rid off entirely.

Second Problem: organisation of artifact repositories IMHO the entire architecture of how node_modules folders are organized and module-versioning is happening is a complete design flaw:

node_modules are just dumped directly into the current build directory without reuse for other projects
node_modules are chained recursively causing big problems on windows platforms with long file paths and the silly windows limitations
node_modules folders are not versioned

Maybe you can already say that maven is sick such that in order to start a small build for the first run you have to download half the internet to your local repo. However maven is putting each artifact version once onto your disk while node.js I have the same artifact version N*M times on the disc.

Is there any chance that this architecture will change in the future? Are we in discussion with the makers? I see a lot of pain-points in this area.

hohwille commented 8 years ago

For the second problem the makers already had the same thoughts and ideas as it seems: https://github.com/npm/npm/issues/3697

We should probably only upgrade and things get better...

hohwille commented 8 years ago

Current build error for latest codebase:

Γö£ΓöÇΓöÇ source-map-support@0.2.10 (source-map@0.1.32) ΓööΓöÇΓöÇ selenium-webdriver@2.45.1 (tmp@0.0.24, rimraf@2.4.3, xml2js@0.4.4, ws@ 0.7.2) [INFO] [INFO] --- exec-maven-plugin:1.3.1:exec (gulp) @ oasp4js-sample --- events.js:141 throw er; // Unhandled 'error' event ^ Error: EISDIR: illegal operation on a directory, read at Error (native) at Object.fs.readSync (fs.js:603:19) at Object.fs.readSync (C:\Projekte\oasp\workspaces\main\oasp4j\oasp4j-sample s\oasp4j-sample-server\src\main\client\node_modules\wiredep\node_modules\bower-c onfig\node_modules\graceful-fs\polyfills.js:218:23) at Object.fs.readFileSync (fs.js:438:24) at C:\Projekte\oasp\workspaces\main\oasp4j\oasp4j-samples\oasp4j-sample-serv er\src\main\client\node_modules\gulp-usemin\lib\blocksBuilder.js:64:24 at Array.forEach (native) at getFiles (C:\Projekte\oasp\workspaces\main\oasp4j\oasp4j-samples\oasp4j-s ample-server\src\main\client\node_modules\gulp-usemin\lib\blocksBuilder.js:61:17 ) at module.exports (C:\Projekte\oasp\workspaces\main\oasp4j\oasp4j-samples\oa sp4j-sample-server\src\main\client\node_modules\gulp-usemin\lib\blocksBuilder.js :89:20) at Transform._transform (C:\Projekte\oasp\workspaces\main\oasp4j\oasp4j-samp les\oasp4j-sample-server\src\main\client\node_modules\gulp-usemin\index.js:16:22 ) at Transform._read (C:\Projekte\oasp\workspaces\main\oasp4j\oasp4j-samples\o asp4j-sample-server\src\main\client\node_modules\gulp-usemin\node_modules\throug h2\node_modules\readable-stream\lib_stream_transform.js:184:10) [13:31:38] Using gulpfile C:\Projekte\oasp\workspaces\main\oasp4j\oasp4j-samples \oasp4j-sample-server\src\main\client\gulpfile.js

dumbNickname commented 8 years ago

Good point. Node modules flattening was optimized in newer releases of node.js, so it would be nice to upgrade here (why not directly to versions above 4.0 - after merge with socket.io). However flattening algorithm seems to be useful only in case when all the dependencies from all libraries are somehow aligned (same versions of dependencies used where possible). If not, the problem might still occur depending on Windows environment setup or the base path where the developer cloned the repository.

Even after applying the upgrade, we need to be aware that Window paths are limited and can cause problems. I would recommend to experiment with all known workarounds such as using symbolic links or switching to UNICODE paths (http://stackoverflow.com/questions/1880321/why-does-the-260-character-path-length-limit-exist-in-windows). When I faced this issue in the past I managed to overcome it with symbolic links approach, but maybe it would be nice to check UNICODE path possibilities like mentioned here: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath

"The Windows API has many functions that also have Unicode versions to permit an extended-length path for a maximum total path length of 32,767 characters. This type of path is composed of components separated by backslashes, each up to the value returned in the lpMaximumComponentLength parameter of the GetVolumeInformation function (this value is commonly 255 characters). To specify an extended-length path, use the "\?\" prefix. For example, "\?\D:\very long path". Note The maximum path of 32,767 characters is approximate, because the "\?\" prefix may be expanded to a longer string by the system at run time, and this expansion applies to the total length."

For sure topic needs some further investigation and a stable solution.

Cheers, Bartek

maybeec commented 8 years ago

Regarding problem1:

I fixed the latest build issues in PR #79. I tried to fix the dynamic dependencies as far as possible. I thing due to the current restrictions, we should try to fix all dependencies as far as possible including transitive ones.

To not start with overwhelming initial effort, I would rely on the currently fixed versions as far as we do not observe dynamic build issues in future. However, if there are build issues in future, we should identify the causing (transitive) component and fix its version in addition if not already done before.

Regarding problem2: node.js update :+1:

hohwille commented 8 years ago

See: https://github.com/oasp/oasp4j-ide/issues/74

hohwille commented 8 years ago

I actually updated node to 5.0.0 and npm to 3.4.1. However, after spending a lot of time to remove all my node_modules folders and calling npm install again I got the same crappy structure in node_modules. This all totally sucks. Why does the JS world have to reinvent the entire universe but doing so many mistakes that have already been solved in other ecosystems like java nicely after a lot of discussion, experience and sweat?

I went trough npm/npm#3697 again and read the entire thread. They only discussed about a nice solution but never implemented it. The whole issue is not fixed but there is a stupid workaround for a windows problem. The entire point is not only that the Windows filesystem sucks (what is for sure) but also that the entire node_modules structure sucks and is a total waste of disk space and directory nodes. It seems that we have to live with this crap!

hohwille commented 8 years ago

I also found this one: http://stackoverflow.com/questions/29786887/how-can-i-make-multiple-projects-share-node-modules-directory

I tested it and ... it does not work. I created a fresh node_modules folder using npm install for a project. Then I move this folder up to my top-level project folder. Calling npm install again simply ignored that top-level node_modules folder and loaded all the modules again from the web taking ages dumping my disk full of redundant stuff. Each node_modules folder of a sub-module of a project has more than thousand nested node_modules folders:

$ find . -name "node_modules" | wc -l
1792

And I have about 70 such sub-modules on my entire disc. A total count is over 50.000 node_modules on my entire disc. And I thought I just started with this modernized Angular and JS based stuff. Where will this end?

hohwille commented 8 years ago

So what I did now is spend another lot of time to delete millions of node_modules folders. Next I created one node_modules folder directly in each major project. For each submodule of such project, I created a symlink pointing on the main node_modules folder of the project. Then I can build my project with all sub-modules and at least only have one physical node_modules folder with thousand of nested node_modules folders for the entire project instead of having those again 10 or more times. This is still paintful but a lot better. To make it easier to create the symlinks in windows I recommend this tool: http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html#download

hohwille commented 8 years ago

So symlink is also not working properly:

npm WARN skippingAction Module is inside a symlinked module: not running add boo
m@2.10.1 node_modules\boom
npm WARN skippingAction Module is inside a symlinked module: not running add cry
ptiles@2.0.5 node_modules\cryptiles
npm WARN skippingAction Module is inside a symlinked module: not running move ht
ml-entities@1.1.3 node_modules\html-entities
npm WARN skippingAction Module is inside a symlinked module: not running add htt
p-signature@0.11.0 node_modules\http-signature
npm WARN skippingAction Module is inside a symlinked module: not running add is-
property@1.0.2 node_modules\is-property
npm WARN skippingAction Module is inside a symlinked module: not running add gen
erate-object-property@1.2.0 node_modules\generate-object-property
npm WARN skippingAction Module is inside a symlinked module: not running add iss
tream@0.1.2 node_modules\isstream
npm WARN skippingAction Module is inside a symlinked module: not running move ja
sminewd@1.1.0 node_modules\jasminewd
npm WARN skippingAction Module is inside a symlinked module: not running add jas
minewd2@0.0.6 node_modules\jasminewd2
npm WARN skippingAction Module is inside a symlinked module: not running move js
on-stringify-safe@5.0.1 node_modules\json-stringify-safe
npm WARN skippingAction Module is inside a symlinked module: not running add jso
npointer@2.0.0 node_modules\jsonpointer
npm WARN skippingAction Module is inside a symlinked module: not running add is-
my-json-valid@2.12.3 node_modules\is-my-json-valid
npm WARN skippingAction Module is inside a symlinked module: not running add har
-validator@1.8.0 node_modules\har-validator
npm WARN skippingAction Module is inside a symlinked module: not running update
istanbul@0.4.0 node_modules\karma-coverage\node_modules\istanbul
npm WARN skippingAction Module is inside a symlinked module: not running remove
glob@5.0.15 node_modules\karma-coverage\node_modules\istanbul\node_modules\files
et\node_modules\glob
npm WARN skippingAction Module is inside a symlinked module: not running update
minimatch@3.0.0 node_modules\karma-coverage\node_modules\minimatch
npm WARN skippingAction Module is inside a symlinked module: not running update
source-map@0.5.3 node_modules\karma-coverage\node_modules\source-map
npm WARN skippingAction Module is inside a symlinked module: not running remove
lodash@3.9.3 node_modules\karma-phantomjs-launcher\node_modules\lodash
npm WARN skippingAction Module is inside a symlinked module: not running remove
unpipe@1.0.0 node_modules\karma\node_modules\connect\node_modules\body-parser\no
de_modules\raw-body\node_modules\unpipe
npm WARN skippingAction Module is inside a symlinked module: not running remove
connect@2.30.2 node_modules\karma\node_modules\connect
npm WARN skippingAction Module is inside a symlinked module: not running remove
glob@5.0.15 node_modules\karma\node_modules\glob
npm WARN skippingAction Module is inside a symlinked module: not running update
graceful-fs@4.1.2 node_modules\karma\node_modules\graceful-fs
npm WARN skippingAction Module is inside a symlinked module: not running remove
http-proxy@0.10.4 node_modules\karma\node_modules\http-proxy
...

npm is so much insane that it gets inconsistent within the same project. I am giving up with npm. I always thought that maven has some drawbacks and would need some improvements but after working with npm and node I have to say that maven is such a great tool...

hohwille commented 8 years ago

tbialecki commented 8 years ago

For deleting node_modules directory i recommend rimraf tool. It can be installed with npm install -g rimraf 23 lis 2015 23:40 "Jörg Hohwiller" notifications@github.com napisał(a):

See also npm/npm#8681 https://github.com/npm/npm/issues/8681

— Reply to this email directly or view it on GitHub https://github.com/oasp/oasp4js/issues/73#issuecomment-159090527.

hohwille commented 8 years ago

http://www.jongleberry.com/why-i-hate-npm.html

hohwille commented 8 years ago

To avoid this extreme redundancies and waste of diskspace I also tried to convert my package.json to a list of global installs like:

npm install -g gulp@3.9.0
npm install -g gulp-angular-templatecache@1.8.0
npm install -g gulp-concat@2.6.0
npm install -g gulp-env@0.2.0
npm install -g gulp-eslint@1.0.0
npm install -g gulp-file@0.2.0
npm install -g gulp-filter@3.0.1
npm install -g gulp-flatten@0.2.0
npm install -g gulp-git@1.6.0
npm install -g gulp-if@2.0.0
npm install -g gulp-imagemin@2.4.0
npm install -g gulp-insert@0.5.0
npm install -g gulp-less@3.0.5
npm install -g gulp-load-plugins@1.1.0
npm install -g gulp-minify-css@1.2.1
npm install -g gulp-minify-html@1.0.4
npm install -g gulp-newer@1.0.0
npm install -g gulp-ng-annotate@1.1.0
npm install -g gulp-plumber@1.0.1
npm install -g gulp-processhtml@1.1.0
npm install -g gulp-protractor@1.0.0
npm install -g gulp-replace@0.5.4
npm install -g gulp-rev@6.0.1
npm install -g gulp-size@2.0.0
npm install -g gulp-sonar@2.0.0
npm install -g gulp-sourcemaps@1.6.0
npm install -g gulp-sync@0.1.4
npm install -g gulp-tslint@3.6.0
npm install -g gulp-uglify@1.5.1
npm install -g gulp-usemin@0.3.15
npm install -g gulp-util@3.0.7
npm install -g gulp.spritesmith@5.0.1
npm install -g gulp-callback@0.0.3
npm install -g http-proxy@1.12.0
npm install -g jasmine-core@2.3.4
npm install -g karma@0.13.15
npm install -g karma-chrome-launcher@0.2.1
npm install -g karma-coverage@0.5.3
npm install -g karma-jasmine@0.3.6
npm install -g karma-junit-reporter@0.3.8
npm install -g karma-phantomjs-launcher@0.2.1
npm install -g karma-systemjs@0.9.0
npm install -g lodash@3.10.1
npm install -g ng-parse-module@0.1.0
npm install -g phantomjs@1.9.18
npm install -g phantomjs-polyfill@0.0.1
npm install -g require-dir@0.3.0
npm install -g string@3.3.1
npm install -g systemjs@0.19.4
npm install -g systemjs-builder@0.14.8
npm install -g through2@2.0.0
npm install -g tsd@0.6.5
npm install -g typescript@1.6.2
npm install -g uglify-save-license@0.4.1
npm install -g yargs@3.30.0
npm install -g main-bower-files@2.9.0

Even after having that installed globally a local install again downloads this to the local node_modules folder. I really have to stop thinking about this stupid tool and just accept that it is causing a world-wild waste of time, disk space and so forth. Sorry, for this harsh flame war on npm but I am really annoyed...

hohwille commented 8 years ago

Shall we close this issue as wontfix?

hohwille commented 8 years ago

Each time I call npm install I get a different error:

>npm install
npm ERR! tar.unpack error reading C:\Users\hohwille\AppData\Local\Temp\npm-9620-
1185affe\registry.npmjs.org\object-assign\-\object-assign-4.0.1.tgz
npm ERR! Windows_NT 6.1.7601
npm ERR! argv "C:\\Projekte\\software\\nodejs\\node.exe" "C:\\Users\\hohwille\\AppData\\Roaming\\npm\\node_modules\\npm\\bin\\npm-cli.js" "install"
npm ERR! node v5.0.0
npm ERR! npm  v3.4.1
npm ERR! code EBADF
npm ERR! errno -4083
npm ERR! syscall fstat

npm ERR! EBADF: bad file descriptor, fstat
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     C:\Projekte\workspaces\main\app\src\main\client\npm-debug.log

>npm install
npm WARN install Couldn't install optional dependency: Unsupported
npm WARN install Couldn't install optional dependency: Unsupported
npm ERR! Windows_NT 6.1.7601
npm ERR! argv "C:\\Projekte\\software\\nodejs\\node.exe" "C:\\Users\\hohwille\\AppData\\Roaming\\npm\\node_modules\\npm\\bin\\npm-cli.js" "install"
npm ERR! node v5.0.0
npm ERR! npm  v3.4.1
npm ERR! code ECONNRESET
npm ERR! errno ECONNRESET
npm ERR! syscall read

npm ERR! network read ECONNRESET
npm ERR! network This is most likely not a problem with npm itself
npm ERR! network and is related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settin
gs.
npm ERR! network
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly.  See: 'npm help config'

Maybe I am too stupid or too smart for this tool - who knows. I still would expect a build tool to be reproducible.

hohwille commented 8 years ago

Also nice:

npm WARN EPACKAGEJSON app@0.0.1 No repository field.
npm WARN EPACKAGEJSON app@0.0.1 No license field.
npm ERR! Windows_NT 6.1.7601
npm ERR! argv "C:\\Projekte\\software\\nodejs\\node.exe" "C:\\Users\\hohwille\\AppData\\Roaming\\npm\\node_modules\\npm\\bin\\npm-cli.js" "install"
npm ERR! node v5.0.0
npm ERR! npm  v3.4.1

npm ERR! Callback called more than once.
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

hohwille commented 8 years ago

https://github.com/nodejs/node-v0.x-archive/issues/6960

maybeec commented 8 years ago

current state:

npm now supports flat file structure. Therefore, the file issue is gone.
shrinkjs enables a complete reproducable build regarding a fixed snapshot of dependencies.

I will close this issue for now. If there are further issues. Please consider stating a new issue if needed.

oasp / oasp4js

issues with build (reproducability, architecture) #73