Build and serve the frameworks on windows or in a docker container

krausest commented 5 years ago

Building natively on windows is currently not possible:

The angular, angular-ivy, angular-noopzone, angular-optimized builds don't work properly on windows (#532) though you can continue by pressing enter (update: doesn't create a valid file though)
Elm fails (ERROR: 'pure_funcs=F2,F3,F4,F5,F6,F7,F8,F9,A2,A3,A4,A5,A6,A7,A8,A9,pure_getters=true,keep_fargs=false,unsafe_comps=true,unsafe=true,passes=2' is not a supported option at DefaultsError.get)
reagent (#377)
reason_react (Fatal error: exception Unix.Unix_error(20, "stat", "C:\Users\Stefa\Javascript\js-framework-benchmark2\frameworks\keyed\reason-react\src\")
- stdweb (error: linking with link.exe failed: exit code: 1). I guess other rust based frameworks might also not work.

Alternative would be a use a docker container for build all frameworks. Benefits:

Reproducible builds
Manage just one environment (no more issues like #533)

Drawbacks:

Requiring docker might add further complexity to the build process
For development a local build might be faster (=nicer)

Consequences:

Require all (?) frameworks to submit a build that works in the docker container. Would that affect acceptance? What about building reflex-dom?

Feel free to comment!

alexfmpe commented 5 years ago

I'd recommend building inside Windows Subsystem for Linux. It's possible that everything just works as usual there.

krausest commented 5 years ago

Currently webdriver-ts has a good chance to run into the following issue which seems to be caused by hyper-v blocking some ports

error running benchmark { Error: listen EACCES: permission denied 0.0.0.0:49957
    at Server.setupListenHandle [as _listen2] (net.js:1241:19)
    at listenInCluster (net.js:1306:12)
    at Server.listen (net.js:1394:7)
    at Promise (C:\Users\Stefa\Javascript\js-framework-benchmark2\webdriver-ts\node_modules\selenium-webdriver\net\portprober.js:159:12)
    at new Promise (<anonymous>)
    at isFree (C:\Users\Stefa\Javascript\js-framework-benchmark2\webdriver-ts\node_modules\selenium-webdriver\net\portprober.js:150:10)
    at Object.findFreePort (C:\Users\Stefa\Javascript\js-framework-benchmark2\webdriver-ts\node_modules\selenium-webdriver\net\portprober.js:178:15)
    at processTicksAndRejections (internal/process/task_queues.js:86:5)

Seems like portprober from selenium isn't doing a good job to find a free port. It sometimes chooses a port that is excluded according to netsh interface ipv4 show excludedportrange protocol=tcp

λ netsh interface ipv4 show excludedportrange protocol=tcp

Portausschlussbereiche für das Protokoll "tcp"

Startport      Endport
----------    --------
      5357        5357
      5426        5426
     49692       49791
     49792       49891
     49892       49991
     49992       50091
     50092       50191
     50501       50600

* - Verwaltete Portausschlüsse.

So we should add an option to select a port (or make this the default?).

krausest commented 5 years ago

This comment collects information building on WSL. Preparation: Install Ubuntu 18.04

Install node and java

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.34.0/install.sh | bash
source /home/stefan/.bashrc
nvm install v10.15.3

sudo add-apt-repository ppa:openjdk-r/ppa
sudo apt-get update
sudo apt-get install openjdk-8-jdk
sudo apt-get install build-essential

run npm install and npm run build-prod Currently one framework fails (which is not bad actually):

pux:


*** Executing npm run build-prod in frameworks/non-keyed/pux

js-framework-benchmark-pux@1.0.0 build-prod /mnt/c/Users/Stefa/Javascript/js-framework-benchmark2/frameworks/non-keyed/pux pulp browserify --optimise --to output/bundle.js

Browserifying project in /mnt/c/Users/Stefa/Javascript/js-framework-benchmark2/frameworks/non-keyed/pux
Building project in /mnt/c/Users/Stefa/Javascript/js-framework-benchmark2/frameworks/non-keyed/pux The futex facility returned an unexpected error code.* Build successful. purs bundle: No input files.

webdriver-ts must be built and run on windows.

ryansolid commented 5 years ago

Oh so you got this all working in WSL? I've been using WSL for ages now but it's always been a bit shoddy, disconnecting a decent amount and since you upgraded lighthouse startup metrics I've never been able to get the startup tests to run properly. Maybe I've been doing it wrong. I will be super interested your setup looks like when you get this all working.

krausest commented 5 years ago

This collects information for a docker with bind mounts. Not really working.

Using bind mount for the js-framework-benchmark directory

Bind mount on windows causes all files to be writeable for root only. Building as root with bind mount causes many issues:

angular-ivy:

> angular-closure@1.0.0 ngc /build/frameworks/keyed/angular-ivy
> ngc -p tsconfig.json

: Error: Module not found: CommonModule
    at SelectorScopeRegistry.lookupScopesOrDie (/build/frameworks/keyed/angular-ivy/node_modules/@angular/compiler-cli/src/ngtsc/annotations/src/selector_scope.js:142:23)
    at /build/frameworks/keyed/angular-ivy/node_modules/@angular/compiler-cli/src/ngtsc/annotations/src/selector_scope.js:172:121
    at Array.map (<anonymous>)
    at SelectorScopeRegistry.lookupScopes (/build/frameworks/keyed/angular-ivy/node_modules/@angular/compiler-cli/src/ngtsc/annotations/src/selector_scope.js:172:87)
    at SelectorScopeRegistry.lookupScopesOrDie (/build/frameworks/keyed/angular-ivy/node_modules/@angular/compiler-cli/src/ngtsc/annotations/src/selector_scope.js:140:31)
    at SelectorScopeRegistry.lookupCompilationScopeAsRefs (/build/frameworks/keyed/angular-ivy/node_modules/@angular/compiler-cli/src/ngtsc/annotations/src/selector_scope.js:111:18)
    at SelectorScopeRegistry.lookupCompilationScope (/build/frameworks/keyed/angular-ivy/node_modules/@angular/compiler-cli/src/ngtsc/annotations/src/selector_scope.js:136:30)
    at ComponentDecoratorHandler.compile (/build/frameworks/keyed/angular-ivy/node_modules/@angular/compiler-cli/src/ngtsc/annotations/src/component.js:180:44)
    at IvyCompilation.compileIvyFieldFor (/build/frameworks/keyed/angular-ivy/node_modules/@angular/compiler-cli/src/ngtsc/transform/src/compilation.js:163:34)
    at IvyVisitor.visitClassDeclaration (/build/frameworks/keyed/angular-ivy/node_modules/@angular/compiler-cli/src/ngtsc/transform/src/transform.js:48:40)

npm ERR! code ELIFECYCLE
npm ERR! errno 2

binding-scala:

> sbt-bin@0.1.2 postinstall /build/frameworks/keyed/binding.scala/node_modules/sbt-bin                     
> node install.js                                                                                          

internal/modules/cjs/loader.js:584                                                                         
    throw err;                                                                                             
    ^                                                                                                      

Error: Cannot find module 'ini'                                                                            
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:582:15)                            
    at Function.Module._load (internal/modules/cjs/loader.js:508:25)                                       
    at Module.require (internal/modules/cjs/loader.js:637:17)                                              
    at require (internal/modules/cjs/helpers.js:22:18)                                                     
    at Object.<anonymous> (/build/frameworks/keyed/binding.scala/node_modules/rc/lib/utils.js:3:12)        
    at Module._compile (internal/modules/cjs/loader.js:701:30)                                             
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10)                               
    at Module.load (internal/modules/cjs/loader.js:600:32)                                                 
    at tryModuleLoad (internal/modules/cjs/loader.js:539:12)                                               
    at Function.Module._load (internal/modules/cjs/loader.js:531:3)

elm

> elm@0.19.0-bugfix2 install /build/frameworks/keyed/elm/node_modules/elm
> binwrap-install

fs.js:114
    throw err;
    ^

Error: EINVAL: invalid argument, open '/build/frameworks/keyed/elm/node_modules/elm/package.json'
    at Object.openSync (fs.js:438:3)
    at Object.readFileSync (fs.js:343:35)
    at Object.Module._extensions..json (internal/modules/cjs/loader.js:718:20)
    at Module.load (internal/modules/cjs/loader.js:600:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:539:12)
    at Function.Module._load (internal/modules/cjs/loader.js:531:3)
    at Module.require (internal/modules/cjs/loader.js:637:17)
    at require (internal/modules/cjs/helpers.js:22:18)
    at Object.<anonymous> (/build/frameworks/keyed/elm/node_modules/binwrap/bin/binwrap-install:4:19)
    at Module._compile (internal/modules/cjs/loader.js:701:30)
npm WARN rollback Rolling back balanced-match@1.0.0 failed (this is probably harmless): ENOTEMPTY: directory not empty, rmdir '/build/frameworks/keyed/elm/node_modules/balanced-match'

glimmer:

*** Executing npm install in frameworks/keyed/glimmer

> node-sass@4.11.0 install /build/frameworks/keyed/glimmer/node_modules/node-sass
> node scripts/install.js

fs.js:114
    throw err;
    ^

Error: EINVAL: invalid argument, open '/build/frameworks/keyed/glimmer/node_modules/node-sass/package.json'
    at Object.openSync (fs.js:438:3)
    at Object.readFileSync (fs.js:343:35)
    at Object.Module._extensions..json (internal/modules/cjs/loader.js:718:20)
    at Module.load (internal/modules/cjs/loader.js:600:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:539:12)
    at Function.Module._load (internal/modules/cjs/loader.js:531:3)
    at Module.require (internal/modules/cjs/loader.js:637:17)
    at require (internal/modules/cjs/helpers.js:22:18)
    at Object.<anonymous> (/build/frameworks/keyed/glimmer/node_modules/node-sass/lib/extensions.js:7:9)
    at Module._compile (internal/modules/cjs/loader.js:701:30)

to be continued. Adding RUN echo "unsafe-perm = true" > /root/.npmrc helps a bit (angular-ivy works then. 9

Seems like building with node.js in docker with bind volumes works much smoother if you shadow map a volume on node_modules --volume C:/Users/Stefa/Source/Javascript/js-framework-benchmark/frameworks/keyed/glimmer:/build --volume mods:/build/node_modules). The build works for glimmer and elm then. But I have no clue how to scale that to 100+ frameworks.

krausest commented 5 years ago

This one avoids sharing folders with windows and instead uses a docker volume and copies file to docker. Sounds awful but it works.

docker\Dockerfile

FROM centos:latest
RUN echo "unsafe-perm = true" > /root/.npmrc
RUN echo "{ \"allow_root\": true }" >  /root/.bowerrc
RUN curl -sL https://rpm.nodesource.com/setup_10.x | bash -
RUN yum install -y nodejs gcc-c++ make git java

RUN mkdir /server
RUN mkdir /build

COPY docker/package.json /server
WORKDIR /server
RUN npm install 

# Volume before chown changes owwner
VOLUME /build
WORKDIR /build

# USER user

# RUN npm install
EXPOSE 8080
CMD ["/server/node_modules/.bin/http-server","-c-1","/build"]

Build the dockerfile with docker build -t js-framework-benchmark-centos -f docker\Dockerfile .
Create a docker volume docker volume create js-framework-benchmark
Run a docker container and use the volume docker run --rm -it -p 8080:8080 --name js-framework-benchmark --volume js-framework-benchmark:/build js-framework-benchmark-centos
Copy files from the host docker cp . js-framework-benchmark:/build
Start installing from the host docker exec -it js-framework-benchmark npm install
Start the build docker exec -it js-framework-benchmark node build.js

krausest commented 5 years ago

Vagrant

Seems like vagrant with a shared folder doesn't work either:

[vagrant@localhost build]$ node build.js --benchmarks_only
*** Executing npm install in frameworks/keyed/angular
npm WARN rollback Rolling back ansi-regex@2.1.1 failed (this is probably harmless): ETXTBSY: text file is busy, unlink '/build/frameworks/keyed/angular/node_modules/ansi-regex/package.json.3757287570'
npm WARN rollback Rolling back @types/node@11.13.0 failed (this is probably harmless): ETXTBSY: text file is busy, unlink '/build/frameworks/keyed/angular/node_modules/@types/node/package.json.2763824670'
npm WARN rollback Rolling back arr-flatten@1.1.0 failed (this is probably harmless): ETXTBSY: text file is busy, unlink '/build/frameworks/keyed/angular/node_modules/arr-flatten/package.json.3817453258'
npm WARN rollback Rolling back @types/estree@0.0.39 failed (this is probably harmless): ETXTBSY: text file is busy, unlink '/build/frameworks/keyed/angular/node_modules/@types/estree/package.json.4197299982'
npm WARN rollback Rolling back arr-diff@2.0.0 failed (this is probably harmless): ETXTBSY: text file is busy, unlink '/build/frameworks/keyed/angular/node_modules/arr-diff/package.json.64237426'
npm WARN rollback Rolling back ansi-styles@2.2.1 failed (this is probably harmless): ETXTBSY: text file is busy, unlink '/build/frameworks/keyed/angular/node_modules/ansi-styles/package.json.3850144723'
npm WARN rollback Rolling back array-unique@0.2.1 failed (this is probably harmless): ETXTBSY: text file is busy, unlink '/build/frameworks/keyed/angular/node_modules/array-unique/package.json.2315464958'
npm WARN rollback Rolling back arr-union@3.1.0 failed (this is probably harmless): ETXTBSY: text file is busy, unlink '/build/frameworks/keyed/angular/node_modules/arr-union/package.json.3760012614'
...
npm ERR! path /build/frameworks/keyed/angular/node_modules/copy-descriptor/package.json.2575146664                                                                                                                      
npm ERR! code ETXTBSY                                                                                                                                                                                                   
npm ERR! errno -26                                                                                                                                                                                                      
npm ERR! syscall rename                                                                                                                                                                                                 
npm ERR! ETXTBSY: text file is busy, rename '/build/frameworks/keyed/angular/node_modules/copy-descriptor/package.json.2575146664' -> '/build/frameworks/keyed/angular/node_modules/copy-descriptor/package.json'       

npm ERR! A complete log of this run can be found in:                                                                                                                                                                    
npm ERR!     /home/vagrant/.npm/_logs/2019-04-14T15_16_20_666Z-debug.log                                                                                                                                                
child_process.js:656                                                                                                                                                                                                    
    throw err;                                                                                                                                                                                                          
    ^                                                                                                                                                                                                                   

Error: Command failed: npm install

krausest commented 5 years ago

Current state: The build works for a container or vm with a native linux filesystem. Currently node/npm don't work for every framework with a shared directory.

krausest commented 5 years ago

Docker [unfinished] Here's a refined docker approach: Install dependencies npm install

Install webdriver-ts and webdriver-ts-results dependencies. Both will be executed on your host machine. npm run install-local

Make sure docker is running. Prepare a docker volume: docker volume create js-framework-benchmark

Build the docker image [maybe one day pull image from registry] npm run docker-build

Start the docker container npm run docker-start

Build all frameworks with docker npm run docker-build-frameworks

Rebuild a framework in docker and run some smoke tests. It first syncs the source code, then deletes and reinstalls the node modules, builds the framework, runs all benchmarks with a small iteration count and checks whether the keyed categorization is correct with a headless chrome. Very recommended before submitting a PR: npm run docker-rebuild non-keyed/surplus

Run the benchmark from the local machine: npm run bench or if you want to control which frameworks cd webdriver-ts run whatever framework you want npm run bench -- --framework vanillajs-keyed --benchmark 01_ --count 3 and update the result table npm run results

Here's a picture of the setup: Docker

Get rid of it again Stop the docker container npm run docker-stop Delete the volume docker volume rm js-framework-benchmark

[To be continued...]

krausest commented 5 years ago

Docker tips & tricks [unfinished]

Building all frameworks failed for some framework and you want to restart from that one after fixing the implementation: npm run docker-build-frameworks -- --restartWith literaljs

krausest commented 5 years ago

Just for reference: Ubuntu 19.10: sudo apt install openjdk-8-jdk sudo update-java-alternatives --set java-1.8.0-openjdk-amd6 [not finished yet]

krausest commented 3 years ago

Closing this issue. Information about the docker build is in the wiki.

krausest / js-framework-benchmark