remove outdated/breaking folders

[WebReflection]

It takes a while to have an overview of all benchmarks in here, and having folders incapable to npm ci && npm run build-prod is not DX friendly at all.

rm -rf frameworks/keyed/react-focal
rm -rf frameworks/keyed/reaml-react
rm -rf frameworks/keyed/reason-react

all these commands where needed, for the keyed benchmark, to be able to follow instructions, and I'd be more than happy to file an MR that at least remove these outdated, non-maintained, or non documented, in terms of procedure, from this repository.

Would that be welcomed? If that's the case, I might as well check which one fails in non-keyed folder.

About "flagging issue though", there are frameworks with severe vulnerabilities in their current state and others not building at all, and these would be the only real issue this benchmark should expose out there, imho, or maybe an occasion to cleanup abandoned, outdated, libraries and "experiments", as there are maintainers and maintainers to consider as well.

[krausest]

(P.S: I'm trying to ignore my overly sensitive sentiment sensor in my stomach)

Here are the results from my investigation:

1) Broken builds You can build all frameworks via npm run build-prod in the root folder as mentioned in the README. The other way is to build via a docker image as described in https://github.com/krausest/js-framework-benchmark/wiki/Build-all-frameworks-with-docker. The docker image is my/the reference build environment and all of the four frameworks build in that environment. I'm trying to rebuild all frameworks before publishing "official" results, i.e. every new chrome version. Should any build fail in that environment an issue will be created if it can't be fixed immediately (e.g. #892). I think about migrating the benchmark to my MacBook Air and I'm very sure all above built recently via rosetta node (react-focal didn't build in arm64 node though as mentioned in https://github.com/krausest/js-framework-benchmark/issues/885#issuecomment-850396741). (Checking for vulnerabilities is out of scope for this benchmark.)

2) Outdated builds The retirement policy is to retire all frameworks without activity for more than a year (which isn't automated and thus it happens that outdated frameworks are included). Here's the result of npm view ${package} time: reaml was published 2020-08-01 reason-react is unmaintained and should be updated to ReScript/React. I created issue #905 for that. react-focal was published 2021-02-04

A PR for an automated check for outdated frameworks would be very welcome :smile:

[WebReflection]

1. I've used npm run build-prod and it failed at those mentioned frameworks. Maybe your docker image has something else too installed, but in my machine, after removing those projects, npm run build-prod builds successfully all other keyed frames
2. didn't know there was a specific policy but I'd be happy to provide an automation for that ... although anything more complex than an sh script looks unnecessary, as first MR, and I wonder if that would work?

[WebReflection]

@krausest not sure about that sensitive sensor, but if it helps, this is the most basic way to fail at those projects (except the preact one that I've removed as it works indeed):

git clone https://github.com/krausest/js-framework-benchmark.git
cd js-framework-benchmark
npm ci
cd frameworks/keyed/react-focal
npm ci
# this fails already on npm ci
cd ../reaml-react
npm ci
# another failure
cd ../reason-react
npm ci
# a bigger failure trying to execute an .exe file ...
# reason-react/node_modules/bs-platform/lib/ninja.exe -f release.ninja

I have all logs. I don't know if you've pinned the Docker to nodejs 8 though, so maybe these fail in latest npm or something?

[WebReflection]

@krausest when you say "retire all frameworks without activity for more than a year" do you mean upstream/npm? because when I've said "there are maintainers and maintainers" I mean that many here are trying, from time to time, to keep their libraries versions and scores updated, while some library looks there since long time. npm view package doesn't help much in here, or better, doesn't cover what I meant. Maybe I should check via git or something if anything hasn't been updated in more than year in this repository ? that might be more significant, so that from time to time libraries authors can come back and give an update, otherwise old experiments that are not complete might stick around forever .... I hope I've better explained what I meant, and I'm curious to know your thoughts around this. Thanks.

[WebReflection]

Not sure this helps, but these are more than year + "errors"

Errors with npm view ${package} from each folder.

Not necessarily errors though, just what npm view said.

angular-nozone binding.scala blazor-wasm callbag-jsx-list doohtml fidan fntags fullweb-helpers fullweb-template glimmer-2 hullo ls-element marionette-backbone react-mobX react-redux-hooks-immutable react-redux-rematch reaml-preact reaml-react reflex-dom resonatejs scarlets-frame sifrr vanillajs-1 vanillajs-wc wasm-bindgen

---

Outdated (1+ years) / Last modified via npm view ${package} | time grepping the modified value.

2014-10-07 alpine 2018-03-18 angularjs 2020-01-23 choo 2012-05-13 crank 2013-03-21 datum 2014-03-26 dominator 2019-04-29 domvm 2018-02-26 ember 2019-10-11 glasgow 2019-01-04 glimmer 2019-01-04 helix 2017-03-27 isotope 2020-05-04 ivi 2019-11-05 knockout 2019-11-29 maple 2015-10-30 miso 2015-12-13 misojs 2020-06-03 nervjs 2020-03-30 react-easy-state 2019-07-15 react-focal 2019-01-05 react-hooks 2019-01-05 react-recoil 2019-02-10 react-redux-hooks 2020-06-09 reason-react 2020-06-02 redom 2017-09-22 s2 2012-05-03 solid 2016-09-12 solid-state 2014-02-17 stdweb 2020-05-21 vanillajs 2014-03-25 yew

---

The script

This check.sh simply loops over keyed folders and try to view projects "by folder name" which is probably not what we want, but I am not sure how to retrieve the library to check otherwise.

#!/usr/bin/env sh

loop() {
  local root="$1"
  local type="$2"
  local path=
  local modified=
  echo -e "\x1b[1m${type}\x1b[0m";
  for path in $(ls "frameworks/${type}"); do
    if [ -f "${root}/frameworks/${type}/${path}/package.json" ]; then
      modified=$(npm view "${path}" time | grep modified | sed "s/^.*: '//; s/T.*//")
      echo "  ${modified} ${path}"
    fi
  done
}

loop "$(pwd)" "keyed"

[ryansolid]

There is a mismatch in many cases between the npm package name and the implementation name which produces many of the false positives.

[leeoniya]

while some frameworks may be truly abandoned, others simply have few known issues and are in a more or less "done" state.

e.g. domvm is what it is, and i have no desire to continually expand its core functionality. it is unlikely to see new releases published any time soon unless bugs are reported and get fixed.

ivi, surplus, and many others are in the same boat, i think.

[WebReflection]

I’ve mentioned the mismatch, and I wonder how could we check properly things instead (maybe by reading dependencies instead of folder name).

about being on the same boat, I hear you, and hyperHTML and domdiff are already in that boat. uhtml uses udondiff which is also done, and I didn’t bother a MR for that.

Long story short, I agree, but at least one project came up nit maintained, and apparently there is a convention about what should stay and what should go.

[krausest]

There's already a mechanism to determine the version of the framework that's shown in the table and I think this would be a better approach. The package.json can contain a js-framework-benchmark.frameworkVersionFromPackage property. This property is a colon separated list of npm package names whose version is determined from package-lock for the table (it's possible to use frameworkVersion if there's no apt npm package. Those would have to be checked manuallly). Maybe webdriver-ts/updateFrameworks.ts is a good starting point for such a script.

One could check the last publication date of the latest version for those npm packages. And I agree with @leeoniya. There must be a careful check whether the framework is really dead.

[WebReflection]

My thinking is that frameworks are also combined, see react-*, and I think it’s a bit tricky to get all combo right. versionFromPackage is what I was thinking too, but what if it’s reasonably modern React, but the combo itself was experimental or abandoned? Using React just as reference, but if I can clear this bit out, I might land a PR or try again via SH to see if it works/help.

[krausest]

Found a little time and created a script. Currently I get the following results (will be merged after cleaning up a bit)

node checkObsolete.js
Last npm update for keyed/aurelia aurelia-framework is older than a year: aurelia-framework:2019-01-19
Last npm update for keyed/choo choo is older than a year: choo:2020-01-23
Last npm update for keyed/crui @crui/core is older than a year: @crui/core:2019-07-28
Last npm update for keyed/domdiff domdiff is older than a year: domdiff:2020-03-15
Last npm update for keyed/glasgow glasgow is older than a year: glasgow:2019-10-11
Last npm update for keyed/isotope @isotope/core is older than a year: @isotope/core:2020-05-26
Last npm update for keyed/ivi ivi is older than a year: ivi:2020-05-04
Last npm update for keyed/knockout knockout is older than a year: knockout:2019-11-05
Last npm update for keyed/nervjs nervjs is older than a year: nervjs:2020-06-03
Last npm update for keyed/redom redom is older than a year: redom:2020-06-02
Last npm update for non-keyed/aurelia aurelia-framework is older than a year: aurelia-framework:2019-01-19
Last npm update for non-keyed/domdiff domdiff is older than a year: domdiff:2020-03-15
Last npm update for non-keyed/moon moon is older than a year: moon:2020-02-08
Last npm update for non-keyed/redom redom is older than a year: redom:2020-06-02
Last npm update for non-keyed/simulacra simulacra is older than a year: simulacra:2019-10-02
Last npm update for non-keyed/vanilla-dom-framework @vanilla-ftw/vanilla-dom is older than a year: @vanilla-ftw/vanilla-dom:2018-09-12
Last npm update for non-keyed/vuera vuera is older than a year: vuera:2020-01-27

The following frameworks must be checked manually
keyed/binding.scala has no frameworkVersionFromPackage
keyed/blazor-wasm has no frameworkVersionFromPackage
keyed/dominator has no frameworkVersionFromPackage
keyed/domvm has no frameworkVersionFromPackage
keyed/doohtml has no frameworkVersionFromPackage
keyed/helix has no frameworkVersionFromPackage
keyed/hullo has no frameworkVersionFromPackage
keyed/maple has no frameworkVersionFromPackage
keyed/miso has no frameworkVersionFromPackage
keyed/misojs has no frameworkVersionFromPackage
keyed/oldskull has no frameworkVersionFromPackage
keyed/reagent has no frameworkVersionFromPackage
keyed/reflex-dom has no frameworkVersionFromPackage
keyed/resonatejs has no frameworkVersionFromPackage
keyed/s2 has no frameworkVersionFromPackage
keyed/stdweb has no frameworkVersionFromPackage
keyed/vanillajs has no frameworkVersionFromPackage
keyed/vanillajs-1 has no frameworkVersionFromPackage
keyed/vanillajs-wc has no frameworkVersionFromPackage
keyed/wasm-bindgen has no frameworkVersionFromPackage
keyed/yew has no frameworkVersionFromPackage
non-keyed/delorean has no frameworkVersionFromPackage
non-keyed/domvm has no frameworkVersionFromPackage
non-keyed/doohtml has no frameworkVersionFromPackage
non-keyed/halogen has no frameworkVersionFromPackage
non-keyed/hullo has no frameworkVersionFromPackage
non-keyed/incr_dom has no frameworkVersionFromPackage
non-keyed/miso has no frameworkVersionFromPackage
non-keyed/polymer has no frameworkVersionFromPackage
non-keyed/seed has no frameworkVersionFromPackage
non-keyed/simi has no frameworkVersionFromPackage
non-keyed/slim-js has no frameworkVersionFromPackage
non-keyed/stdweb has no frameworkVersionFromPackage
non-keyed/stem has no frameworkVersionFromPackage
non-keyed/vanillajs has no frameworkVersionFromPackage
non-keyed/vanillajs-1 has no frameworkVersionFromPackage

[krausest]

Here's the commit: https://github.com/krausest/js-framework-benchmark/commit/8f1e6b89ccfb4fc7b69e6e4f70037e951931780b

[WebReflection]

It’d be great to flag known obsolete as such, and give authors a window to either update the version, express their interest in keeping the benchmark in, or be fine with removal. I’d be fine with domdiff removal, as example, as I’ve no interest in keeping it around this space (there’s another repository specifically for pure DOM differs, so these are imho irrelevant in this benchmark).

[krausest]

Looking at github activity the following frameworks look dead:

• glasgow
• simulacra
• nerv
• vuera

I'm not sure about aurelia, moon, vanilla-js-dom. I'd keep them but I haven't found much activity. The other look (at least a bit) alive.

Will be tracked in #906