Closed lentinj closed 8 months ago
Most of the dependencies in README.markdown aren't necessary for running OneZoom, even more so now we have the tree-build repo. I've never installed most of the Python & Perl dependencies. @hyanwong could we tidy this up?
I am going to try switching over to using the tree-build repo this weekend. I will try to tidy up the dependency list then.
I am going to try switching over to using the tree-build repo this weekend. I will try to tidy up the dependency list then.
Note that in the tree-build repo, the dependencies are resolved automatically via setup.cfg (see https://github.com/OneZoom/tree-build/blob/main/setup.cfg#L12-L19). So there is no manual step. Also, it's all Python at this point, with no Perl.
I'm guessing the remaining Perl scripts are ancient history nowadays, and if we don't delete them then at least their dependencies don't need to be this prominent.
Note that pymysql
or mysql-connector-python
aren't dependencies of the site, as web2py bundles it's own copy. All the listed python dependencies are for things in OZprivate/ServerScripts/Utilities
AFAICS. Maybe there should be a requirements.txt
in that directory, and a README.md
explaining how to install them? (A package like @davidebbo has done would be better, but I'm guessing that's too much for code that is probably very infrequently used).
The new tree-build code supersedes subtree_extract.pl
and tree_and_meta_parser.pl
(in OZprivate/ServerScripts/TreeBuild
). There are various other Perl scripts scattered around that I'm not familiar with and can't comment on.
@hyanwong According to README.markdown we assume Python 3.7, and the Gruntfile has python3.7
hard-coded in it. I'm guessing this assumption needs to move with the times a bit, 3.7 is about to fall off the security updates radar. Any preferences what to? Are the server(s) still on 3.7?
Good point. Let's move to something more modern. I'll ping you on slack. 3.8 is possible I think.
FWIW, I've been using 3.10 since I got started on OZ.
I guess we should just go with 3.10, to avoid too many further updates? I will try this now.
I've been on 3.10 without issues, so that's probably a safe bet. That being said, I see the latest stable is now 3.12, and the 3.11 release notes claim some notable perf improvements, so it may be worth exploring.
And as a side note, it would be nice to move OZtree to a virtual env, like we have for tree-build. It avoids having to install systemwide packages (which can cause conflicts, ...).
Yes, both true. I agree about venvs, and perhaps we should just bite the bullet and go for 3.12?
perhaps we should just bite the bullet and go for 3.12?
Debian stable (which isn't that old atm) is only at 3.11. Whilst we don't actively use Debian, I'd be wary of assuming newer than stable without some justification.
venvs
A venv for OZtree will need to happen soon, when switching operating systems/python versions is the obvious time.
Ah, and I see FreeBSD 13.2 (which is what we use on the server) only has 3.8 by default, although I guess I can install this: https://www.freshports.org/lang/python311?
I don't know much BSD, but presumably so.
These are the npm messages I get when I try a clean install on my laptop (OS X). I assume most of these can be ignored (anyway, the node modules are only used to compile static JS code, so are presumably not security risks)
(py311) yan@Yans-New-Air OZtree % npm --v
10.2.5
(py311) yan@Yans-New-Air OZtree % npm install
npm WARN deprecated cryptiles@3.1.4: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated sntp@2.1.0: This module moved to @hapi/sntp. Please make sure to switch over as this distribution is no longer supported and may contain bugs and critical security issues.
npm WARN deprecated har-validator@5.0.3: this library is no longer supported
npm WARN deprecated boom@4.3.1: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated boom@5.2.0: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated hoek@4.2.1: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated w3c-hr-time@1.0.2: Use your platform's native performance.now() and performance.timeOrigin.
npm WARN deprecated chokidar@2.1.8: Chokidar 2 does not receive security updates since 2019. Upgrade to chokidar 3 with 15x fewer dependencies
npm WARN deprecated hawk@6.0.2: This module moved to @hapi/hawk. Please make sure to switch over as this distribution is no longer supported and may contain bugs and critical security issues.
npm WARN deprecated request@2.83.0: request has been deprecated, see https://github.com/request/request/issues/3142
added 1215 packages, and audited 1426 packages in 43s
102 packages are looking for funding
run `npm fund` for details
34 vulnerabilities (9 moderate, 23 high, 2 critical)
To address issues that do not require attention, run:
npm audit fix
To address all issues possible (including breaking changes), run:
npm audit fix --force
To get grunt to work, I also needed to:
export NODE_OPTIONS=--openssl-legacy-provider
(see https://stackoverflow.com/questions/69692842/error-message-error0308010cdigital-envelope-routinesunsupported; npm audit fix --force
did not work for me
(anyway, the node modules are only used to compile static JS code, so are presumably not security risks)
Yes, we vendor very little, if any, Javascript into the client. And all the server-side javascript we do run is just for builds
export NODE_OPTIONS=--openssl-legacy-provider
Upgrading will be the way to solve this though. Again, not a vast security risk, but eventually this cheating option will disappear as they get bored of maintaining old OpenSSL builds.
export NODE_OPTIONS=--openssl-legacy-provider
Upgrading will be the way to solve this though. Again, not a vast security risk, but eventually this cheating option will disappear as they get bored of maintaining old OpenSSL builds.
I'm not sure what to upgrade here? npm
is on 10.2.5.
I'm not sure what to upgrade here?
One of our NPM dependencies will be requiring the option to work, and will need bumping / removing.
Tidy up package.json, so that npm ci can be used instead
In #687 I have upgraded to webpack 5.0.0, and it seems to compile the site just fine: I haven't checked npm ci
though.
I didn't assume python 3.7 in the above, installed python 3.9 and Grunt gets upset.
Also in #687 I installed it all using a conda install of python 3.11, and it seems fine, so I have changed the hardcoded python version in the Gruntfile to 3.11, on the assumption that we'll be able to install that new version on the new server soon.
I haven't checked npm ci though.
npm ci
is a simpler version of npm install
. npm install
will update transitive dependencies in package-lock.json
if it wants to. npm ci
OTOH refuses to do so, it just installs what's in package-lock.json
.
So once npm install
works without making a change to package-lock.json
, npm ci
should similarly be happy.
So with #687 I get
(py311) yan@Yans-New-Air OZtree % npm ci
npm WARN deprecated cryptiles@3.1.4: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated sntp@2.1.0: This module moved to @hapi/sntp. Please make sure to switch over as this distribution is no longer supported and may contain bugs and critical security issues.
npm WARN deprecated har-validator@5.0.3: this library is no longer supported
npm WARN deprecated boom@4.3.1: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated boom@5.2.0: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated hoek@4.2.1: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated uuid@3.4.0: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
npm WARN deprecated hawk@6.0.2: This module moved to @hapi/hawk. Please make sure to switch over as this distribution is no longer supported and may contain bugs and critical security issues.
npm WARN deprecated request@2.83.0: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated @babel/polyfill@7.12.1: 🚨 This package has been deprecated in favor of separate inclusion of a polyfill and regenerator-runtime (when needed). See the @babel/polyfill docs (https://babeljs.io/docs/en/babel-polyfill) for more information.
npm WARN deprecated grunt-curl@2.5.1: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.
npm WARN deprecated core-js@2.6.12: core-js@<3.23.3 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Some versions have web compatibility issues. Please, upgrade your dependencies to the actual version of core-js.
added 984 packages, and audited 1220 packages in 8s
145 packages are looking for funding
run `npm fund` for details
7 vulnerabilities (4 moderate, 2 high, 1 critical)
I guess that's fine @lentinj : would you recommend that I try to update anything else (e.g. the ones that are mentioned there), or would this break stuff?
I've been doing some spring cleaning, and as part of it rebuilding my aged OneZoom container. My notes on doing so are:
package.json
, so thatnpm ci
can be used instead (solved in 0d5a0680eb5d6e75486b2442ebd759d467847ebf)Obviously feeding this into README.markdown would be a good choice, even if it is a bit Debian-centric:
./node_modules/.bin/grunt
, so no need to suggest installing your own../private/appconfig.ini.example
, don't copy-paste a separate copy