search

openwebwork / webwork2

Course management front end for WeBWorK
http://webwork.maa.org/wiki/Main_Page
Other
144 stars 165 forks source link

Upgrade javascript libraries #1015

Closed taniwallach closed 3 years ago

taniwallach commented 5 years ago

We should consider upgrading some of the other javascript libraries that are utilized also. For example, the versions of jQuery (1.7.2) and jQuery UI (1.9.0) that WeBWorK uses are quite old (released in 2012).

Originally posted by @drgrice1 in https://github.com/openwebwork/webwork2/issues/1013#issuecomment-532908420

taniwallach commented 5 years ago

Right now WW uses the deprecated bower to help handle JavaScript dependencies and quite a few outdated JavsScript libraries.

Below are some mails about discussions to replace it...

We should consider whether the JavaScript dependencies can be kept out of the main Git repo and pulled in later as "sub-modules" or in some other manner during installation.

On Sun, Aug 26, 2018 at 4:12 PM Nathan Wallach taniwallach@gmail.com wrote:

Dear Mike, Peter, and Jason,

The webwork2/htdocs/js/README_js_organization file refers to the use of bower to manage JavaScript libraries, so there should be a bower.json file somewhere... but the webwork2 git repository does not seem to have one.

I'm trying something which will require updating jQuery to a newer version (at least 1.9.1) and installing the jstree plugin which is available for bower (https://github.com/vakata/jstree.git

Can one of you send me a relatively recent bower.json file?

Thanks, Nathan http://webwork.maa.org/pipermail/webwork-devel/2013-June/001069.html http://webwork.maa.org/pipermail/webwork-devel/2013-June/001070.html

On Aug 26, 2018, at 11:00 AM, Nathan Wallach tani@mathnet.technion.ac.il wrote:

This is starting to look even more daunting to me... 

  1. jsquery had major upgrades to 1.9 and 3.0 which apparently required work to handle the changes to the framework, 
  2. there seem to be several versions of jQuery scattered in the htdocs/js tree and custom versions of jquery-ui
  3. bower has been deprecated in favor of Yarn:   https://yarnpkg.com/en/docs/getting-started Anyone have a suggestion on how to work on migrating WW to a recent jQuery, and who is likely to be a helpful partner in trying to work on that?

jQuery update notes:

jQueryUI update notes:

Tani

From: "mgage@ur.rochester.edu" gage@math.rochester.edu To: Nathan Wallach tani@mathnet.technion.ac.il Cc: Michael Gage gage@math.rochester.edu, Michael Gage michael.gage@rochester.edu, Peter Staab peter.staab@gmail.com, Jason Aubrey aubreyja@gmail.com Subject: Re: WeBWorK bower.json file - missing in Git

Hi Tani, I think that bower was only used in ww3 code — which is still under development.

I’m willing to help with migration of webwork develop to a modern jquery — at least I can provide consultation on what has been done historically in webwork.  I’m not any kind of expert on jquery however.

Peter — would you have time to work on this with us?  You know more javaScript than the rest of us.

Take care,

Mike

Date: Mon, 3 Sep 2018 21:35:45 -0400 From: Peter Staab peter.staab@gmail.com To: Michael Gage gage@math.rochester.edu Cc: Nathan Wallach tani@mathnet.technion.ac.il, Michael Gage michael.gage@rochester.edu, Jason Aubrey aubreyja@gmail.com Subject: Re: WeBWorK bower.json file - missing in Git

a couple of things: 1) all of the JS libraries for ww3 is is in the webwork3/ directory, so anything in htdocs doesn’t affect it.  Upgrading jQuery is a good move.  2) It appears that bower is deprecating itself.  I just played with yarn as a library manager this summer and it seems fine.  

I could help if some ww2 issues arise when migrating to a newer jQuery.  I don’t think there is a lot in ww2 that uses it, so it shouldn’t be too difficult if something comes up. 

Peter

pstaabp commented 5 years ago

Weighing in here.

  1. I haven't done anything on the webwork2 libraries using javascript in years. There are a few of the pages that use javascript (I'm thinking the setDetail and libraryBrowser pages). I know that there's a lot to clean up there and I'm happy to help, when I have time.

  2. A number of years ago, I was using bower to manage libraries as as @taniwallach pointed out above, that it is deprecated in favor of yarn.

  3. I've been working primarily on the new webwork3 (ww3) code which is heavily dependent on javascript, however everything I'm writing is in typescript right now and get's translated to javascript. All of that is in webwork3/node_modules and as I mention it, I'm using yarn to manage.

  4. When webwork3 gets deployed, all of the javascript libraries will be minimized and compressed into a fairly small package and as I plan, there will be a single "web app" that runs all of webwork. This is definitely the way things are going these days and is much easier to handle but is much different model of coding for the web. I've updated some of the development pages and will try to get that going too.

A couple of things I'd recommend. 1) in many ways, if it ain't broke, don't fix it and unless there are security holes, we can run some old libraries. 2) in the longer term, a good clean-up would be helpful and try to upgrade packages that need it;

taniwallach commented 5 years ago

@pstaabp wrote:

A couple of things I'd recommend. 1) in many ways, if it ain't broke, don't fix it and unless there are security holes, we can run some old libraries. 2) in the longer term, a good clean-up would be helpful and try to upgrade packages that need it;

In general, I agree, but I don't think anyone has tried to check whether there are any security implications of continuing to use the old libraries, not am I sure how qualified most of us are to determine that without an inordinate amount of effort.

If the "plan" it to try to move to WW 3 in the next 2 years or so, then I would tend to think that minimal effort should be made in major refactoring work like this on WW 2.

On the other hand, if WW 2 is going to continue to be the production platform for several more academic years, then I suspect that it would be wise to consider both the security aspect and the issue of maintainability of the current collection of old libraries.

taniwallach commented 5 years ago
  1. When webwork3 gets deployed, ... and as I plan, there will be a single "web app" that runs all of webwork. This is definitely the way things are going these days and is much easier to handle but is much different model of coding for the web. I've updated some of the development pages and will try to get that going too.

I think we may want to consider whether some aspects (PDF generation, large DB tasks) can be moved into independent micro-services, and the primary web app would call these back-end services. Most likely problem rendering could also be handled by an independent micro-service. If such if feasible, then horizontal scaling to support very large scale WW "servers" my be quite feasible.

taniwallach commented 3 years ago

The old jQuery version in use by WW triggers warnings from security tools. See for example https://webwork.maa.org/moodle/mod/forum/discuss.php?d=4872

It would seem worthwhile to see what can be done about upgrading jQuery, maybe using a version from a CDN instead of providing it locally in htdocs.

It might also be a good idea to try to replace jQuery-ui which is not being updated/maintained (although there is activity on GitHub https://github.com/jquery/jquery-ui/tree/master there are no releases since late 2016). Some quick searches turned up the following:

For example, Drupal is removing jQuery-ui and there is a discussion of what is being used to replace sortable at

See:

taniwallach commented 3 years ago

I also noticed that WW used a very old version of backbone which includes jQuery 1.7.1 at htdocs/js/legacy/backbone/test/vendor/jquery-1.7.1.js.

Backbone is mentioned at https://webwork.maa.org/wiki/Roadmap_Details .

taniwallach commented 3 years ago

@drgrice1 posted a branch which updates to jQuery 3.4.1 at https://github.com/drgrice1/webwork2/tree/jquery-update .

That is an early version, and all the changes are in one commit. A future PR will hopefully use jQuery 3.5.1 and separate library changes and webwork file changes into different commits... so reviewing the changes to WW code will be feasible.

In any case - it is ready for people to start checking what gets broken.

My testing setup:

First issue + fix: I discovered that in the set editor problems could not be reordered with the drag-and-drop method.

Uncaught TypeError: $.ui.isOverAxis is not a function
    jQuery 10
jquery.nestedSortable.js:315:32

Replacing htdocs/js/vendor/jquery/modules/jquery.nestedSortable.js  with the contents of https://raw.githubusercontent.com/ilikenwf/nestedSortable/master/jquery.mjs.nestedSortable.js  from https://github.com/ilikenwf/nestedSortable seems to fix this. As a quick fix I just overwrote the old file using the old name with the new contents.

I'm submitting a PR to @drgrice1's branch with those changes: https://github.com/drgrice1/webwork2/pull/2

pstaabp commented 3 years ago

Starting on @drgrice1' branch sounds like a sound plan. Much of what I chimed in on the email thread was based on memory, which is not a good thing these days. :) I will carve out some time to look into this over the next few days.

taniwallach commented 3 years ago

Some of the discussion was in a series of emails and the key parts are being posted here for the project records/history.

Key points from 2 mails I sent:

@pstaabp wrote:

Unfortunately, there is very little needed javascript in the webwork2 branch, but just enough that we should do something about it. It came about 10 years ago or so when some ideas from what is trying to become webwork3 what imported into webwork2, mainly the library browser.

A few thoughts from my point of view

Unfortunately, I don't see webwork3 on the horizon anytime soon. When I'm teaching, I have very little time to do any development work, but would love if someone would step in with a hand. However, this renderer project that is moving along I think we help split webwork2 and PG and make webwork3 a viable standalone alternative to webwork2, which will make it easier to develop.

I can probably carve out some time in the next month or so to take a look at this issue and see what needs to be done.

@drgrice1 wrote:

I don't think that getting rid of jQuery is a good idea. Note that jQuery does much more than XMLHttpRequests. That is about all that Axios does. I don't think we even use jQuery for that purpose much at all. The library browser may be the only place that uses jQuery in that way. So replacing jQuery with Axios does not make a lot of sense.

We should consider replacing jQuery-UI with something that is more modern, and that work better for mobile (particularly touch) devices. One possibility for that is jQuery Mobile. There are lots of other javascript user interface libraries to work with though.

There is a branch in my fork of webwork that upgrades jQuery to version 3.4.1, and jQuery-UI to version 1.12.1. I think we should start there. I have at least identified much of what needs to be done with jQuery related stuff. See https://github.com/drgrice1/webwork2/tree/jquery-update. I am not in favor of using a CDN for javascript. CDN reliability is not as good as you think. I would prefer that we use either npm or Yarn to manage javascript dependencies. Another option (and probably the way to go) would be to make it configurable as to whether javascript is used from a CDN or served locally.

taniwallach commented 3 years ago

Some comments in https://github.com/openwebwork/webwork2/issues/1013 which belong in this thread:

@drgrice1 (https://github.com/openwebwork/webwork2/issues/1013#issuecomment-707322996) wrote:

I also don't think that we should keep static versions of javaScript libraries in the git repository. These should be managed via a JavaScript package manager like npm or Yarn. If set up correctly, this would add a minor step to the installation of webwork.

@dlglin (https://github.com/openwebwork/webwork2/issues/1013#issuecomment-707848739) wrote:

I agree with @drgrice1 that we should not keep static versions of external libraries in the WW git repositories. If WW requires an external package such as MathJax, then it should be listed as a dependency and not rolled into the WW code (which is the case for MathJax 2 in the current release of WW). As for local install vs. CDN, my vote is that this be made a configuration option. I think forcing people to use a CDN is unadvisable for the reasons that Glenn mentioned, as well as the fact that it introduces another point of failure for the system. The WW installation instructions/scripts will have to cover at least one option of how to install these JS libraries.

taniwallach commented 3 years ago

I think the consensus from the various discussion threads is that use of CDN loading should be an option, which also means that work to enable it is less urgent.

However, it we set up configuration variables for the location from which the main JavaScript libraries are loaded, and use those variables wherever necessary, it should make the option of using a CDN pretty simple to turn on/off. We could include sample CDN lines in similar to what conf/localOverrides.conf.dist has for $webworkURLs{MathJax}.

Although starting with @drgrice1's branch can give us a quick test-bed for an update for jQuery and jQuery-UI, it seems that we would want to make the actual changes in a somewhat different manner.

Here is a proposal for a plan of action on the JavaScript library reorganization. building on ideas from the discussion threads:

  1. Consider creating a feature branch off develop to use for the work on the JavaScript library changes. That will allow pull requests and their discussions to be tracked in the main repository, but minimize interfering with other work on the develop branch.
  2. JavaScript libraries which can easily be installed using a JavaScript package manager will be installed in that manner, and the necessary configuration files for the package manager will be managed in the webwork2 repository - for now in the develop (or special feature) branch.
  3. Create a script to run which does those installations.
    • Install such libraries under a subdirectory for "managed" libraries.
    • The same script could hopefully be used both for regular and Docker installs (for Docker being run at an appropriate stage of the Dockerfile) unless there are technical reasons to maintain parallel scripts.
  4. Legacy libraries and other things which cannot be handled by a JavaScript package manager will be installed by a second script, pulling in the necessary files either from a dependable upstream location (if still available) or from one or more special Git repositories under https://github.com/openwebwork intended only to provide those legacy files.
    • Try to keep those files under a different subdirectory.
  5. The old static libraries under https://github.com/openwebwork/webwork2 will be removed from relevant branch as the new installation methods are able to provide them.
  6. .gitignore will be edited as progress is made.
  7. Try to have all relevant main libraries loaded from locations set in the main control files and conf/localOverrides.conf so that moving between serving files locally and redirecting to a CDN is easy.
  8. As feasible or necessary update/replace libraries and modify how they are installed so that as much as possible is handled by the selected package manager.
  9. Consider a plan of what old libraries should be replaced and the priority for such changes, and add items about these tasks to the project boards.
  10. Attempt to add comments to the WeBWorK code to indicate where there are JavaScript dependencies, and to indicate what they are. Hopefully, over time that will help maintain things, and provide an approach to determining what needs to be tested when libraries make significant changes to their behavior.

Note: One worry I have is how easy it will be to switch Git branches while we are working on the transition, when there may be directories/files of "external" JavaScript which are not managed by Git. Hopefully, by suitable decisions of where to locate things, we can make it feasible to switch back and forth between branches with the old and new JavaScript library setup without any issues.

pstaabp commented 3 years ago

@drgrice1 What is the status of the transition? I just tried your branch and I couldn't tell that the library were updated. It may be that I'm not successfully building everything correctly with docker--it has been my bane (development-wise) for a year or so.

drgrice1 commented 3 years ago

@pstaabp: The jquery update branch is working rather well. It would be good to have others test it though. That is the only reason that I have it listed as "Do not merge yet".

drgrice1 commented 3 years ago

@pstaabp: What do you mean that you couldn't tell that the library were updated?

pstaabp commented 3 years ago

Looking at all of the sources being pulled in on a given webpage had jQuery 1.7.2. I think it's getting my development machine working better. I tried it with Docker and I know there's hoops to jump through to get changes up and running. I'll get back to you on this and try to test.

taniwallach commented 3 years ago

I'm closing this. We have jQuery updated, also MathJax. We need to open a new issue for future updates.