search

ComPlat / chemotion_ELN

Electronic Lab Notebook
https://www.chemotion.net
GNU Affero General Public License v3.0
126 stars 54 forks source link

Upgrade codebase to latest Rails 6.1 and at least Ruby 2.7 #962

Closed jhmegorei closed 5 months ago

jhmegorei commented 2 years ago

As an ELN-Administrator I want the ELN code to be based on a supported language and framework version that receives security patches.

Rails 5 support expired June 1st 2022, while Ruby 2.6.6 support already expired April 1st 2022. Rails should be updated to at least 6.1, as 6.0 only receives security updates until June 1st 2023. Ruby should preferably updated to at least 3.0, as 2.7 ends its lifecycle on April 1st 2023. Correspondingly, the underlying docker images might need to be updated as well.

Additionally, the ruby gems and javascript packages need to be updated, as there are several packages that have not received updates for a very long time (up to 8 years).

Sources: https://endoflife.date/rails, https://endoflife.date/ruby

Decision: As discussed in our planning meeting, this needs to be done Timeframe: as soon as possible

jhmegorei commented 2 years ago

Research on gems with pinned versions or github links:

jhmegorei commented 2 years ago

Development Gems

As far as I know, the main ELN installation processes run via Docker. Still there are a lot of capistrano gems in the Gemfile.

Can they be removed or are some people still deploying manually with Capistrano?

Answer provided by Paggy in our regular Meeting Capistrano is still used on several servers, so the gems have to stay there and be updated accordingly

jhmegorei commented 2 years ago

Javascript Packages

The frontend uses a lot of javascript packages in very old versions. We need to check which packages are no longer in use and can be safely removed and which ones need to be updated

WIP

jhmegorei commented 2 years ago

Test Gems

jhmegorei commented 2 years ago

We upgraded the first batch of Gems (https://github.com/ComPlat/chemotion_ELN/pull/970, first 3 commits)

Some changes were required due to the upgrade of Grape.

Easy fix: within the ProfileAPI the base class of a Validator needed to be updated (https://github.com/ComPlat/chemotion_ELN/pull/970/commits/2179a4611ce9efeb162e85785cd1c26e5832b97f)

Hard fix: The Report API used a pretty convoluted definition of input parameters, including a brainmelting definition of columns (https://github.com/ComPlat/chemotion_ELN/pull/970/commits/c47ff414ef33b49d57a1d728c529798a2df03570#diff-b93f92964b6eefb5bc9857b164d156bc46a39739f9fca19e1f2fc1eb2d1d96f6L8) and doubly json encoded input parameters (https://github.com/ComPlat/chemotion_ELN/pull/970/commits/c47ff414ef33b49d57a1d728c529798a2df03570#diff-07bfdbc84d53bc264e9130cd933d44b6c1d8e8057e2099baf1cf0ac238d5bc2fL211 in combination with https://github.com/ComPlat/chemotion_ELN/pull/970/commits/c47ff414ef33b49d57a1d728c529798a2df03570#diff-c12b65d17272e83c800b64e8dce659793169c5272730232daa3504efaa981c7dL35). This no longer worked in the new Grape version, so it had to be fixed.

We were able to refactor the unnecessarily complicated code paths by adhering to the Grape documentation and removing the double json encoding. Took a while to understand the whole issue though...

jhmegorei commented 2 years ago

Next batch of gem upgrades: https://github.com/ComPlat/chemotion_ELN/pull/996

Most changes were straightforward, but an Encoding error while saving reports took a while to investigate. The Solution was that rubyzip changed its internal process by writing the data in binary format, which causes Tempfile#write to raise an error if there are mismatches between the ASCI-8Bit encoding of rubyzip and the UTF-8 encoding Tempfile#write was expecting. The fix was easy though, just set the IOstream that is written into binmode (https://github.com/ComPlat/chemotion_ELN/pull/996/files#diff-3aa32a5ac6f922542dbf8429620d4a0c3936a99b3c66d61c54d3123920e92119R41)

Gem Changes

axlsx: Replaced by caxlsx as the original gem was no longer maintained faraday_middleware: replaced by faraday-follow_redirects, which was the only middleware used from the previous gem swot: removed as it was no longer maintained and already unused by ELN uglifier: removed as it was no longer in use rubyXL: removed as it was only used in a CLI helper class (which is removed as well)

Gem Updates

actioncable 5.2.7.1 -> 5.2.8.1 actionmailer 5.2.7.1 -> 5.2.8.1 actionpack 5.2.7.1 -> 5.2.8.1 actionview 5.2.7.1 -> 5.2.8.1 activejob 5.2.7.1 -> 5.2.8.1 activemodel 5.2.7.1 -> 5.2.8.1 activerecord 5.2.7.1 -> 5.2.8.1 activestorage 5.2.7.1 -> 5.2.8.1 activesupport 5.2.7.1 -> 5.2.8.1 barby 0.6.6 -> 0.6.8 bcrypt 3.1.15 -> 3.1.18 bindata 2.4.10 -> 2.4.13 bootsnap 1.7.3 -> 1.13.0 byebug 11.0.1 -> 11.1.3 capistrano 3.9.1 -> 3.17.1 capistrano-bundler 1.5.0 -> 2.1.0 capistrano-rails 1.4.0 -> 1.6.2 closure_tree 7.0.0 -> 7.4.0 countries 3.0.0 -> 4.2.3 database_cleaner 1.7.0 -> 2.0.1 devise 4.7.1 -> 4.8.1 erubis: removed factory_bot 4.11.1 -> 6.2.1 factory_bot_rails 4.11.1 -> 6.2.0 faker 1.6.6 -> 2.22.0 faraday 0.12.2 -> 2.6.0 graphql 2.0.14 -> 2.0.15 haml 5.1.2 -> 6.0.7 haml_rails 1.0.0 -> 2.1.0 html2haml: removed i18n 0.9.5 -> 1.12.0 i18n_data 0.8.0 -> 1.12.0 jquery-rails 4.4.0 -> 4.5.0 json-jwt 1.13.0 -> 1.16.0 launchy 2.4.3 -> 2.5.0 msgpack 1.3.3 -> 1.6.0 multipart-post: removed naughty_or_nice: removed net-sftp 2.1.2 -> 3.0.0 oauth 1.4.7 -> 2.0.9 omniauth 1.9.1 -> 1.9.2 omniauth-oauth2 1.7.2 -> 1.7.3 openid_connect 1.3.0 -> 1.4.2 pandoc-ruby 2.0.2 -> 2.1.6 pg 0.20.0 -> 1.4.4 pg_search 2.1.7 -> 2.3.6 pundit 2.0.1 -> 2.2.0 rack 2.0.9.1 -> 2.2.4 rack-oauth2 1.12.0 -> 1.21.3 rails 5.2.7.1 -> 5.2.8.1 railties 5.2.7.1 -> 5.2.8.1 responders 2.4.1 -> 3.0.1 rmagick 4.1.2 -> 5.0.0 roo 2.8.2 -> 2.8.3 rspec 23.8.0 -> 3.11.0 rspec-core 3.8.2 -> 3.11.0 rspec-expectations 3.8.6 -> 3.11.1 rspec-mocks 3.8.2 -> 3.11.1 rspec-rails 3.8.2 -> 5.1.2 rspec-support 3.8.3 -> 3.11.1 rubocop 1.36.0 -> 1.37.1 rubocop-ast 1.21.0 -> 1.23.0 rubocop-rails 2.16.1 -> 2.17.0 rubocop-rspec 2.13.2 -> 2.14.1 ruby-mailchecker 3.2.29 -> 5.0.3 rubyzip 1.3.0 -> 2.3.2 spring 2.0.2 -> 3.1.1 tilt 2.0.10 -> 2.0.11 unicode_utils: removed validate_url 1.0.13 -> 1.0.15 web-console 2.3.0 -> 3.7.0 with_advisory_lock 4.0.0 -> 4.6.0

Changes required after gem upgrades

invocation of Faraday changed
    GateAPI
    LiteratureAPI
    GateTransferJob
    DataCite::LiteraturePaser

Changed include for PgSearch
    Element
    Reaction
    Sample
    Screen
    Wellplate

Config Change after Rails Upgrade
    config/application.rb: Allow some classes to be serialized for DB (seehttps://discuss.rubyonrails.org/t/https://github.com/advisories/GHSA-3hhc-qp5v-9p2j-possible-rce-escalation-bug-with-serialized-columns-in-active-record/81017)

Removed Code
    User#academic_email (was no longer used for validation and was the only place that used the swot gem)
    Chemotion::XlsxToJson: whole module was deleted as it was rarely used anyway and was the only place where the rubyXL gem was used
jhmegorei commented 2 years ago

Showstopper: Ketcher-rails depends on Rails 5.2, so it will have to be updated to work with newer rails versions https://github.com/ComPlat/ketcher-rails/blob/master/ketcherails.gemspec#L20

jhmegorei commented 2 years ago

We started a draft PR with the upgrade to Rails 6: https://github.com/ComPlat/chemotion_ELN/pull/1000

The following changes had to be made:

jhmegorei commented 2 years ago

We managed to update pretty much everything we could (except Ketcher-rails which is currently being upgraded by Mehreen as we are told).

➤ docker exec -it rails6-app-1 /bin/bash
chemotion-dev@fdefa45861ff:~/app$ bundle outdated --strict --groups
Fetching https://github.com/complat/chem_scanner.git
Fetching https://github.com/fl9/fun_sftp.git
Fetching https://github.com/ComPlat/inchi-gem.git
Fetching https://github.com/datacite/omniauth-orcid
Fetching https://github.com/ComPlat/openbabel-gem.git
Fetching https://github.com/CamAnNguyen/rdkit_chem
Fetching https://git.scc.kit.edu/ComPlat/rinchi-gem.git
Fetching https://github.com/ComPlat/sablon
Fetching https://github.com/toretore/semacode.git
NOTE: Gem::Specification#has_rdoc= is deprecated with no replacement. It will be removed on or after 2018-12-01.
Gem::Specification#has_rdoc= called from /home/chemotion-dev/.asdf/installs/ruby/2.7.6/lib/ruby/gems/2.7.0/bundler/gems/semacode-4e31e0b79a1a/semacode.gemspec:35.
Fetching https://github.com/merlin-p/thumbnailer.git
Fetching https://github.com/rails/webpacker
Fetching gem metadata from https://rubygems.org/.......
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies.......................

Outdated gems included in the bundle:
===== Group "default" =====
  * inchi-gem (newest 1.06.1 9a95669, installed 1.06.1 91b2684, requested = 1.06.1)
  * rdkit_chem (newest 2020.04.30.1 4b6c183, installed 2020.04.30.1 cf8a389)
  * webpacker (newest 6.0.0.rc.6 dc14827, installed 6.0.0.beta.7 032c2d1)

inchi and rdkit are Complat custom compiled gems, which we will check if they can be upgraded. We will try webpacker as well, but the whole frontend code is another matter, as the packages there might be heavily outdated as well and not play nice with upgrading (not yet tested though).

Also we upgraded all infrastructure files we could find within this Git Repo to include the currently used versions of ruby, node and bundler

PiTrem commented 5 months ago

🙏 resolved with https://github.com/ComPlat/chemotion_ELN/pull/1000