nokogiri version does not support Ruby 2.3

[robodude666]

Origen's dependency on nokogiri 1.6.7.2 makes it challenging to install from gem on Ruby 2.3.

Could the version be updated to at least 1.6.8 which adds Ruby 2.3 support?

[welguisz]

Right now, we have nokogiri linked to 1.6.7.2. I am going to do the following to see what happens

• Change the version from being hard-coded to being to upgrade within 1.6.* , e.g. '~>1.6.7'
• Change the version to upgrade to 1.7, e.g. ~>'1.7.0'

If it supports 1.7.0 without failing any tests, I wil move it to 1.7.0

[ginty]

Thanks for reporting the issue @robodude666, definitely we need to come up with a solution to enable you asap.

So, the background here is that Origen itself doesn't actually require Nokogiri at all. However, Nokogiri is Ruby's premier XML parser and it is widely used by Origen plugins that are concerned with importing register and pin data and so on from XML-based sources.

The problem with it is that it is mainly written in C, and as such it requires a C build environment to install it. If you are running Ubuntu or some other modern Linux distro, then the plugins that use Nokogiri could simply add it as a dependency in the normal way and you would see no problems. However, something a bit older (RHEL 5/6) is common in a corporate environment such as ours, and smooth installation on that platform is not at the forefront of the Nokogiri maintainer's minds. Subsequently, we have had many issues with individual users not being able to install Origen apps which depend on Nokogiri.

Our solution then, was to lock the whole eco-system down to a single version of Nokogiri by adding it to Origen, take the pain once by installing it to a central location, and then making Origen always use it from there via our company configuration file - see http://origen-sdk.org/origen/guides/starting/company/.

I think the easiest way forward in the short term might be to open up the Origen dependency, make it ">= 1.6.7.2" and then we have an internal plugin that we use in almost all of our applications and we could lock it to 1.6.7.2 there.

In parallel, we can look at bumping our environment to Ruby 2.3 and then we could bump the Nokogiri version to 1.6.8 or higher.

Windows is also a concern too, some Nokogiri versions in the past have not supported Windows, so we need to bear that in mind when deciding on which version we upgrade to.

@priyavadan, @chrisnappi, any comments on the proposal?

[ginty]

Hi @robodude666, what platform are you trying to install on?

I just noticed that on Linux we have Origen working fine with Ruby 2.3.0 and Nokogiri 1.6.7.2:

$ nokogiri -v
# Nokogiri (1.6.7.2)
    ---
    warnings: []
    nokogiri: 1.6.7.2
    ruby:
      version: 2.3.0
      platform: x86_64-linux
      description: ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-linux]
      engine: ruby

And on Windows, if you follow the installation guide it should give you Ruby 2.2.4 which also works fine:

C:\origen>nokogiri -v
# Nokogiri (1.6.7.2)
    ---
    warnings: []
    nokogiri: 1.6.7.2
    ruby:
      version: 2.2.4
      platform: x64-mingw32
      description: ruby 2.2.4p230 (2015-12-16 revision 53155) [x64-mingw32]
      engine: ruby

[robodude666]

I'm running on Windows 7, and oddly enough Ruby 2.2.4 did not work for me. I received a symlink() function is unimplemented on this machine error while trying to install the origen gem. Updating to Ruby 2.3 seemed to fix that issue, but causes other issues with nokogiri's dependency to Ruby <2.3.

[ginty]

Seems related to this, though no final solution: https://github.com/rubygems/rubygems/issues/1623

[ginty]

Finding and removing the offending symlink in Origen is probably the easiest fix if someone fancies looking for it.

[coreyeng]

Our windows users found this as well, but we found that its more tied to the version of ruby gems than ruby itself. We backtracked to ruby-gems 2.2.3 and were able to install Nokogiri alright. That said though, I am using Ruby 2.2.4p230 as well. A command that was in the docs at one point for this was:

@powershell (new-object System.Net.WebClient).DownloadFile('https://github.com/rubygems/rubygems/releases/download/v2.2.3/rubygems-update-2.2.3.gem’,'C:\rubygems-update-2.2.3.gem') && gem install --local C:\rubygems-update-2.2.3.gem && update_rubygems --no-ri --no-rdoc && gem uninstall rubygems-update -x

We used the above instead of the current powershell command in the Windows install guide.

I meant to look into this but haven't had the time.

As an aside, nokogiri can also be installed with pre-compiled DLLs, which should fix this as well. But then I believe nokogiri gets slowed down some. I'd recommend trying ruby gems version 2.2.3 first.

-Edit: also need to add the --source http://rubygems.org when installing Origen. This was another step we had to add that is different than the install guide. Though, the install guide does have a comment about this issue.

[priyavadan]

@ginty I agree, we need to look into bumping Nokogiri to 1.6.8 and beyond. Will need to be careful about breaking something else in the older RHEL 5/6 environment while doing this.

[ginty]

@robodude666, just had a look in origen for a symlink and I can't see any. Is it in one of Origen's dependencies maybe, any indication of that from your terminal? Finding the offending symlink and getting rid of it seems to be the safest/quickest way to a solution here.

[ginty]

@info-rchitect, you just did a fresh install on Windows right, no issue like this?

[coreyeng]

I don't think the symlink is in Origen. I think its in the build script for nokogiri. Its not something we can remove. The current powershell script installs rubygems 2.6.7 or something, which has the symlink removed. I think the best solution, which you've all said too, is allowing for new versions of nokogiri to be used by Origen, and letting company specific plugins dictate if a static version is needed. A temporary solution though would be for Origen's install guide to backtrack to rubygems 2.2.3, where Nokogiri can build alright.

[coreyeng]

Example stack trace:

NotImplementedError: symlink() function is unimplemented on this machine
  C:/tools/ruby22/lib/ruby/site_ruby/2.2.0/rubygems/package.rb:388:in `symlink'
  C:/tools/ruby22/lib/ruby/site_ruby/2.2.0/rubygems/package.rb:388:in `block (2 levels) in extract_tar_gz'
  C:/tools/ruby22/lib/ruby/site_ruby/2.2.0/rubygems/package/tar_reader.rb:65:in `each'
  C:/tools/ruby22/lib/ruby/site_ruby/2.2.0/rubygems/package.rb:365:in `block in extract_tar_gz'
  C:/tools/ruby22/lib/ruby/site_ruby/2.2.0/rubygems/package.rb:459:in `block in open_tar_gz'
  C:/tools/ruby22/lib/ruby/site_ruby/2.2.0/rubygems/package.rb:456:in `wrap'
  C:/tools/ruby22/lib/ruby/site_ruby/2.2.0/rubygems/package.rb:456:in `open_tar_gz'
  C:/tools/ruby22/lib/ruby/site_ruby/2.2.0/rubygems/package.rb:364:in `extract_tar_gz'
  C:/tools/ruby22/lib/ruby/site_ruby/2.2.0/rubygems/package.rb:345:in `block (2 levels) in extract_files'
  C:/tools/ruby22/lib/ruby/site_ruby/2.2.0/rubygems/package/tar_reader.rb:65:in `each'
  C:/tools/ruby22/lib/ruby/site_ruby/2.2.0/rubygems/package.rb:342:in `block in extract_files'
  C:/tools/ruby22/lib/ruby/2.2.0/open-uri.rb:36:in `open'
  C:/tools/ruby22/lib/ruby/2.2.0/open-uri.rb:36:in `open'
  C:/tools/ruby22/lib/ruby/site_ruby/2.2.0/rubygems/package/file_source.rb:30:in `with_read_io'
  C:/tools/ruby22/lib/ruby/site_ruby/2.2.0/rubygems/package.rb:339:in `extract_files'
  C:/tools/ruby22/lib/ruby/site_ruby/2.2.0/rubygems/installer.rb:772:in `extract_files'
  C:/tools/ruby22/lib/ruby/site_ruby/2.2.0/rubygems/installer.rb:302:in `install'
  C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/bundler-1.13.6/lib/bundler/source/rubygems.rb:144:in `block in install'
  C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/bundler-1.13.6/lib/bundler/rubygems_integration.rb:181:in `preserve_paths'
  C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/bundler-1.13.6/lib/bundler/source/rubygems.rb:136:in `install'
  C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/bundler-1.13.6/lib/bundler/installer/gem_installer.rb:55:in `install'
  C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/bundler-1.13.6/lib/bundler/installer/gem_installer.rb:15:in `install_from_spec'
  C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/bundler-1.13.6/lib/bundler/installer/parallel_installer.rb:104:in `block in worker_pool'
  C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/bundler-1.13.6/lib/bundler/worker.rb:65:in `call'
  C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/bundler-1.13.6/lib/bundler/worker.rb:65:in `apply_func'
  C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/bundler-1.13.6/lib/bundler/worker.rb:60:in `block in process_queue'
  C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/bundler-1.13.6/lib/bundler/worker.rb:57:in `loop'
  C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/bundler-1.13.6/lib/bundler/worker.rb:57:in `process_queue'
  C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/bundler-1.13.6/lib/bundler/worker.rb:29:in `block (2 levels) in initialize'

[coreyeng]

And example environment:

Bundler 1.13.6 Rubygems 2.6.7 Ruby 2.2.4p230 (2015-12-16 revision 53155) [x64-mingw32] GEM_HOME C:/Users/rxxxxx/.origen/gems/ruby/2.2.0 GEM_PATH Git not installed

[welguisz]

Do we even want to keep the dependency with Nokogiri and make it up the app developer to choose their version that they want to support?

I am leaning to taking out the dependency since it forces people to use Nokogiri while there are other XML parsers out there that are much easier to use, e.g. Ox.

[priyavadan]

David,

I believe that other gems that are used by Origen have nokogiri dependencies too, so even if we take out Nokogiri from Origen we still will need it to be present on the system? unless my understanding is incorrect.

- Priyavadan

[welguisz]

@priyavadan ... be careful using the reply function from github. Your number appeared in the previous comment. Another reason not to include a version of a gem unless it is needed is how bundler works.

Consider the following: Origen has the following gems for dependencies: Gem A and Gem B. If Gem A and Gem B has the following gem dependencies, bundler will crash and say that it is not possible because bundler can't figure out which Gem N verison to use. So if Gem A is not needed, then bundler is able to handle the situation. Gem A dependencies:

• Gem M, '~>1.0'
• Gem N, '>1.3.0, <2.0.0'
• Gem P, '>2.7.1'

Gem B dependencies:

• Gem N , '>2.0.1'

[priyavadan]

Thanks David!

[ginty]

@robodude666, @coreyeng, the latest Origen now requires Nokogiri 1.7.2, please try and feedback if that fixes the issue. I'll close this ticket after a while if I don't hear anything.

I decided not to remove the absolute version dependency in the end, this guy has caused a lot of installation issues and I think it is better to keep a tight reign on it.

I have added a new origen_updater plugin which did make the transition from 1.6.7.2 to 1.7.2 reasonably painless within our environment, there are details on that here if you are interested: http://origen-sdk.org/origen/guides/starting/workspace/#Fix_My_Workspace

[ginty]

Assuming this is now resolved.

[coreyeng]

@ginty Sorry for the late response. I verified that all the changes still work on our end though. Looks good.

[info-rchitect]

All,

Why are we hard-coding a single version of nokogiri?

spec.add_runtime_dependency "nokogiri", "1.7.2"

Can we just make it >= some version? Here is my dependency issue:

Resolving dependencies...
Bundler could not find compatible versions for gem "nokogiri":
  In Gemfile:
    clark_kent was resolved to 0.1.0, which depends on
      axlsx was resolved to 3.0.0.pre, which depends on
        nokogiri (>= 1.8.2, ~> 1.8)

    sire was resolved to 0.1.0, which depends on
      origen was resolved to 0.29.0, which depends on
        nokogiri (~> 1.7.2)

thx

[welguisz]

@info-rchitect I think that it should be up to the app developer to specify the Nokogiri version. I would prefer

spec.add_runtime_dependency "nokogiri", ">= 1.7.2"

This way we don't lock it down and allows for the app developer to tie it to a version later.

[info-rchitect]

@welguisz thx for the quick response, I agree. I made an emergency patch for my local copy of Origen but think this should be default behavior. BTW, the reason this suddenly changed is because I had to link directly to the axlsx github site version, which changed its nokogiri requirements. The reason I had to link directly is because of this issue with the roo gem.

[ginty]

@welguisz, I think everyone would prefer that, if you want to co-ordinate that without causing havoc throughout NXP please go ahead.

[info-rchitect]

@ginty can we make this specific version of nokogiri be applied only if @ NXP?

[welguisz]

Pull Request for this Issue

[chrisnappi]

Still continues to be a pain point, I agree with @welguisz that we need to look at Ox - but maybe we need to look at the most complex use case rather than core first. Would that be Cross Origen? Or the NXP internal version of same?

[ginty]

The problem though is that this is a kind of viral gem, just by not using it ourselves will not prevent it from coming in via the backdoor. Here's 213 pages of gems which depend on it, and for example I can see on the first page is roo which is used quite often in our apps.

[ginty]

Sorry, meant to paste this link - https://rubygems.org/gems/nokogiri/reverse_dependencies