Windows installation code ?

MSP-Greg commented 5 years ago

@SteveALee

Considering this issue isn't related to EventMachine, I thought I'd add the details here.

One thing that is important is that there are no spaces in the path where gems are installed.

My system defaults are to use a gem install path with a space, and almost everything works fine. One of Jekyll's gem dependencies on does have a problem.
Unless noted, my normal shell is PowerShell.
'GEM_HOME can redefine where gem install will install gems, assuming one doesn't use the --user-install option. I often change it when testing installation issues, as it leaves the Ruby install folder alone.

Using a fresh install of Ruby 2.6:

# change GEM_HOME just for testing installation code
$env:GEM_HOME = "<special folder>"

gem install jekyll --platform=ruby
gem install minima tzinfo-data wdm --platform=ruby

jekyll new C:/Greg/jekyll

# cd to  project folder
cd C:/Greg/jekyll

bundle lock --add-platform=x64-mingw32
bundle install

bundle exec jekyll serve --livereload

HTH, Greg

SteveALee commented 5 years ago

Thanks very much for this help Greg!!

Sadly I still get the bad file descriptor error, but this time after a socket related issue. The file in question is not pronted this time.

#<Thread:0x00000000080c3c28@C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/jekyll-3.7.4/lib/jekyll/commands/serve/live_reload_reactor.rb:42 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
        7: from C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/jekyll-3.7.4/lib/jekyll/commands/serve/live_reload_reactor.rb:45:in `block in start'
        6: from C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7-x64-mingw32/lib/eventmachine.rb:196:in `run'
        5: from C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7-x64-mingw32/lib/em/pure_ruby.rb:144:in `run_machine'
        4: from C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7-x64-mingw32/lib/em/pure_ruby.rb:547:in `run'
        3: from C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7-x64-mingw32/lib/em/pure_ruby.rb:547:in `loop'
        2: from C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7-x64-mingw32/lib/em/pure_ruby.rb:553:in `block in run'
        1: from C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7-x64-mingw32/lib/em/pure_ruby.rb:595:in `crank_selectables'
C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7-x64-mingw32/lib/em/pure_ruby.rb:595:in `select': An operation was attempted on something that is not a socket. (Errno::ENOTSOCK)
        7: from C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/jekyll-3.7.4/lib/jekyll/commands/serve/live_reload_reactor.rb:45:in `block in start'
        6: from C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7-x64-mingw32/lib/eventmachine.rb:196:in `run'
        5: from C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7-x64-mingw32/lib/em/pure_ruby.rb:144:in `run_machine'
        4: from C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7-x64-mingw32/lib/em/pure_ruby.rb:562:in `run'
        3: from C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7-x64-mingw32/lib/em/pure_ruby.rb:559:in `ensure in run'
        2: from C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7-x64-mingw32/lib/em/pure_ruby.rb:559:in `each'
        1: from C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7-x64-mingw32/lib/em/pure_ruby.rb:559:in `block in run'
C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7-x64-mingw32/lib/em/pure_ruby.rb:559:in `close': Bad file descriptor (Errno::EBADF)
jekyll 3.7.4 | Error:  Bad file descriptor

Some notes

I installed Ruby 2.6 from https://rubyinstaller.org/downloads/
I had to install bundler so I could use bundle. I used --platform-ruby

My gem paths is without spaces - I guess username is the potential issue?

GEM PATHS:
 - C:/Ruby26-x64/lib/ruby/gems/2.6.0
 - C:/Users/steve/.gem/ruby/2.6.0

Unable to load the EventMachine C extension; To use the pure-ruby reactor, require 'em/pure_ruby' so I added require em/pure_ruby C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7-x64-mingw32/lib/eventmachine.rb`
Still get Please add the following to your Gemfile to avoid polling for changes: gem 'wdm', '>= 0.1.0' if Gem.win_platform? but not sure if i should somehow specify --platform-ruby as you did when manually installed wdm. Anyway do this simple delays the error 'till I update a file - as you would expect.
In any case it helps - here is the repo in question - https://github.com/w3c/wai-coga

On a separate issue I've not quite go to the bottom of best practice for checking in lockfiles to VC (same issues as npm and yarn I guess). The projet I'm working on had a lockfil for OS X dev but was old versions of gems so I deleted it..

MSP-Greg commented 5 years ago

@SteveALee

Thanks for the response.

Believe it or not, this issue exposes quite a nice collection of issues with Bundler/RubyGems and Ruby 2.6 & 2.7. Some are windows specific, others appear because of details with pre-compiled gems, but per se aren't platform specific.

I'm running CI on Jekyll here on Appveyor, and I've got issues with tzinfo. Re your issues, I could explain some and also help with some fixes, I don't know if you want to work on this or not...

I'm not quite sure why your system is installing eventmachine-1.2.7-x64-mingw32, it should be eventmachine-1.2.7, but some of that may be related to the gem, and some to RubyGems.

SteveALee commented 5 years ago

Hehe - I'm always successful at finding problems :)

OK - lets try to fix the issue to help others as well. I'might be a little slower in trying things out though.

MSP-Greg commented 5 years ago

OK - lets try to fix the issue to help others as well.

I saw OpenDirective, and wasn't sure how you'd feel about 'help others', as sooner or later it's about available time and the particular task...
Long story, but with what I'm involved in, I'm familiar with new Ruby versions, and issues related to that caused by Ruby, and Bundler/RubyGems. Also, since I often work with Ruby trunk, my normal workflow is to bypass Bundler, but I guess it's time for me to (hopefully) help with it.
Given how you've described working, I would not recommend using Ruby 2.6, or at least not in 'production'. We may be able to work around the issues, but it may require changes other than in Jekyll.
Given the above, I can describe the issues (which hopefully might be useful when Ruby 2.7 is released), and suggest some work arounds.

If you're interested, I can start writing, I may invite one of the Jekyll members that uses Windows...

SteveALee commented 5 years ago

I saw OpenDirective, and wasn't sure how you'd feel about 'help others', as sooner or later it's about available time and the particular task...

I'm not sure What impression OpenDirective might have given you :) It's my company as a freelance Dev. so I get to choose. I'm currenty on a full time fixed contract but still have some time. Open Dev is my thing, accessibility the main focus. If a tool is broken then I'm happy to help out. Given I'm a long time WIn user/dev (since Windows 3 for workgroups and albeit usually with a bash of some sort or other). I feel Win should be supported as well *nix for dev (like npm is now). I know I could try WSL now it is so functional but it's slower and I'm still mostly happy to use cmd and git 4 Windows bash.

Long story, but with what I'm involved in, I'm familiar with new Ruby versions, and issues related to that caused by Ruby, and Bundler/RubyGems. Also, since I often work with Ruby trunk, my normal workflow is to bypass Bundler, but I guess it's time for me to (hopefully) help with it.

Sounds like you are well placed then. I know almost nothing about Ruby (I read the comic when it was launched) or Jeckyll (use github pages). This project is my 1st real contact. The official Jeckyll docs indicate Win is supported but as we can see, My Mileage Varies on 1st attempt;) I'd also like to get local dev going for https://github.com/mulberrysymbols/mulberry-symbols website on github.io

Given how you've described working, I would not recommend using Ruby 2.6, or at least not in 'production'. We may be able to work around the issues, but it may require changes other than in Jekyll.

This is purely for my local dev on a shared project. The other devs and CI are using linux or OS X (BSD). I think platform choice should not be an issue on open source projects (I know I'm a dreamer)

Given the above, I can describe the issues (which hopefully might be useful when Ruby 2.7 is released), and suggest some work arounds.

If you're interested, I can start writing, I may invite one of the Jekyll members that uses Windows...

Lets do it :)

SteveALee commented 5 years ago

PS I used cmd rather than PowerShell but think it very unlikely that could make any difference. Possibly PATH or subshell differences. But if you think it's worth it I could roll back and trying using PowerShell. My preference is not to spend time on that.

MSP-Greg commented 5 years ago

Writing while you were. The ps1 vs cmd thing isn't an issue. But, the Git Bash shell is. Honestly, I, along with a lot of people, never considered running Windows Ruby from a Bash shell. With 2.6 & trunk, most (all?) of the bin files don't work.

Might that be why you stated 'I had to install bundler so I could use bundle.`?
I would suggest using Bundler 1.7.3 with 2.6.0. Use gem install bundler:1.7.3. Bundler 2.x was pulled from 2.6.0 at the last minute, and is being used for testing in trunk.
You need to mess around with copying and editing files to get the normal Ruby things to work in a bash shell. Any bin file (cmd, bat) you will run, copy, remove the extension, clean the file up to the shebang line, and then change the shebang to simply #! ruby
This change will break the current Windows test setup used by Jekyll (above is the fix).

The detail - Assuming that you have some Ruby versions <= 2.5, you'll notice pairs of files in the bin folder. One had a bat/cmd extension, the other had none. All the bat/cmd files did was launch ruby with the other file as a script. The extension-less files were the same files used for *nix systems. During 2.6 development, a means of allowing all the functionality when run from a Windows shell to be condensed to one bat/cmd file, and the other was removed. That breaks running the commands from a Git Bash shell...

I'm not sure What impression OpenDirective might have given you

A good one. Years ago I volunteered teaching 'disadvantaged' kids for a few years. I'm an old coder, wrote code on Apple II's and the original IBM & Compaq PC's, even used 'time-share' systems with acoustic couplers before that.

I maintain a trunk build that was the first to fully test Ruby on Windows. It's also used by a lot of repos for CI testing.

I thought I'd take the day off of writing code, but I may fix the 'Bash shell' issue in my build system. So, delving into the issues with Jekyll/Bundler/RubyGems may need to wait till later... Thanks, Greg

SteveALee commented 5 years ago

Sorry for any confusion - I stopped using bash as soon as I hit problems. Cmd is fine with me for now.

Years of using bash on Microsoft (I started using MKS on DOS!!!) have taught me that the path mapping is never perfect - MSYS1 used in G4W worked well but still had edge cases. Plus I think you pointed out fork and sockets are isssues in a non POSIX environemnt. Fun!

That said, I'd love to use git bash, but It might just be me :)

One issue you might have is in launching the correct subshells. Usually it should be the same shell as that used in calling the command itself. Windows has no standard way to specify the shell. COMMAND and CMD both use %COMSPEC% but MSYS bash does not. I've no idea about PS. In NPM I found it always looked in COMSPEC and so never knew it was called from bash and so should spawn it as a subshell. Somewhere there's a long an painfull issue where I went through this in great detail.

I also recall messing about with .bat files to launch others so know some of the issues you are facing. I spent many hours becoming a bat guru and finding clever hacks to work around limitations, especially around DOS 3. There are ways to say which extensions cmd trys to run and more such dusty corners but usually it all remains painful

Have fun. I'm in no hurry to fix --livereload as i can hit F5

MSP-Greg commented 5 years ago

Given that the livereload issues appears connected to EventMachine, a word about pre-compiled gems. On Windows, these are gems that end in -x64-mingw32 or -x86-mingw32.

If you look in your

C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7-x64-mingw32/lib/

folder, you'll see folders 2.0 thru 2.3. These are the versions of Ruby that the gem is compiled for. Hence, it can't be used with Ruby 2.4 and later. There is something missing from the gem file, but there are also issues with RubyGems using pre-compiled gems. JFYI, the next EventMachine release will fix all of these issues, and it will be compatible with Ruby 2.0 thru 2.6 (and sort of 2.7).

Hence, that's where the Unable to load the EventMachine C extension; message comes from.

This is why you've seen me suggest --platform=ruby, as that forces RubyGems to use the standard gem, which it then compiles only for the version of Ruby you're installing it in....

ashmaroli commented 5 years ago

In any case it helps - here is the repo in question...

Hello Guys, @SteveALee I'm one of the devs at Jekyll using Windows. I cloned your repo at https://github.com/w3c/wai-coga and was facing a similar issue with Ruby 2.4 and eventmachine-1.2.7-x64-mingw32

I tried again using an older Ruby installation (Ruby 2.3 using RubyInstaller-1) and everything worked properly as expected.

So, something's amiss with eventmachine. Period.

MSP-Greg commented 5 years ago

@ashmaroli

So, something's amiss with eventmachine. Period.

The gem is missing a Ruby version constraint. Try gem install Psych with Ruby 2.6.0. You'll see why 'Period' isn't quite correct...

ashmaroli commented 5 years ago

@MSP-Greg I wasn't complaining. Just emphasizing on the fact that the issue isn't with Jekyll.. Hopefully your contributions to eventmachine-1.3 resolves the main issue here.

MSP-Greg commented 5 years ago

@ashmaroli

Re EventMachine, it will also be built with OpenSSL...

the fact that the issue isn't with Jekyll

Apologies if you got the impresssion I thought that. I've tried to make clear that I do not consider this a jekyll issue, but a combination of Ruby, Bundler, and RubyGems, with a litttle EventMachine twist thrown in...

I'm going to continue to work on CI with trunk (along with other things). If I get it working, I'll let you know...

SteveALee commented 5 years ago

Thanks everyone for helping! It's a delight to take part in such a positive and polite community :)

@MSP-Greg

So If I understand. some Gems come with precompiled binary versions for particular Ruby versions. Obviosly this can't be created on demad as the installed Ruby is a different version. This sort of implies the precompiled binaries are provided for every version of ruby are distributed with the Gems? That seems very bloaty compared to build on install (say, rather like Gentoo linux). especially when it will mean versions of Ruby times number of platforms supported packages distributed?

Also I think you are saying platform=ruby means use the pre built ruby + platform versions that was compiled during installation? But I thought that still failed when I tried it? And I had to use "pureRuby" which I don't understand yet. I guess thats the other problem you speak off

@ashmaroli So my earlier hunch of trying Ruby 2.3 was not totally crazy! :)

Is my best action to just wait a bit to see what you guys come up with?

@MSP-Greg PS BTW I love the Weather Report Album :)

ashmaroli commented 5 years ago

Is my best action to just wait a bit to see what you guys come up with?

@SteveALee You can proceed with Ruby 2.3 from RubyInstaller-1 if you want to use Jekyll with --livereload right away. Otherwise, you may wait for a new release from eventmachine. On a related note, Jekyll v4.0 (under development) is considering to go and require Ruby 2.4 at minimum. So, unless absolute necessary, there's a good chance that installing Ruby 2.3 at this point is not worth it especially since there wont be any more new releases for Ruby 2.3 on Windows.

This sort of implies the precompiled binaries are provided for every version of ruby are distributed with the Gems?

The provision is not for every version of Ruby but for every minor-version-series of Ruby. e.g. : 2.0, 2.1, 2.2, 2.3, etc.

SteveALee commented 5 years ago

@ashmaroli

You can proceed with Ruby 2.3 from RubyInstaller-1

I already tried the 2.3 download from https://rubyinstaller.org/downloads/ but it required a dev env which is not included in the download and the instructions looked more involved than I wanted right now. I have MSYS2 installed but think it was looking for MSYS.

I'll might try the full 2.3 install later

SteveALee commented 5 years ago

@MSP-Greg On a complete tangent

All the bat/cmd files did was launch ruby with the other file as a script

The REALLY annoying thing about using bat file wrappers is when using the Ctrl C interrupt you get the "Terminate batch job (Y/N)" prompt. At least in CMD. Despite hours of seraching and googling there appears to be no way arround this as long a a bat file is used :(

ashmaroli commented 5 years ago

there appears to be no way around this

A crude hack would be:

trap('INT') { abort }

SteveALee commented 5 years ago

I don't think cmd has that but will check - it's not so hot on signals

MSP-Greg commented 5 years ago

@SteveALee

I came across this in a comment in Jekyll's code:

# EventMachine's SSL support in Windows requires building the gem's
# native extensions against OpenSSL and that proved to be a process
# so tedious that expecting users to do it is a non-starter.

That would be a good example of why pre-compiled gems are needed. We could work on getting your setup to compile EventMachine, but I suspect a new pre-compiled Eventmachine gem should be released soon, and it is correctly built with OpenSSL and (currently) supports Ruby 2.0 thru 2.7.

JFYI, a general term for gems that require compiling is extension gems. One could divide them into three categories:

Only compiler tools are needed - example http_parser
External packages/libraries are required, but not libraries used to build Ruby - example nokogiri, which uses libxml2 and libxslt (which also have their source bundled with the gem).
External packages/libraries are required, but the libraries are also used to build Ruby - example eventmachine and puma, as both compile with OpenSSL.

Case 1 is the simplest, case 3 the most difficult, as one must make sure that the package used matches the package used by the particular Ruby version being used.

SteveALee commented 5 years ago

@MSP-Greg

That would be a good example of why pre-compiled gems are needed :)

I'll wait for the update to save us both time. I really appreciate the offer though.

Thanks forthe extra info - quite enlightning. Who said "it's just software"!? LOL And of cause there are C runtime DLL variations to get right :)

SteveALee commented 5 years ago

Are you happy to leave this issue open untill the updates land?

MSP-Greg commented 5 years ago

Yes, we can leave it open. BTW, you're the first person that's mentioned Weather Report... Thanks.

SteveALee commented 5 years ago

FYI my workaround for now is to use the VSCode Debugger for Firefox extension to autoload with changes to the _site folder. Works just fine. https://github.com/w3c/wai-coga/issues/1#issuecomment-457828253

SteveALee commented 5 years ago

Is there an update on when the fix will be released? Thanks

MSP-Greg commented 5 years ago

As to an update to EventMachine, I contribute there, but I don't decide on when to release a new version.

As to the quirks with RubyGems/Bundler, there's a lot of work going on there, but I haven't kept track of the specific issues affecting Jekyll, nor what they've released...

Sorry. If anything changes, I'll add to this issue.

SteveALee commented 5 years ago

Thank you!

MSP-Greg commented 3 years ago

@elizabethwarden

Not a good day for me to be checking this, but maybe tonite or tomorrow. What version of Ruby are you using (ruby -v in a console)?

MSP-Greg / jekyll

Windows installation code ? #1