utf-8 strings and unexpected end-of-input

[zenbro]

Hi!

Every method call with utf-8 string as argument ends with following error: unexpected end-of-input, expecting keyword_end

ruby -wc returns Syntax OK.

Simple code to reproduce this:

def my_method
  puts 'привет'  # utf-8 string
end

Atom version: 1.12.6 linter-ruby version: 1.2.4 System: macOS 10.12.1 Ruby versions: 2.1.10, 2.2.3, 2.3.3

[zenbro]

Adding # encoding: utf-8 solves the problem. Btw, ruby installed via rvm.

[zenbro]

Starting atom from terminal also solves the problem. Seems like problem in environment variables.

[alistairholt]

This is quite irritating. Anyone know why this has recently broken?

[Arcanemagus]

Hmmm, in v1.2.4 the argument sent to Ruby was changed from -E utf-8 to -Eutf-8 (https://github.com/AtomLinter/linter-ruby/commit/f4a60265b66417fc7038730855956877db273c28) as the first method on my computer (Windows 10 x64 + Ruby v2.3.1p112) attempts to set the language to utf-8 (note the space), which doesn't exist. Maybe Ruby is "special" and handles this differently on different platforms? 😕

@alistairholt what platform are you on?

---

The proper solution is as @zenbro already said: Adding the following at the start of the file:

# encoding: utf-8

This tells Ruby the encoding of the file in the proper method, the argument is only meant to be a fallback to try to catch cases where the developer forgot it.

[timhaines]

@Arcanemagus With Ruby 2.0 utf-8 is the default encoding, and that encoding comment is no longer required.

2.0 Release notes

FWIW I struck this "bug" today too and came looking for a solution. In my case it's thinking ’ is terminating a single quoted string.

[stevehanson]

I'm also seeing this issue. I just quit Atom and restarted from Terminal per @zenbro's recommendation and am not seeing the issue anymore. I'm on MacOS Sierra.

[black-snow]

@timhaines I'm on ruby 2.2.3 and still having this without the encoding-comment. MacOS Sierra, linter-ruby 1.2.4, atom 1.13.1 x64

[fabien-michel]

/!\ This workaround make linter-ruby to not detect any errors Workaround : Open the linter-ruby/lib/main.js file and add a space between -E and utf-8 on line 39 const execArgs = ['-wc', '-E utf-8'];

[alistairholt]

@Arcanemagus I'm on macOS Sierra 10.12.3. As @timhaines said, we shouldn't need to add the magic encoding comment as UTF-8 is the default script encoding since Ruby v2.0.

My Ruby (ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin16]) man page shows -E external[:internal]. @fabien-michel 's change fixed my issue.

[wdspkr]

I tried @fabien-michel 's workaround and that actually leads to the linter not working anymore: The unexpected end-of-input error vanishes, but so do all other real errors!

The only way to get the linter working correctly for me right now is @stevehanson 's solution: Restarting atom from the terminal.

[alistairholt]

@wdspkr which platform are you on and which Ruby are you running?

[fabien-michel]

@wdspkr You are true, I've updated my comment to warn readers

[Arcanemagus]

Can somebody on macOS try the changes from https://github.com/AtomLinter/linter-ruby/pull/113?

[fabien-michel]

@Arcanemagus It does not fix the problem. linter work but unicode chars in string still raise the "unexpected end-of-input" error

[Arcanemagus]

And this is only an issue when Atom is launched from the GUI, correct? It works if launched from a terminal?

[Arcanemagus]

If anyone is willing to help debug this I'm available on Atom's Slack. I have no access to a macOS machine, and it seems that the fix that works for some of you isn't working for others, so it's a rather complex issue with how Ruby works differently between OS versions.

[fabien-michel]

I've rolled some tests :

1. -U execArg, does not work
2. Tried run same command linter ruby run from command line, work with both ruby 2.0 and ruby 2.3 and any execArgs (-U, -Ku, -Eutf-8, internal-encoding, etc...)
3. Previous -Ku execArgs make it works as expected
4. Changed main.js to not send stdIn but only file path make it work as exepcted

const execArgs = [
         '-c', // Check syntax only, no execution
         '-w', // Turns on warnings
         // Set the encoding to UTF-8
         '-Eutf-8',
         filePath
       ];
const execOpts = {
      //    stdin: fileText,
          stream: 'stderr',
          allowEmptyStderr: true,
        };

[Arcanemagus]

@fabien-michel That completely removes the ability to lint the current contents of the editor and instead only lints the on-disk status.

[wdspkr]

I tried the changes in #113 on MacOS. They do not fix the error. @fabien-michel 's changes on the other hand do the trick.
What do you mean by 'ability to lint the current contents' @Arcanemagus ? As far as I have experienced it, the linter only lints when a file has been saved to disk. Or am I missing something?

[Arcanemagus]

@wdspkr Did you disable "Lint on Change" in the Linter settings? This provider is currently marked as supporting linting files as they change.

The entire problem here is that when sending the content to ruby over stdin we need to tell it the encoding of said content, and on macOS for some bizarre reason they used a different format for the options than every other OS. Even stranger for some people on macOS the change to -E utf-8 works, while for others nothing seems to properly specify the encoding.

113 is an attempt to work around the mismatching option style, since -Eutf-8 is simply a shortcut for specifying both the internal and external encoding schemes, and luckily the was of setting those individually is the same on all platforms. So #113 just sets them both individually, which still works for all other platforms and works for some macOS people, but apparently not all.

It's looking like to figure this out I'll need somebody where the changes from #113 aren't working to let me investigate their system to see if we can find something that works for everyone.

[wdspkr]

@Arcanemagus TIL! Sweet. Indeed I had "Lint on Change" disabled.
I still don't get all the inner workings of Linter, but I accidentally came to a configuration that works also on change. It's basically @fabien-michel 's solution but with the stdin: fileText line uncommented. There's no difference if the encoding is -Eutf-8 or --external-encoding=utf-8 though. The only thing that makes the difference is whether the filepath is given as an execArg or not.

const execArgs = [
  '-c', // Check syntax only, no execution
  '-w', // Turns on warnings
  // Set the encoding to UTF-8
  '--external-encoding=utf-8',
  '--internal-encoding=utf-8',
  filePath
];
const execOpts = {
  stdin: fileText,
  stream: 'stderr',
  allowEmptyStderr: true,
};

[Arcanemagus]

@wdspkr Can you verify that it's actually linting the contents of the editor? I thought that if you provided a file name it ignored anything on stdin.

And yes, the entire point of switching to --external-encoding=utf-8 and --internal-encoding=utf-8 is that it is the exact same as -Eutf-8 (or -E utf-8), but works on all platforms.

[wdspkr]

@Arcanemagus You are right. stdin is ignored. I was tricked by the octocat linter spinner that spins after typing, but it does not show any linter errors until the file is saved. 😞

I'm running out of ideas and guess I will just resort to starting atom from the terminal (which still works). Thanks for your time though. This project really helps my workflow! ❤️

[phoenixweiss]

I have the same issue and it still solvable only by starting atom from terminal or by adding this string # encoding: utf-8 in the beginning of file (but it's quite a dumb decision). Such confusing.

[wdspkr]

This https://github.com/atom/atom/issues/15634 kind of breaks the "start atom from terminal" workaround. 😞

[jarvis6561]

@zenbro Any update on #128 ? Still facing this issue.

macOS: 10.12.3 Atom: 1.34.0 linter-ruby: 1.3.0 Ruby version: 2.0.0, 2.4.1

[zenbro]

@jarvis6561 no updates yet. Only solution that helps - starting atom from terminal.