Ruby 2.7 support - Githubissues

deepj commented 4 years ago

Is there any plan to add new Ruby 2.7 goodies in near future? In my list are these:

REPL improvements (the colors and syntax highlighting in IRB!!!)
pattern matching (+ possible some performance optimization behind MRI implementation)
Enumerator.produce

eregon commented 4 years ago

As written in https://github.com/oracle/truffleruby/blob/master/doc/contributor/workflow.md#running-specs-for-ruby-27-features

we welcome pull requests for Ruby 2.7 features as long as they don't conflict significantly with Ruby 2.6 semantics.

So adding 2.7 features as long as they don't conflict significantly with Ruby 2.6 semantics is fine.

Regarding 1, I guess it's mostly changes the irb gem. So the work there seems mostly to support the latest irb gem. Trying gem install irb; irb shows we need Ripper.lex (#1747, #1585)

Regarding 2, it seems quite some work and it's unclear if much code uses it yet. Also it's experimental in 2.7:

$ ruby -ve 'case [3]; in [e]; p e; end'
ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]
-e:1: warning: Pattern matching is experimental, and the behavior may change in future versions of Ruby!
3

So PRs for that are definitely welcome, but I guess it's not used in many gems yet, and so not so important for compatibility in general (yet).

Regarding 3, that's probably easy to implement in pure Ruby.

For each of these it would be helpful to have a separate issue so we can prioritize and triage it as needed.

Finally, regarding general plans for 2.7 at the moment I'm hesitant because of keyword arguments changes. Ruby 2.7 has broken *args-delegation and *args, **kwargs-delegation is not working in 2.7 (so the only thing that works is ruby2_keywords, details). As a result, it's hard to migrate and fix warnings in Ruby 2.7. So I'm not sure if much production apps will ever run on 2.7, it's unclear to me if they would rather wait for 2.8/3.0 which fixes *args, **kwargs-delegation and has much clearer semantics for keyword arguments in general.

Implementing the keyword arguments semantics of 2.7 seem a lot of work (at least to support the warnings), for semantics that only last for one Ruby version (semantics are significantly simpler in Ruby 2.8+). So we might want to skip the 2.7 "transition release" and go straight to Ruby 2.8 in terms of reported RUBY_VERSION. I've not made my mind yet though and I'mhappy to hear opinions on this (e.g., significant adoption of 2.7 would change things).

Individual features from 2.7 that are compatible are all welcome though. I guess it doesn't make sense to report as RUBY_VERSION 2.7 if the keyword arguments changes are not implemented though.

eregon commented 4 years ago

We're now planning to target Ruby 2.7 for the GraalVM/TruffleRuby 21.0.0 release.

eregon commented 3 years ago

I merged initial support for Ruby 2.7 in 08187f1f6e59dc72971e0f0acd94beb01306615c. That updated the stdlib, default and bundled gems, and some basic fixes needed to get the tests running. The new IRB seems to work well. Bundler is stdlib is now Bundler 2 like MRI.

Remaining:

[x] Implement keyword arguments changes (https://github.com/ruby/ruby/blob/v2_7_0/NEWS#the-spec-of-keyword-arguments-is-changed-towards-30-) @eregon
[ ] Implement def m(**nil) (https://github.com/oracle/truffleruby/commit/9aaf6be7245b99fa5a8fa065a79228174cbc1b8b allows to parse it but treats it like **kw)
[ ] Implement the new *_kw functions in the C API.
[x] Implement beginless Ranges. @LillianZ #2155
[ ] Implement pattern matching (https://github.com/oracle/truffleruby/commit/d58c9430b1dc7caaaeb3ea8d7be62b50951c7b6a, https://github.com/oracle/truffleruby/commit/a3a2c85be19a806e8b2252662651ca33a0dba9f2) @razetime
[x] Remove tainting since it seems deprecated and noop in 2.7. @bjfish

New tagged specs:

New excluded MRI tests:

eregon commented 3 years ago

I'll also copy the checklist from https://github.com/ruby/spec/issues/745 as it might be useful to track progress, and to show individual things to fix. Some of these items might already be done.

NOTE: https://rubyreferences.github.io/rubychanges/2.7.html gives more details for many features and changes. From https://github.com/ruby/ruby/blob/v2_7_0/NEWS:

NEWS for Ruby 2.7.0

This document is a list of user visible feature changes made between releases except for bug fixes.

Changes since the 2.6.0 release

Language changes

Pattern matching

[ ] @razetime Pattern matching is introduced as an experimental feature. Feature #14912

case [0, [1, 2, 3]]
in [a, [b, *c]]
  p a #=> 0
  p b #=> 1
  p c #=> [2, 3]
end

case {a: 0, b: 1}
in {a: 0, x: 1}
  :unreachable
in {a: 0, b: var}
  p var #=> 1
end

case -1
in 0 then :unreachable
in 1 then :unreachable
end #=> NoMatchingPatternError

json = <<END
{
  "name": "Alice",
  "age": 30,
  "children": [{ "name": "Bob", "age": 2 }]
}
END

JSON.parse(json, symbolize_names: true) in {name: "Alice", children: [{name: name, age: age}]}

p name #=> "Bob"
p age  #=> 2

JSON.parse(json, symbolize_names: true) in {name: "Alice", children: [{name: "Charlie", age: age}]}
#=> NoMatchingPatternError

See the following slides for more details:

https://speakerdeck.com/k_tsj/pattern-matching-new-feature-in-ruby-2-dot-7
Note that the slides are slightly obsolete.
[x] The warning against pattern matching can be suppressed with {-W:no-experimental option}[#label-Warning+option].

The spec of keyword arguments is changed towards 3.0

[x] Non-symbols are allowed as keyword argument keys if the method accepts arbitrary keywords. Feature #14183

Non-Symbol keys in a keyword arguments hash were prohibited in 2.6.0, but are now allowed again. Bug #15658

def foo(**kw); p kw; end; foo("str" => 1) #=> {"str"=>1}

[x] **nil is allowed in method definitions to explicitly mark that the method accepts no keywords. Calling such a method with keywords will result in an ArgumentError. Feature #14183

def foo(h, **nil); end; foo(key: 1)       # ArgumentError
def foo(h, **nil); end; foo(**{key: 1})   # ArgumentError
def foo(h, **nil); end; foo("str" => 1)   # ArgumentError
def foo(h, **nil); end; foo({key: 1})     # OK
def foo(h, **nil); end; foo({"str" => 1}) # OK

[x] Passing an empty keyword splat to a method that does not accept keywords no longer passes an empty hash, unless the empty hash is necessary for a required parameter, in which case a warning will be emitted. Remove the double splat to continue passing a positional hash. Feature #14183

h = {}; def foo(*a) a end; foo(**h) # []
h = {}; def foo(a) a end; foo(**h)  # {} and warning
h = {}; def foo(*a) a end; foo(h)   # [{}]
h = {}; def foo(a) a end; foo(h)    # {}

[x] Above warnings can be suppressed also with {-W:no-deprecated option}[#label-Warning+option].

Numbered parameters

[x] Numbered parameters as default block parameters are introduced. Feature #4475

[1, 2, 10].map { _1.to_s(16) }    #=> ["1", "2", "a"]
[[1, 2], [3, 4]].map { _1 + _2 }  #=> [3, 7]

You can still define a local variable named _1 and so on, and that is honored when present, but renders a warning.

_1 = 0            #=> warning: `_1' is reserved for numbered parameter; consider another name
[1].each { p _1 } # prints 0 instead of 1

proc/lambda without block is deprecated

[x] Proc.new and Kernel#proc with no block in a method called with a block is warned now.
```
def foo
  proc
end
foo { puts "Hello" } #=> warning: Capturing the given block using Kernel#proc is deprecated; use `&block` instead
```
This warning can be suppressed with {-W:no-deprecated option}[#label-Warning+option].

[x] Kernel#lambda with no block in a method called with a block raises an exception.

def bar
  lambda
end
bar { puts "Hello" } #=> tried to create Proc object without a block (ArgumentError)

Other miscellaneous changes

[x] A beginless range is experimentally introduced. It might be useful in case, new call-sequence of the Comparable#clamp, constants and DSLs. Feature #14799
```
ary[..3]  # identical to ary[0..3]

case RUBY_VERSION
when ..."2.4" then puts "EOL"
# ...
end

age.clamp(..100)

where(sales: ..100)
```

[x] Quoted here-document identifiers must end within the same line.

<<"EOS
" # This had been warned since 2.4; Now it raises a SyntaxError
EOS

[x] The flip-flop syntax deprecation is reverted. Feature #5400
[x] Comment lines can be placed between fluent dot now.
```
foo
  # .bar
  .baz # => foo.baz
```
[x] Calling a private method with a literal self as the receiver is now allowed. Feature #11297 Feature #16123

[x] Modifier rescue now operates the same for multiple assignment as single assignment. Bug #8279

a, b = raise rescue [1, 2]
# Previously parsed as: (a, b = raise) rescue [1, 2]
# Now parsed as:         a, b = (raise rescue [1, 2])

[x] yield in singleton class syntax is warned and will be deprecated later. Feature #15575.

def foo
 class << Object.new
   yield #=> warning: `yield' in class syntax will not be supported from Ruby 3.0. [Feature #15575](https://bugs.ruby-lang.org/issues/15575)
 end
end
foo { p :ok }

This warning can be suppressed with {-W:no-deprecated option}[#label-Warning+option].

[x] Argument forwarding by (...) is introduced. Feature #16253
```
def foo(...)
  bar(...)
end
```
All arguments to foo are forwarded to bar, including keyword and block arguments. Note that the parentheses are mandatory. bar ... is parsed as an endless range.
[x] Access and setting of $SAFE is now always warned. $SAFE will become a normal global variable in Ruby 3.0. Feature #16131
[x] Object#{taint,untaint,trust,untrust} and related functions in the C-API no longer have an effect (all objects are always considered untainted), and are now warned in verbose mode. This warning will be disabled even in non-verbose mode in Ruby 3.0, and the methods and C functions will be removed in Ruby 3.2. Feature #16131
[x] Refinements take place at Object#method and Module#instance_method. Feature #15373

Command line options

Warning option

The -W option has been extended with a following :, to manage categorized warnings. Feature #16345 Feature #16420

[x] To suppress deprecation warnings:

$ ruby -e '$; = ""'
-e:1: warning: `$;' is deprecated

$ ruby -W:no-deprecated -e '$; = //'

[x] It works with the RUBYOPT environment variable:
```
$ RUBYOPT=-W:no-deprecated ruby -e '$; = //'
```

[x] To suppress experimental feature warnings:

$ ruby -e '0 in a'
-e:1: warning: Pattern matching is experimental, and the behavior may change in future versions of Ruby!

$ ruby -W:no-experimental -e '0 in a'

[x] To suppress both by using RUBYOPT, set space separated values:

$ RUBYOPT='-W:no-deprecated -W:no-experimental' ruby -e '($; = "") in a'

See also Warning in {Core classes updates}[#label-Core+classes+updates+-28outstanding+ones+only-29].

Core classes updates (outstanding ones only)

Array

[x] Added Array#intersection. Feature #16155
[x] Added Array#minmax, with a faster implementation than Enumerable#minmax. Bug #15929

Comparable

[x] Comparable#clamp now accepts a Range argument. Feature #14784

```ruby
-1.clamp(0..2) #=> 0
 1.clamp(0..2) #=> 1
 3.clamp(0..2) #=> 2
# With beginless and endless ranges:
-1.clamp(0..)  #=> 0
 3.clamp(..2)  #=> 2
```

Complex

[x] Added Complex#<=>. So 0 <=> 0i will not raise NoMethodError. Bug #15857

Dir

[x] Dir.glob and Dir.[] no longer allow NUL-separated glob pattern. Use Array instead. Feature #14643

Encoding

[ ] Added new encoding CESU-8. Feature #15931

Enumerable

[x] Added Enumerable#filter_map. Feature #15323

[1, 2, 3].filter_map {|x| x.odd? ? x.to_s : nil } #=> ["1", "3"]

[x] Added Enumerable#tally. Feature #11076

["A", "B", "C", "B", "A"].tally #=> {"A"=>2, "B"=>2, "C"=>1}

Enumerator

[x] Added Enumerator.produce to generate an Enumerator from any custom data transformation. Feature #14781

require "date"
dates = Enumerator.produce(Date.today, &:succ) #=> infinite sequence of dates
dates.detect(&:tuesday?) #=> next Tuesday

[x] Added Enumerator::Lazy#eager that generates a non-lazy enumerator from a lazy enumerator. Feature #15901

a = %w(foo bar baz)
e = a.lazy.map {|x| x.upcase }.map {|x| x + "!" }.eager
p e.class               #=> Enumerator
p e.map {|x| x + "?" }  #=> ["FOO!?", "BAR!?", "BAZ!?"]

[x] Added Enumerator::Yielder#to_proc so that a Yielder object can be directly passed to another method as a block argument. Feature #15618

Fiber

[x] Added Fiber#raise that behaves like Fiber#resume but raises an exception on the resumed fiber. Feature #10344

File

[x] File.extname now returns a dot string for names ending with a dot on non-Windows platforms. Bug #15267
```
  File.extname("foo.") #=> "."
```

FrozenError

[x] Added FrozenError#receiver to return the frozen object on which modification was attempted. To set this object when raising FrozenError in Ruby code, FrozenError.new accepts a :receiver option. Feature #15751

GC

[x] (MRI-specific) Added GC.compact method for compacting the heap. This function compacts live objects in the heap so that fewer pages may be used, and the heap may be more CoW (copy-on-write) friendly. Feature #15626

Details on the algorithm and caveats can be found here: https://bugs.ruby-lang.org/issues/15626

IO

[ ] Added IO#set_encoding_by_bom to check the BOM and set the external encoding. Bug #15210

Integer

[x] Integer#[] now supports range operations. Feature #8842

 0b01001101[2, 4]  #=> 0b0011
 0b01001100[2..5]  #=> 0b0011
 0b01001100[2...6] #=> 0b0011
 #   ^^^^

Method

[x] Method#inspect shows more information. Feature #14145

Module

[x] Added Module#const_source_location to retrieve the location where a constant is defined. Feature #10771
[x] Added Module#ruby2_keywords for marking a method as passing keyword arguments through a regular argument splat, useful when delegating all arguments to another method in a way that can be backwards compatible with older Ruby versions. Bug #16154
[x] Module#autoload? now takes an inherit optional argument, like Module#const_defined?. Feature #15777
[x] Module#name now always returns a frozen String. The returned String is always the same for a given Module. This change is experimental. Feature #16150

NilClass / TrueClass / FalseClass

[x] NilClass#to_s, TrueClass#to_s, and FalseClass#to_s now always return a frozen String. The returned String is always the same for each of these values. This change is experimental. Feature #16150

ObjectSpace::WeakMap

[x] ObjectSpace::WeakMap#[]= now accepts special objects as either key or values. Feature #16035

Proc

[x] Added Proc#ruby2_keywords for marking the proc as passing keyword arguments through a regular argument splat, useful when delegating all arguments to another method or proc in a way that can be backwards compatible with older Ruby versions. Feature #16404

Range

[x] Added Range#minmax, with a faster implementation than Enumerable#minmax. It returns a maximum that now corresponds to Range#max. Bug #15807
[x] Range#=== now uses Range#cover? for String arguments, too (in Ruby 2.6, it was changed from Range#include? for all types except strings). Bug #15449

String

[x] Update Unicode version and Emoji version from 11.0.0 to 12.0.0. Feature #15321
[x] Update Unicode version to 12.1.0, adding support for U+32FF SQUARE ERA NAME REIWA. Feature #15195
[x] Update Unicode Emoji version to 12.1. Feature #16272

Symbol

[x] Added Symbol#start_with? and Symbol#end_with? methods. Feature #16348

Time

[x] Added Time#ceil method. Feature #15772
[x] Added Time#floor method. Feature #15653
[x] Time#inspect is separated from Time#to_s and it shows the time's sub second. Feature #15958 (we don't support Rational subseconds)

UnboundMethod

[x] Added UnboundMethod#bind_call method. Feature #15955

umethod.bind_call(obj, ...) is semantically equivalent to umethod.bind(obj).call(...). This idiom is used in some libraries to call a method that is overridden. The added method does the same without allocation of an intermediate Method object.
```
  class Foo
    def add_1(x)
      x + 1
    end
  end
  class Bar < Foo
    def add_1(x) # override
      x + 2
    end
  end

  obj = Bar.new
  p obj.add_1(1) #=> 3
  p Foo.instance_method(:add_1).bind(obj).call(1) #=> 2
  p Foo.instance_method(:add_1).bind_call(obj, 1) #=> 2
```

Warning

[x] Added Warning.[] and Warning.[]= to manage emitting/suppressing some categories of warnings. Feature #16345 Feature #16420

$LOAD_PATH

[x] Added $LOAD_PATH.resolve_feature_path. Feature #15903 Feature #15230

Stdlib updates (outstanding ones only)

[x] Pathname.glob now delegates 3 arguments to Dir.glob to accept base keyword. Feature #14405

Compatibility issues (excluding feature bug fixes)

[x] The following libraries are no longer bundled gems. Install corresponding gems to use these features.
- CMath (cmath gem)
- Scanf (scanf gem)
- Shell (shell gem)
- Synchronizer (sync gem)
- ThreadsWait (thwait gem)
- E2MM (e2mmap gem)

Proc

[x] The Proc#to_s format was changed. Feature #16101

Range

[x] #777 Range#minmax used to iterate on the range to determine the maximum. It now uses the same algorithm as Range#max. In rare cases (e.g. ranges of Floats or Strings), this may yield different results. Bug #15807

Stdlib compatibility issues (excluding feature bug fixes)

[x] Promote stdlib to default gems
- The following default gems were published on rubygems.org
- benchmark
- cgi
- delegate
- getoptlong
- net-pop
- net-smtp
- open3
- pstore
- readline
- readline-ext
- singleton
- The following default gems were only promoted at ruby-core, but not yet published on rubygems.org.
- monitor
- observer
- timeout
- tracer
- uri
- yaml
[x] The did_you_mean gem has been promoted up to a default gem from a bundled gem

pathname

[x] Kernel#Pathname when called with a Pathname argument now returns the argument instead of creating a new Pathname. This is more similar to other Kernel methods, but can break code that modifies the return value and expects the argument not to be modified.

C API updates

[ ] Many *_kw functions have been added for setting whether the final argument being passed should be treated as keywords. You may need to switch to these functions to avoid keyword argument separation warnings, and to ensure correct behavior in Ruby 3.
[x] The : character in rb_scan_args format string is now treated as keyword arguments. Passing a positional hash instead of keyword arguments will emit a deprecation warning.
[x] C API declarations with ANYARGS are changed not to use ANYARGS. See https://github.com/ruby/ruby/pull/2404

ssnickolay commented 3 years ago

I'll start with Added Complex#<=>., if you do not mind //cc @eregon

norswap commented 3 years ago

I'll take a stab at the rescue parser bug.

norswap commented 3 years ago

I'm marking "The flip-flop syntax deprecation is reverted." as done, since it doesn't seem we emitted the deprecation warning in the first place.

LillianZ commented 3 years ago

Could you also mark enumerator#produce as done per https://github.com/oracle/truffleruby/pull/2160 ? Thanks!

gogainda commented 3 years ago

taking Integer#[] now supports range operations

LillianZ commented 3 years ago

Pathname.glob now delegates 3 arguments to Dir.glob to accept base keyword. Feature #14405 already occurs in TruffleRuby, and thus can be removed/ticked off. The examples in the Redmine post al function.

chrisseaton commented 3 years ago

Are there specs for it? If so do they need to be untagged, if not we should add at least a couple.

eregon commented 3 years ago

Very few things left from the list above:

Pattern matching is partially implemented, but in 2.7 it's experimental so likely not used much.
Keyword arguments changes were done, except for some edge cases which make more sense to fix when targeting Ruby 3 (it would be work that would be basically thrown away when moving to Ruby 3)
The new encoding CESU-8, needs support in JCodings.
Fiber#raise
IO#set_encoding_by_bom
C-API *_kw functions

gogainda commented 3 years ago

I looked into C-API *_kw functions but didn't figure out how to approach this works. @eregon Can we do 1 function so I can work on the rest by the example?

eregon commented 3 years ago

@gogainda https://github.com/oracle/truffleruby/blob/169bce9afead36cde38ae32a19d27b0b95cf9f6d/src/main/c/cext/class.c#L79-L82 is an example. The *_kw functions can be found by git grep -E 'rb_.+_kw\b' lib/cext/include/ruby.

gogainda commented 3 years ago

I thought that // Ignoring kw_splat for now should be replaced by actual code, no?

eregon commented 3 years ago

@gogainda At this point it seems unnecessary, because of how we handle keyword arguments in TruffleRuby. I think we'll only need to deal with this when migrating to Ruby 3.0. So it's OK to just ignore it and add this comment as a reminder when we move to Ruby 3.

eregon commented 3 years ago

I think it's time to mark this issue as solved, we already support most of 2.7 since 21.0.0/21.1.0.

Remaining from the list is:

Pattern matching is partially implemented, but in 2.7 it's experimental so likely not used much.
The new encoding CESU-8, needs support in JCodings.
C-API *_kw functions, part of them are already there.

Those don't seem used much so I think we'll just pragmatically add them when we see an app/gem needing them.

oracle / truffleruby

Ruby 2.7 support #2004

NEWS for Ruby 2.7.0

Changes since the 2.6.0 release

Language changes

Pattern matching

The spec of keyword arguments is changed towards 3.0

Numbered parameters

proc/lambda without block is deprecated

Other miscellaneous changes

Command line options

Warning option

Core classes updates (outstanding ones only)

Stdlib updates (outstanding ones only)

Compatibility issues (excluding feature bug fixes)

Stdlib compatibility issues (excluding feature bug fixes)

C API updates