Ruby support

[ahumenberger]

We want to add language support for Ruby.

• [x] Install Ruby in Docker containers
• [x] Introduce Ruby as a new language
• [x] Register the Ruby language #327
• [x] Implement the ExecuteTests method in the Ruby language #327
• [x] Implement write-tests task
  • [x] Create "plain" repository for Ruby
  • [x] Translate the light repository using a LLM
  • [x] Support Ruby in symflower test
• [x] Implement _coderepair task
  • [x] Translate the mistakes repository using a LLM
  • [x] Implement the Mistakes method in the Ruby language
  • [x] Add missing tests #327
• [x] Implement transpile task
  • The transpile repository used examples from the light repository, so no need to translate them again
  • [x] Change the way we get the origin/destination languages for transpilation
  • [x] The transpile package names must be e.g. for Go transpile-java, transpile-ruby, instead of just transpile so it is clear which language we are transpiling from
  • [x] Add missing tests #327
• [x] Documentation
  • [x] Update documentation with new Ruby support #327
  • [x] New features #327
  • [x] Supported providers
  • [x] New Reward points
• [x] Merge fixes #327

Follow-up

• [x] collect all mistakes and remove duplicates: https://github.com/symflower/eval-dev-quality/pull/326#discussion_r1732496767

[bauersimon]

For new transpilation tasks, there needs to be some small changes to how the eval figures out which languages are targeted. Cause right now if we have Go as transpilation source, it automatically assumes Java must be the target cause no other language existed yet.

[ruiAzevedo19]

Checking for errors on the Ruby mistakes repository

For Java and Go we check for compilation errors with:

• Go: go build
• Java: mvn compile

Since Ruby is an interpreted language, there is no compilation step. I tried the following approaches.

---

Check Ruby's syntax with ruby -c lib/<ruby-file> (see Command line options)

• argumentsMissing: Syntax OK
• endKeywordMissing: lib/end_keyword_missing.rb:9: syntax error, unexpected end-of-input, expecting keyword_end
• importMissing: SyntaxOK
• typeUnknown: SyntaxOK
• variableUnknown: SyntaxOK

Conclusion: only only error caught

---

Running rake test at the root of the repository

• argumentsMissing
  Output

Run options: --seed 64839

# Running:

EEE

Finished in 0.000889s, 3373.6712 runs/s, 0.0000 assertions/s.

  1) Error:
TestArgumentMissing#test_argument_missing2:
ArgumentError: wrong number of arguments (given 1, expected 0)
    /home/rui/dev/ruby/playground/mist/argumentMissing/lib/argument_missing.rb:1:in `argument_missing'
    /home/rui/dev/ruby/playground/mist/argumentMissing/test/argument_missing_test.rb:15:in `test_argument_missing2'

  2) Error:
TestArgumentMissing#test_argument_missing3:
ArgumentError: wrong number of arguments (given 1, expected 0)
    /home/rui/dev/ruby/playground/mist/argumentMissing/lib/argument_missing.rb:1:in `argument_missing'
    /home/rui/dev/ruby/playground/mist/argumentMissing/test/argument_missing_test.rb:22:in `test_argument_missing3'

  3) Error:
TestArgumentMissing#test_argument_missing1:
ArgumentError: wrong number of arguments (given 1, expected 0)
    /home/rui/dev/ruby/playground/mist/argumentMissing/lib/argument_missing.rb:1:in `argument_missing'
    /home/rui/dev/ruby/playground/mist/argumentMissing/test/argument_missing_test.rb:8:in `test_argument_missing1'

3 runs, 0 assertions, 0 failures, 3 errors, 0 skips
rake aborted!
Command failed with status (1)

Tasks: TOP => test
(See full trace by running task with --trace)

• endKeywordMissing

Output

``` /home/rui/dev/ruby/playground/mist/endKeywordMissing/lib/end_keyword_missing.rb:4: warning: mismatched indentations at 'end' with 'if' at 2 /home/rui/dev/ruby/playground/mist/endKeywordMissing/lib/end_keyword_missing.rb:7: warning: mismatched indentations at 'end' with 'if' at 5 Traceback (most recent call last): 6: from /usr/lib64/ruby/gems/2.5.0/gems/rake-12.3.3/lib/rake/rake_test_loader.rb:5:in `

' 5: from /usr/lib64/ruby/gems/2.5.0/gems/rake-12.3.3/lib/rake/rake_test_loader.rb:5:in `select' 4: from /usr/lib64/ruby/gems/2.5.0/gems/rake-12.3.3/lib/rake/rake_test_loader.rb:17:in `block in

' 3: from /usr/lib64/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require' 2: from /usr/lib64/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require' 1: from /home/rui/dev/ruby/playground/mist/endKeywordMissing/test/end_keyword_missing_test.rb:2:in `' /home/rui/dev/ruby/playground/mist/endKeywordMissing/test/end_keyword_missing_test.rb:2:in `require_relative': /home/rui/dev/ruby/playground/mist/endKeywordMissing/lib/end_keyword_missing.rb:9: syntax error, unexpected end-of-input, expecting keyword_end (SyntaxError) return 0 ^ rake aborted! Command failed with status (1) Tasks: TOP => test (See full trace by running task with --trace) ```

• importMissing

Output

``` /home/rui/dev/ruby/playground/mist/importMissing/test/import_missing_test.rb:9: warning: mismatched indentations at 'end' with 'def' at 5 Run options: --seed 60902 # Running: EEEEEE Finished in 0.001216s, 4932.5231 runs/s, 0.0000 assertions/s. 1) Error: TestImportMissing#test_invalid_json: NameError: uninitialized constant TestImportMissing::JSON /home/rui/dev/ruby/playground/mist/importMissing/test/import_missing_test.rb:31:in `test_invalid_json' 2) Error: TestImportMissing#test_valid_json: NameError: uninitialized constant JSON /home/rui/dev/ruby/playground/mist/importMissing/lib/import_missing.rb:2:in `parse_json' /home/rui/dev/ruby/playground/mist/importMissing/test/import_missing_test.rb:8:in `test_valid_json' 3) Error: TestImportMissing#test_json_with_nested_objects: NameError: uninitialized constant JSON /home/rui/dev/ruby/playground/mist/importMissing/lib/import_missing.rb:2:in `parse_json' /home/rui/dev/ruby/playground/mist/importMissing/test/import_missing_test.rb:26:in `test_json_with_nested_objects' 4) Error: TestImportMissing#test_json_with_numbers: NameError: uninitialized constant JSON /home/rui/dev/ruby/playground/mist/importMissing/lib/import_missing.rb:2:in `parse_json' /home/rui/dev/ruby/playground/mist/importMissing/test/import_missing_test.rb:20:in `test_json_with_numbers' 5) Error: TestImportMissing#test_json_with_array: NameError: uninitialized constant JSON /home/rui/dev/ruby/playground/mist/importMissing/lib/import_missing.rb:2:in `parse_json' /home/rui/dev/ruby/playground/mist/importMissing/test/import_missing_test.rb:37:in `test_json_with_array' 6) Error: TestImportMissing#test_empty_json: NameError: uninitialized constant JSON /home/rui/dev/ruby/playground/mist/importMissing/lib/import_missing.rb:2:in `parse_json' /home/rui/dev/ruby/playground/mist/importMissing/test/import_missing_test.rb:14:in `test_empty_json' 6 runs, 0 assertions, 0 failures, 6 errors, 0 skips rake aborted! Command failed with status (1) Tasks: TOP => test (See full trace by running task with --trace) ```

• typeUnknown

Output

``` Run options: --seed 56356 # Running: EEE Finished in 0.001033s, 2904.8122 runs/s, 0.0000 assertions/s. 1) Error: TestTypeUnknown#test_type_is_wrong1: NameError: uninitialized constant Intt /home/rui/dev/ruby/playground/mist/typeUnknown/lib/type_unknown.rb:2:in `type_unknown' /home/rui/dev/ruby/playground/mist/typeUnknown/test/type_unknown_test.rb:8:in `test_type_is_wrong1' 2) Error: TestTypeUnknown#test_type_is_wrong3: NameError: uninitialized constant Intt /home/rui/dev/ruby/playground/mist/typeUnknown/lib/type_unknown.rb:2:in `type_unknown' /home/rui/dev/ruby/playground/mist/typeUnknown/test/type_unknown_test.rb:22:in `test_type_is_wrong3' 3) Error: TestTypeUnknown#test_type_is_wrong2: NameError: uninitialized constant Intt /home/rui/dev/ruby/playground/mist/typeUnknown/lib/type_unknown.rb:2:in `type_unknown' /home/rui/dev/ruby/playground/mist/typeUnknown/test/type_unknown_test.rb:15:in `test_type_is_wrong2' 3 runs, 0 assertions, 0 failures, 3 errors, 0 skips rake aborted! Command failed with status (1) Tasks: TOP => test (See full trace by running task with --trace) ```

• variableUnknown
  Output

/home/rui/dev/ruby/playground/mist/variableUnknown/lib/variable_unknown.rb:2: warning: possibly useless use of == in void context
Run options: --seed 59116

# Running:

EEE

Finished in 0.000951s, 3153.5992 runs/s, 0.0000 assertions/s.

  1) Error:
TestVariableUnknown#test_variable_unknown2:
NameError: undefined local variable or method `y' for #<TestVariableUnknown:0x000055b56a9accf0>
    /home/rui/dev/ruby/playground/mist/variableUnknown/lib/variable_unknown.rb:2:in `variable_unknown'
    /home/rui/dev/ruby/playground/mist/variableUnknown/test/variable_unknown_test.rb:15:in `test_variable_unknown2'

  2) Error:
TestVariableUnknown#test_variable_unknown1:
NameError: undefined local variable or method `y' for #<TestVariableUnknown:0x000055b56a9ac160>
    /home/rui/dev/ruby/playground/mist/variableUnknown/lib/variable_unknown.rb:2:in `variable_unknown'
    /home/rui/dev/ruby/playground/mist/variableUnknown/test/variable_unknown_test.rb:8:in `test_variable_unknown1'

  3) Error:
TestVariableUnknown#test_variable_unknown3:
NameError: undefined local variable or method `y' for #<TestVariableUnknown:0x000055b56a6eddc0>
    /home/rui/dev/ruby/playground/mist/variableUnknown/lib/variable_unknown.rb:2:in `variable_unknown'
    /home/rui/dev/ruby/playground/mist/variableUnknown/test/variable_unknown_test.rb:22:in `test_variable_unknown3'

3 runs, 0 assertions, 0 failures, 3 errors, 0 skips
rake aborted!
Command failed with status (1)

Tasks: TOP => test
(See full trace by running task with --trace)

Conclusion: It catches all errors, but the structure of syntax errors are different from runtime errors.

---

Adding a call.rb that calls the function defined in lib/<ruby-source-file, then running ruby call.rb and check the output

Example of call.rb

require_relative 'lib/end_keyword_missing'

end_keyword_missing(1)

• argumentsMissing

Output

``` Traceback (most recent call last): 1: from call.rb:3:in `

' /home/rui/dev/ruby/playground/mist/argumentMissing/lib/argument_missing.rb:2:in `argument_missing': undefined local variable or method `x' for main:Object (NameError) ```

• endKeywordMissing

Output

``` Traceback (most recent call last): 1: from call.rb:1:in `

' call.rb:1:in `require_relative': /home/rui/dev/ruby/playground/mist/endKeywordMissing/lib/end_keyword_missing.rb:9: syntax error, unexpected end-of-input, expecting keyword_end (SyntaxError) return 0 ```

• importMissing

Output

``` Traceback (most recent call last): 1: from call.rb:3:in `

' /home/rui/dev/ruby/playground/mist/importMissing/lib/import_missing.rb:2:in `parse_json': uninitialized constant JSON (NameError) ```

• typeUnknown

Output

``` Traceback (most recent call last): 1: from call.rb:3:in `

' /home/rui/dev/ruby/playground/mist/typeUnknown/lib/type_unknown.rb:2:in `type_unknown': uninitialized constant Intt (NameError) ```

• variableUnknown

Output

``` Traceback (most recent call last): 1: from call.rb:3:in `

' /home/rui/dev/ruby/playground/mist/variableUnknown/lib/variable_unknown.rb:2:in `variable_unknown': undefined local variable or method `y' for main:Object (NameError) ```

Conclusion: better error messages. All errors have the same structure

---

Conclusion

• Using ruby -c only checks for syntax errors. We can only catch one error within 5 packages
• Using rake test
  • We get the errors for all packages, but syntax errors have a different error structure
  • Implies running all tests
  • A little bit harder to parse
• Using the call.rb
  • All errors have the same format
  • Implies adding a file to each package calling the function with the errors
  • Easier to parse

[bauersimon]

Claude suggested rubocop, it has JSON formatted output: https://docs.rubocop.org/rubocop/formatters.html#json-formatter

[Munsio]

All PRs are merged so closing this one.