search

codewars / codewars.com

Issue tracker for Codewars
https://www.codewars.com
BSD 2-Clause "Simplified" License
2.09k stars 218 forks source link

About the New Runner #1336

Closed kazk closed 5 years ago

kazk commented 6 years ago

I'm planning to release the new runner for Codewars during this week (maybe early next week depending on how things goes) and wanted to let Codewars users know about the changes. The preview site uses it already, so some of the features can be tested there for now.

Background

Codewars and Qualified uses codewars-runner-cli to execute submitted code. The project started more than 4 years ago and requirements have changed significantly over the years.

We needed a drop-in replacement that's more flexible. So I've refactored the runner and the server, then rebuilt the infrastructure.

We're going to eventually open source parts of this (Dockerfile, test frameworks, etc) after some planning and cleaning up.

Changes

This release focused mainly on backwards compatibility with codewars-runner-cli and the goal was to build a more flexible system that we can easily change depending on the requirements.

Note that some of the new features may not be available on Codewars until the frontend is updated.

New Features

Enhancements

Bug Fixes

Deprecations

Compatibility

Tested against all published kata (including beta) and it's currently compatible with ~98% of the contents. I'll create a wiki page (will post a comment when I do) listing affected kata so power users can help fixing them. Most of them should be fixed by adjusting few things. However, those kata that uses things that we never officially supported is not considered and needs major changes. This includes Shell kata written in Perl or Python.

Stats

Voileexperiments commented 6 years ago

this number includes some noise like author's solution failing random tests

Well, in this case it's still an issue (even if it's unrelated to this topic), so it'd be helpful if you list them too (probably in #1315).

kazk commented 6 years ago

Started List of Affected Kata. Just listing Python kata affected by avoiding concatenation for now.

I'll update it again later to include other issues.

DonaldKellett commented 6 years ago

Glad to hear that the new runner is finally out on preview, I can barely contain my excitement! :smile: One thing though - am I allowed to author Fortran kata/translations ATM? If authoring them will interfere with the main site negatively them I'm going to stick to plain Kumite until further notice, cheers :smiley:

Voileexperiments commented 6 years ago

@DonaldKellett well, we did have written bf katas (and got them approved I think?) while it was in beta, so...

kazk commented 6 years ago

@DonaldKellett I don't think that's possible yet because the frontend doesn't know about it (you can't select Fortran). I wanted you to at least try some Kumite so I opened a PR to add Fortran support last week, but I'm not sure when it'll be merged/deployed because Jake has been busy with other tasks.

Bubbler-4 commented 6 years ago

Fixed all of my katas, and will start fixing others in Python as soon as I get some spare time. Looks like many have problems in common (such as Linked List series or CIS series, though I really think CIS ones should be just retired asap), so it won't take too much time to fix most of them, though re-publishing a kata often times out these days unfortunately.

kazk commented 6 years ago

Kata with issues will show a warning message like the following on the preview site. image

Once someone completes or republish successfully, this message will go away.

Bubbler-4 commented 6 years ago

Just for reference, the "Caution" message is removed only if:

Also, some katas in the affected list don't have issues raised (so not open for edit right now).

kazk commented 6 years ago

Just an update, we've decided to release on this Wednesday.

Sorry for the delay. I had some issues finalizing the production setup :sweat_smile: We'll make new languages and language versions available on Codwars soon if everything goes well.

kazk commented 6 years ago

I forgot to mention, but testing Scala will be significantly faster soon.

The new runner is able to run ScalaTest without JUnitRunner (avoiding gradle test). Using Gradle allowed adding Groovy/Kotlin/Scala all at once, but it wasn't a good choice for our use case.

So minimum kata takes (includes compiling):

Because Codewars only supports one test framework, existing 25 Scala kata needs to be changed. If tests are using ScalaTest, migration will be just removing lines that references JUnit 4. I hope it's not a huge pain migrating those written with JUnit 4. This won't be included in tomorrow's launch, but I'd like to switch soon because it'll help reducing the server load.

I'll look into doing similar for Kotlin and Groovy.

kazk commented 6 years ago

I've released the new runner! Please let me know if you find anything weird.

kazk commented 6 years ago

For Red users, there're some features that I didn't have the time to add at the moment. Like real time output.

kazk commented 6 years ago

Also, the time displayed at the top of the test results will be different from before. It includes all the time it took to execute the code, including some overhead and compiling. For most interpreted languages, the difference is few hundred milliseconds. Some compiled languages will be more different because now it includes the time it took to compile.

I'll work on improving this after other higher priority tasks.

kazk commented 6 years ago

Looks like there's some weird issue with Python encoding.

Python 2 doesn't allow non-ASCII characters unless the encoding is declared at the top (# -*- coding: utf-8 -*-), but there're some Python 2 only kata with non-ASCII characters :/ I don't know how those got published with Python 2 or if it was working on the old runner. The new runner supports the magic comment, but the old one didn't because of concatenation. I don't know if python -c "code" (the old runner did this for unknown reason) handles the code differently.

Blind4Basics commented 6 years ago

Hi,

It seems that this one has been broken by the new python runner (I assume, but not confirmed: I didn't do it, so I cannot look at the edit panel): https://www.codewars.com/kata/58635f1b2489549be50003f1/discuss#5adfa6410774db3be9000056

Cheers

kazk commented 6 years ago

@Blind4Basics See List of Affected Kata. Many power users have been helping to fix them.

Blind4Basics commented 6 years ago

Looking at it. Unfortunately, most of those are katas I didn't do yet, apparently. I opened a bunch of issues to get access (note: for the nonogram, I published a fork with what's needed)

kazk commented 6 years ago

I opened a bunch of issues to get access

Wait, so a kata needs to have some issues open or it's not editable? I didn't know that... I guess that makes sense to prevent abuse.

Thanks for your help!

Voileexperiments commented 6 years ago

A kata is only open for editing if it has any issues at least 1 week old.

gbdj commented 6 years ago

Previously tested Haskell solution for TinyThreePassCompiler kata failing now with errors:

Could not find module `Text.Parsec'
Could not find module `Text.Parsec.String'
Could not find module `Text.Parsec.Expr'
Could not find module `Text.Parsec.Token'
Could not find module `Text.Parsec.Language'

May be parsec library was forgotten after Code Execution Engine upgrade?

kazk commented 6 years ago

@gbdj thanks for letting me know. I'll look into it.

kazk commented 6 years ago

I'm not sure why the old image had parsec installed :/ The new image have all of the packages specified in the old Dockerfile:

# Install Haskell Packages
# To install additional packages use "RUN cabal install <pkg-name>"
RUN cabal install split ifelse
RUN cabal install persistent-sqlite persistent-template
RUN cabal install haskell-src-exts lens
RUN cabal install hspec hspec-core hspec-discover

https://github.com/Codewars/codewars-runner-cli/blob/d1c1d8bfb9919ee61d520370484894be27f48739/docker/haskell.docker#L30-L35

I'll just add parsec explicitly.

kazk commented 6 years ago

@gbdj I've deployed the fix. You should be able to use parsec now.

gbdj commented 6 years ago

@kazk Thanks. It works now.

10XL commented 6 years ago

May be parsec library was forgotten after Code Execution Engine upgrade?

I have a similar issue with a Ruby kata that uses nokogiri, where it seems to not be installed on Ruby 2.5

@kazk running this kumite shows that some Ruby gems from 2.3 are missing on 2.5.

kazk commented 6 years ago

@10XL Ruby 2.5 doesn't have gems installed at the moment because I want to update package versions. Kata shouldn't have 2.5 enabled if it can't be completed. If it does, then it's another bug from the auto upgrade script :(

10XL commented 6 years ago

Kata shouldn't have 2.5 enabled if it can't be completed. If it does, then it's another bug from the auto upgrade script :(

Yes, that is the case. The kata has 2.5 as unavailable/unsupported in the trainer as it should.

kazk commented 6 years ago

OK, good. I'll add gems for 2.5 next week. I can't guarantee all of the gems from 2.3 will be installed or if they're compatible. But I'll make sure to install commonly used ones like nokogiri.

Voileexperiments commented 6 years ago

@kazk Is Solidity 0.4.19 even working at all? Because I just saw this kata comment.

kazk commented 6 years ago

It works, but it's not compatible with 0.4.13. Also, you shouldn't need to require('web3') because Truffle injects it globally.

kazk commented 6 years ago

On 0.4.19 which uses Truffle 4, you don't need to set web3 provider because it starts internal server automatically.

kazk commented 6 years ago

I don't know why it has 0.4.19 enabled when it doesn't work. Probably just another bug from auto update script :(

kazk commented 6 years ago

@10XL I've added gems to Ruby 2.5. gem list also lists gems that are installed as dependency of others so don't assume all of them will continue to exist. I'll try to update the wiki soon for list of gems that we installed.

nomennescio commented 6 years ago

So where's the new runner repo? The main webpage are still pointing to the old repo, which is marked deprecated. I want to submit a bug to the NASM runner, which issue tracker to use?

kazk commented 6 years ago

@nomennescio we don't have one yet.

We're going to eventually open source parts of this (Dockerfile, test frameworks, etc) after some planning and cleaning up.

So we're still using codewars-runner-cli to track runner issues.

nomennescio commented 6 years ago

We're going to eventually open source parts of this (Dockerfile, test frameworks, etc) after some planning and cleaning up.

So we're still using codewars-runner-cli to track runner issues.

That's great! Marking the whole repository deprecated seems however a bit premature if the issues are still tracked there. Maybe comment on the mainpage that it is still used for issue tracking?

nomennescio commented 6 years ago

Duh! It does mention it on the mainpage now.

kazk commented 6 years ago

@nomennescio yeah, I changed it because I realized that it's confusing after you mentioned it :)

FArekkusu commented 4 years ago

@kazk the list of affected katas has been cleared up (except 2 katas which should be retired). With those ones resolved the wiki page could be also removed.

kazk commented 4 years ago

Thanks!