Add test case support for Java & Haskell

benjaminkostiuk commented 3 years ago

Overview

On release of version 1.0.0 Puff has plans to release with support for 3 languages: Python, Java & Haskell. I've already implemented a mvp version of a python worker to run test cases.

Breakdown

Implement the logic + Dockerfiles to run test cases in Java & Haskell

Languages

For Java: Idea was to support test cases written in JUnit5. Possible roadblocks:

Wrapping the user code in braces correctly could prove tricky.
Handling compilation + execution and returning the correct error code. (Additional code may need to be checked on compilation failure)

For Haskell: Idea was to support test cases written using HUnit. (I have no idea how it works, but it seems simple) Possible roadblocks:

How to accept user input and the formatting for building out the cases.
Building a Docker image for Haskell. (+ Handling compilation files like Java above)

Also would like to support QuickCheck cases if possible in future.

Outline

Update CodeService.java asyncRunTestCasesInSuite() with cases for Haskell & Java
Update CodeService.java buildTestCaseCode() with functions to build the body of the test cases for Java & Haskell tests (like buildPythonTestFunction)
Update the Dockerfiles for Java & Haskell to create images for running the test cases (I've tried to use alpine to keep things small)
Create new entry files to compiling + running the Java + Haskell code, limiting resources with ulimit, similar to python_entry.sh
Update application-local.yml in docker.build.java and docker.build.haskell with the location of new entry files and dockerfiles.
It would be great to have a sample project + test cases to be able to run for java + haskell (like the one for python)
Tests for any methods you add (After I fix the project so tests can be run).

benjaminkostiuk commented 3 years ago

@OZoneGuy

OZoneGuy commented 3 years ago

For Haskell, HUnit looks great, but I feel like if this is planned to be used by people from McMaster, we should at least make QuickCheck the default testing framework. Since it is the one people will be using in classes.

benjaminkostiuk commented 3 years ago

For Haskell, HUnit looks great, but I feel like if this is planned to be used by people from McMaster, we should at least make QuickCheck the default testing framework. Since it is the one people will be using in classes.

Yes I'm trying to cater to McMaster students but I feel like QuickCheck is not really the target of what I envisioned. You don't write quick and dirty tests using QuickCheck for your program, you have to sit down and really work through it. When you're writing your function the first time you don't write a quickcase to test it, you load in Ghci and run a couple tests. If possible (and you want to) I'd like to support both but I think HUnit and the simple input/output format is what I originally had envisoned.

OZoneGuy commented 3 years ago

At the end of the day, you are the boss and this is your idea. I will work on HUnit first, but will try to provide QiuckCheck if I can.

puffproject / test-runner