Command check bug

[Michionlion]

The command check has two issues with it -- the first, a minor one, is that an incorrect/misleading message is set as its check output, as shown below.

$ pipenv run python gatorgrader.py --nowelcome --json --command "true" --count 0
{"check": "The  in  has at least 0 line(s)", "outcome": true, "diagnostic": ""}

Second, the --executes flag does not work. The following error output happens, no matter the command run.

$ pipenv run python gatorgrader.py --nowelcome --command "true" --executes
Traceback (most recent call last):
  File "gatorgrader.py", line 10, in <module>
    exit_code = orchestrate.check(sys.argv[1:])
  File "/home/m/mahlauheinerts/projects/gatorgrader/gator/orchestrate.py", line 304, in check
    output_list = report.output_list(report.get_details(), OUTPUT_TYPE)
  File "/home/m/mahlauheinerts/projects/gatorgrader/gator/report.py", line 96, in output_list
    output_function(dictionary_result, created_output_list)
  File "/home/m/mahlauheinerts/projects/gatorgrader/gator/report.py", line 125, in output_text
    check = dictionary_result[CHECK]
KeyError: 'check'

[gkapfham]

Hello @Michionlion, thanks for raising this issue. I have resolved all of the problems that you outlined in this example. With that said, I should note that I don't think that, as the sample labs are currently written, it is possible to use GatorGrader correctly when it is called from the Gradle plugin. Please refer to this issue:

https://github.com/travis-ci/travis-ci/issues/8408

I think that it is mandatory to include the unset _JAVA_OPTIONS in before_install section of a Travis configuration file. If you take that step and use the newest version of GatorGrader, then I think that these types of checks will work correctly.

This was a complex issue to resolve. Part of the problem with this issue was related to the implementation of GatorGrader. But, part of it is also related to how Travis runs Java programs. With that said, I am going to close it now. If you notice further problems, please comment here or raise a new issue. Thanks!

[gkapfham]

Also, @Michionlion and @jjumadinova, I would like to point out that I have added, as part of the resolution to this issue, code that fully supports the --executes flag. This means that it should be the case that --executes works and, additionally, all of the problems are resolved with, for instance, pipenv run python gatorgrader.py --nowelcome --json --command "true" --count 0.

For the sake of posterity (and because it is not yet documented in the sample labs), here is my current GatorGrader configuration for a laboratory assignment in CMPSC 100:

---
name: cmpsc-111-fall-2018-lab1
break: true
indent: 4
---
# --> check the source code for various characteristics
# note that without an "--exact" the check is an "at least"
src/main:
    java:
        labone/DisplayOutput.java:
            --single 2 --language Java
            --multi 2 --language Java
            --fragment "println(" --count 4 --exact
            --fragment "new Date(" --count 1 --exact
# --> check the technical writing
# mdl is a Markdown linting tool
# proselint checks writing for common mistakes
writing/reflection.md:
    mdl
    proselint
    --paragraphs 1 --exact
    --words 200
# --> check the number of commits beyond a threshold
--commits 10
# --> check that the program executes and produces 4 lines of output
--command "gradle -q --console plain run" --executes
--command "gradle -q --console plain run" --count 4 --exact

For the sake of posterity (and because it is not yet documented in the sample labs), here is my current Travis configuration for a laboratory assignment in CMPSC 100:

# use Java and non-root
dist: trusty
sudo: false
language: Java
jdk: oraclejdk8

# ignore the virtualenv that Travis creates
env:
  global:
    - PIPENV_VENV_IN_PROJECT=1
    - PIPENV_IGNORE_VIRTUALENVS=1

# install the Markdown linting program called mdl
before_install:
  - gem install mdl
  # download gradle as a zip file
  - wget https://services.gradle.org/distributions/gradle-4.6-bin.zip
  - unzip -d $HOME gradle-4.6-bin.zip
  # delete the downloaded file
  - rm -rf gradle-4.6-bin.zip
  # add gradle bin to path at the beginning to ensure it overwrites old gradle
  - export PATH="$HOME/gradle-4.6/bin:$PATH"
  # switch to Python 3.6.3 globally, install python dependencies
  - pyenv global 3.6.3
  - pip install --upgrade pip
  - pip install proselint
  - pip install pipenv
  # do not allow Travis to use standard error for debugging purposes
  - unset _JAVA_OPTIONS

# cache to improve build speed
before_cache:
  - rm -f  $HOME/.gradle/caches/modules-2/modules-2.lock
  - rm -fr $HOME/.gradle/caches/*/plugin-resolution/
cache:
  directories:
    - $HOME/.pip-cache/
    - $HOME/.gradle/caches/
    - $HOME/.gradle/wrapper/

# delete any cached version of GatorGrader and require a fresh installation
before_script:
  - rm -rf $HOME/.local/share/gatorgrader

# run checks on the README file with mdl and proselint
# run checks on a student's Java code through gradle and GatorGrader
# run checks on a student's Markdown writing through GatorGrader
# note: "gradle grade" will install GatorGrader and run all checks
script:
  # lint the assignment's documentation
  - mdl README.md
  - proselint README.md
  # Java: see build.gradle
  - gradle --version
  - gradle clean
  - gradle check
  - gradle build
  - gradle run
  # GatorGrader: see config/gatorgrader.yml
  - gradle grade