Build error when creating a new exercise

Kooper16 commented 4 months ago

Describe the bug

When creating an exercise, Artemis tries to validate the template and solution. However it sometimes has a build error when trying to execute the build script. Restarting the process fixes the issue. The error appeared inconsistent at first but we encounter it more and more often now.

To Reproduce

Go to 'Exercises' in 'Manage' view.
Click on 'Create Programming Exercise'
Create any new C Programming exercise.
Clone all repositories associated with that new exercise.
Edit files locally.
Commit and Push changes back to Artemis
Refresh Page of the new exercise.
'Solution Result' failed. (Output below) (9. Clicking on 'Resubmit last submission' fixes the issue. The script now finishes without any errors.)

Expected behavior

The build process to finish on its first attempt.

Screenshots

No response

Which version of Artemis are you seeing the problem on?

7.4.4

What browsers are you seeing the problem on?

Safari

Additional context

Our instructor uses a make file to automatically clone the repositories, edit the files and then commit + push them. The command he uses to execute the script is "make push (which first clones the repositories. Then adds his changes to the repositories. Then it commits and pushes those changes.)

Make file:

include ../../../gmsl/gmsl

## Config
Author := AUTHOR_NAME
Username := USERNAME
Exercise := SHORTNAME_OF_EXERCISE
Course := COURSE
Host := HOST

## Derived Values
URL-Base := https://$(Username)@$(Host)
Dir-Template := $(call lc, $(Course)$(Exercise))-exercise
Dir-Solution := $(call lc, $(Course)$(Exercise))-solution
Dir-Tests := $(call lc, $(Course)$(Exercise))-tests
URL-Template := $(URL-Base)/git/$(call uc, $(Course)$(Exercise))/$(Dir-Template).git
URL-Solution := $(URL-Base)/git/$(call uc, $(Course)$(Exercise))/$(Dir-Solution).git
URL-Tests := $(URL-Base)/git/$(call uc, $(Course)$(Exercise))/$(Dir-Tests).git

## Tests
Start-Instr := start> matching=\"exact\" show_input=False show_expected=True rstrip=True printable_ascii=True
End-Instr := end> 0 \n
Std-Tests := $(wildcard std-tests/*.txt)
Ext-Tests := $(wildcard ext-tests/*.txt)

define expandTest
$(Start-Instr)$(shell cat $1)\n$(End-Instr)\n\n
endef

A := $(foreach f, $(Tests), 1)
disc:
    @echo "Author: $(Author)"
    @echo "Exercise: $(Exercise)"
    @echo "Template URL: $(URL-Template)"
    @echo "Solution URL: $(URL-Solution)"
    @echo "Tests URL: $(URL-Tests)"
    @echo "Tests: $(Std-Tests) $(Ext-Tests)"

clone:
    rm -rf $(Dir-Template) $(Dir-Solution) $(Dir-Tests)
    git clone $(URL-Template)
    git clone $(URL-Solution)
    git clone $(URL-Tests)

tests/exercise_io.txt: $(wildcard std-tests/*.txt) $(wildcard ext-tests/*.txt) Makefile
    mkdir -p tests
    rm -rf tests/exercise_io.txt
    $(foreach f, $(Std-Tests) $(Ext-Tests), echo "$(Start-Instr)" >> tests/exercise_io.txt; cat $f >> tests/exercise_io.txt; echo "$(End-Instr)\n" >> tests/exercise_io.txt;)

readme.md: readme.in $(wildcard std-tests/*.txt) Makefile
    rm -rf readme.md
    cat readme.in >> readme.md
    $(foreach f, $(Std-Tests), echo "\`\`\`" >> readme.md; cat $f >> readme.md; echo "\`\`\`\n" >> readme.md;)

copy: clone $(wildcard template/*) $(wildcard solution/*) $(wildcard tests/*) tests/exercise_io.txt readme.md
    cp readme.md $(Dir-Template)/
    cp -r template/* $(Dir-Template)/
    cp -r solution/* $(Dir-Solution)/
    cp -r tests/* $(Dir-Tests)/

push: copy
    cd $(Dir-Template); git add *; git commit -m "setup" -a; git push
    cd $(Dir-Solution); git add *; git commit -m "setup" -a; git push
    cd $(Dir-Tests); git add *; git commit -m "setup" -a; git push

clean:
    rm -rf tests/exercise_io.txt
    rm -rf readme.md
    rm -rf $(Dir-Template) $(Dir-Solution) $(Dir-Tests)
    rm -rf *~

Relevant log output

25.07.2024 08:32:17
~~~~~~~~~~~~~~~~~~~~ Start Build Job 1151721889137390 ~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~ Inspecting docker image sharingcodeability/fact:latest ~~~~~~~~~~~~~~~~~~~~
25.07.2024 08:32:17
~~~~~~~~~~~~~~~~~~~~ Started container local-ci-1151721889137390 for build job 1151721889137390 ~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~ Populating build job container with repositories and build script ~~~~~~~~~~~~~~~~~~~~
25.07.2024 08:32:17
~~~~~~~~~~~~~~~~~~~~ Executing Build Script for Build job 1151721889137390 ~~~~~~~~~~~~~~~~~~~~
25.07.2024 08:32:17
⚙️ executing setup_the_build_environment
25.07.2024 08:32:17
chown: cannot access 'assignment/': Not a directory
25.07.2024 08:32:18
⚙️ executing build_and_run_all_tests
25.07.2024 08:32:18
cp: cannot stat 'tests/Makefile': Not a directory
25.07.2024 08:32:18
~~~~~~~~~~~~~~~~~~~~ Finished Executing Build Script for Build job 1151721889137390 ~~~~~~~~~~~~~~~~~~~~
25.07.2024 08:32:18
Building and testing submission for repository dap1ws2425w4e0-solution and commit hash d031df5ada0e0d10107bef8387c65777ea6df51f took 545,92ms

FloezeTv commented 1 month ago

We also encounter various issues about files not being found, directories not existing, the runner failing to clone or delete the repository more or less randomly after having started tests for about 500 submissions during the correction of our exam.

Some (anonymized) error messages from the build system include:

chmod: cannot access 'tests/run.sh': Not a directory (where tests/run.sh is starting our tests, called from the build script
Cannot lock checked-out-repos/<some_id>/<exercise_name>/<exercise_name>-tests/.git/index. Ensure that no other process has an open file handle on the lock file checked-out-repos/<some_id>/<exercise_name>/<exercise_name>-tests/.git/index.lock, then you may delete the lock file and retry.
Error while executing build job <id>: de.tum.cit.aet.artemis.core.exception.LocalCIException: Error while deleting repository

Some error messages from our (OCaml) tests trying to access files include:

Unix.Unix_error(Unix.ENOENT, "realpath", "./solution/src")
Sys_error("./assignment: Not a directory")

I don't know why exactly these errors happen (somehow the filesystem is inconsistent?), but I don't think this is an error on our side, seeing as these failures seem random and most runs succeed.

michael-schwarz commented 1 month ago

Additional context: This is a blocker for us for releasing the FPV exam grades of the retake.

ls1intum / Artemis