Kitware / CDash

An open source, web-based software testing server
http://www.cdash.org/
Other
215 stars 76 forks source link

CDash spawns multiple instances of apache2 after submit #750

Closed umar456 closed 6 years ago

umar456 commented 6 years ago

I have a docker container running the latest version of CDash master(0b369f51158ec05c27760f3ca62c2a4016be1019) to maintain my project.

I have noticed that the space on the server keeps running out after a week or so and the utilization has spiked for extended period time. The space runs out because of the excessive logs that CDash creates. Here is an example:

cdash_1  | 127.0.0.1 - - [28/Oct/2018:06:09:33 +0000] "GET /submit.php?project=ArrayFire&FileName=sigma___HAHA___20181028-0524-Experimental___XML___Configure.xml&MD5=e2ce728f2cf4a7ce4273967a435f034c/ajax/dailyupdatescurl.php?projectid=1 HTTP/1.1" 200 640 "-" "-"
cdash_1  | 127.0.0.1 - - [28/Oct/2018:06:09:33 +0000] "GET /submit.php?project=ArrayFire&FileName=sigma___HAHA___20181028-0524-Experimental___XML___Build.xml&MD5=12143f69a331b8d7e9d0c45cd73fdab2/ajax/dailyupdatescurl.php?projectid=1 HTTP/1.1" 200 640 "-" "-"
cdash_1  | 127.0.0.1 - - [28/Oct/2018:06:09:33 +0000] "GET /submit.php?project=ArrayFire&FileName=sigma___HAHA___20181028-0524-Experimental___XML___Test.xml&MD5=19d9b693d652702f4d542c933967deec/ajax/dailyupdatescurl.php?projectid=1 HTTP/1.1" 200 640 "-" "-"
cdash_1  | 127.0.0.1 - - [28/Oct/2018:06:09:33 +0000] "GET /submit.php?project=ArrayFire&FileName=sigma___HAHA___20181028-0524-Experimental___XML___Configure.xml&MD5=e2ce728f2cf4a7ce4273967a435f034c/ajax/dailyupdatescurl.php?projectid=1 HTTP/1.1" 200 640 "-" "-"
cdash_1  | 127.0.0.1 - - [28/Oct/2018:06:09:33 +0000] "GET /submit.php?project=ArrayFire&FileName=sigma___HAHA___20181028-0524-Experimental___XML___Build.xml&MD5=12143f69a331b8d7e9d0c45cd73fdab2/ajax/dailyupdatescurl.php?projectid=1 HTTP/1.1" 200 640 "-" "-"
cdash_1  | 127.0.0.1 - - [28/Oct/2018:06:09:34 +0000] "GET /submit.php?project=ArrayFire&FileName=sigma___HAHA___20181028-0524-Experimental___XML___Test.xml&MD5=19d9b693d652702f4d542c933967deec/ajax/dailyupdatescurl.php?projectid=1 HTTP/1.1" 200 640 "-" "-"
cdash_1  | 127.0.0.1 - - [28/Oct/2018:06:09:34 +0000] "GET /submit.php?project=ArrayFire&FileName=sigma___HAHA___20181028-0524-Experimental___XML___Configure.xml&MD5=e2ce728f2cf4a7ce4273967a435f034c/ajax/dailyupdatescurl.php?projectid=1 HTTP/1.1" 200 640 "-" "-"

Notice how the logs are repeating. This does not stop until the system runs out of space. The only workaround is to destroy and restart the container. This seems to happen every time I submit a build to CDash. I have recreated this on a clean container on my local machine.

CTest also displays the following error:

Submit files (using http)
   Send to track: Experimental
   Using HTTP submit method
   Drop site:http://localhost/submit.php?project=ArrayFire
   Upload file: /home/umar/devel/arrayfire/build3/Testing/20181028-0524/Build.xml to http://localhost/submit.php?project=ArrayFire&FileName=sigma___HAHA___20181028-0524-Experimental___XML___Build.xml&MD5=12143f69a331b8d7e9d0c45cd73fdab2 Size: 1036
   Uploaded: /home/umar/devel/arrayfire/build3/Testing/20181028-0524/Build.xml
   Upload file: /home/umar/devel/arrayfire/build3/Testing/20181028-0524/Configure.xml to http://localhost/submit.php?project=ArrayFire&FileName=sigma___HAHA___20181028-0524-Experimental___XML___Configure.xml&MD5=e2ce728f2cf4a7ce4273967a435f034c Size: 1470
   Uploaded: /home/umar/devel/arrayfire/build3/Testing/20181028-0524/Configure.xml
   Upload file: /home/umar/devel/arrayfire/build3/Testing/20181028-0524/Test.xml to http://localhost/submit.php?project=ArrayFire&FileName=sigma___HAHA___20181028-0524-Experimental___XML___Test.xml&MD5=19d9b693d652702f4d542c933967deec Size: 49871
   Uploaded: /home/umar/devel/arrayfire/build3/Testing/20181028-0524/Test.xml
   Using HTTP trigger method
   Trigger site: 
   Trigger url: ?xmlfile=sigma___HAHA___20181028-0524-Experimental___XML___Build.xml
   Error when triggering: ?xmlfile=sigma___HAHA___20181028-0524-Experimental___XML___Build.xml
   Error message was: <url> malformed
   Problems when triggering via HTTP
Errors while running CTest

I can send you additional information if necessary.

zackgalbreath commented 6 years ago

This part looks suspicious:

Using HTTP trigger method Trigger site: Trigger url: ?xmlfile=sigmaHAHA20181028-0524-ExperimentalXMLBuild.xml Error when triggering: ?xmlfile=sigmaHAHA20181028-0524-ExperimentalXMLBuild.xml

I presume you're using a CTest driver script that calls ctest_submit() to upload results to CDash? If so, you probably need to modify the CTestConfig.cmake file in your source directory. Here's an example of what mine looks like when I'm submitting to my toy Docker instance:

set(CTEST_PROJECT_NAME "TestProject")
set(CTEST_NIGHTLY_START_TIME "01:00 UTC")
set(CTEST_DROP_METHOD "http")
set(CTEST_DROP_SITE "localhost:8080")
set(CTEST_DROP_LOCATION "/submit.php?project=TestProject")
set(CTEST_DROP_SITE_CDASH TRUE)
umar456 commented 6 years ago

For this example I used ctest to upload the results. My main CTestConfig.cmake looks like this:

set(CTEST_PROJECT_NAME "ArrayFire")
set(CTEST_NIGHTLY_START_TIME "01:00:00 UTC")

set(CTEST_DROP_METHOD "http")
set(CTEST_DROP_SITE "67.207.87.39")
set(CTEST_DROP_LOCATION "/submit.php?project=ArrayFire")
set(CTEST_DROP_SITE_CDASH TRUE)

Of course I changed the drop site to localhost before I ran the ctest command. I am getting the issue using both the CTest scripts and the ctest command.

umar456 commented 6 years ago

I just tried to submit another result to cdash on my local instance and I did not see the error on the client side but I still see the same issue in my original post. Here is the output of the ctest command:

ctest -VV -D Experimental -j24 -R basic_cpu
...
   globbing for coverage in: /home/umar/devel/arrayfire/build3/test/gtest/googletest/CMakeFiles/rebuild_cache.dir
   globbing for coverage in: /home/umar/devel/arrayfire/build3/test/gtest/googletest/CMakeFiles/test.dir
 Cannot find any GCov coverage files.
 Not a valid Intel Coverage command.
 Cannot find any Python Trace.py coverage files.
 Cannot find Cobertura XML file: /home/umar/devel/arrayfire/build3/coverage.xml
 Cannot find GTM coverage file: /home/umar/devel/arrayfire/build3/gtm_coverage.mcov
 Cannot find Cache coverage file: /home/umar/devel/arrayfire/build3/cache_coverage.cmcov
 Cannot find Jacoco coverage files: /home/umar/devel/arrayfire/*jacoco.xml
 Cannot find BlanketJS coverage files: /home/umar/devel/arrayfire/*.json
 Cannot find Delphi coverage files: /home/umar/devel/arrayfire/build3/*(*.pas).html
 Cannot find any coverage files. Ignoring Coverage request.
UpdateCTestConfiguration  from :/home/umar/devel/arrayfire/build3/DartConfiguration.tcl
Parse Config file:/home/umar/devel/arrayfire/build3/DartConfiguration.tcl
Submit files (using http)
   Using HTTP submit method
   Drop site:http://localhost/submit.php?project=ArrayFire
   Upload file: /home/umar/devel/arrayfire/build3/Testing/20181030-1704/Build.xml to http://localhost/submit.php?project=ArrayFire&FileName=sigma___HAHA___20181030-1704-Experimental___XML___Build.xml&MD5=f50918184e946174f82862b964798436 Size: 1036
   Uploaded: /home/umar/devel/arrayfire/build3/Testing/20181030-1704/Build.xml
   Upload file: /home/umar/devel/arrayfire/build3/Testing/20181030-1704/Configure.xml to http://localhost/submit.php?project=ArrayFire&FileName=sigma___HAHA___20181030-1704-Experimental___XML___Configure.xml&MD5=bc5e25ffbf974da8e21241688167f750 Size: 1470
   Uploaded: /home/umar/devel/arrayfire/build3/Testing/20181030-1704/Configure.xml
   Upload file: /home/umar/devel/arrayfire/build3/Testing/20181030-1704/Test.xml to http://localhost/submit.php?project=ArrayFire&FileName=sigma___HAHA___20181030-1704-Experimental___XML___Test.xml&MD5=ac53b2b02f855bd9d3d5306b72e58ec6 Size: 2255
   Uploaded: /home/umar/devel/arrayfire/build3/Testing/20181030-1704/Test.xml
   Submission successful
zackgalbreath commented 6 years ago

"GET /submit.php?project=ArrayFire&FileName=sigmaHAHA20181028-0524-ExperimentalXMLConfigure.xml&MD5=e2ce728f2cf4a7ce4273967a435f034c/ajax/dailyupdatescurl.php?projectid=1"

I see now. This is a bug with get_server_URI for when $CDASH_BASE_URL is not set. Instead of /ajax/dailyupdatescurl.php we're accessing /submit.php with a bunch of random junk in the query string. This ultimately results in another request back to submit.php, which results in the full Apache logs you're experiencing.

Thanks for reporting this, I'll work on a fix shortly.

umar456 commented 6 years ago

Setting the $CDASH_BASE_URL fixes the problem. Thank you for looking into this!

zackgalbreath commented 6 years ago

Fixed in #758