Test all languages and platforms

neomatrix369 commented 5 years ago

script to generate and test run all languages for Linux and MacOS
currently tested on MacOS and Linux

Usage:

./generate_and_test_run_all.sh .     <== detects platform and runs the generate and tests, logs generated in the logs folder

TARGET_PLATFORM=linux ./generate_and_test_run_all.sh .     <==  runs the generate and tests for Linux, logs generated in the logs folder

TARGET_LANGUAGES="java scala" ./generate_and_test_run_all.sh .     <==  runs the generate and tests for detected platform and iterates through languages java and scala, logs generated in the logs folder

Logs (examples)

logs/tdl-runner-macos-java-generate.logs
logs/tdl-runner-macos-java-test-run.logs
logs/tdl-runner-linux-java-generate.logs
logs/tdl-runner-linux-java-test-run.logs

As the names of the logs say, they contain outputs from the different steps for the respective language and platform

The script, in theory, should work in Windows if run in git-bash or cygwin although haven't been tested

Once this script when done should give us good headstart for the batch script (Windows version).

neomatrix369 commented 5 years ago

MacOS tests

~~~ Summary of test executions ~~~
  ~~~ Passed Tests ~~~
  platform=macos|language=java
  platform=macos|language=scala
  platform=macos|language=python
  platform=macos|language=ruby
  platform=macos|language=csharp
  platform=macos|language=fsharp
  platform=macos|language=vbnet
  7 test(s) passed

  ~~~ Failed Tests ~~~
  0 test(s) failed

You can easily use these results when verifying for valid builds during your CI/CD pipeline execution.

neomatrix369 commented 5 years ago

@julianghionoiu Let me know if within each step we want to test further? We could use the config credentials and do the real test but it will have its own dependencies i.e. the credential and also aborting of the app once its start.

By real test I mean doing what the end-user would do - it possible to do it but is it needed?

neomatrix369 commented 5 years ago

Linux tests

~~~ Summary of test executions ~~~
  ~~~ Passed Tests ~~~
  platform=linux|language=java
  platform=linux|language=scala
  platform=linux|language=python
  platform=linux|language=ruby
  platform=linux|language=csharp
  platform=linux|language=fsharp
  platform=linux|language=vbnet
  7 test(s) passed

  ~~~ Failed Tests ~~~
  0 test(s) failed

neomatrix369 commented 5 years ago

Windows tests under `Cgywin` and `gitbash`

~~~ Summary of test executions ~~~
~~~ Passed Tests ~~~
0 test(s) passed
~~~ Failed Tests ~~~
platform=windows\|language=java
platform=windows\|language=scala
platform=windows\|language=python
platform=windows\|language=ruby
platform=windows\|language=csharp
platform=windows\|language=fsharp
platform=windows\|language=vbnet
7 test(s) failed
--

Cgywin wasn't able to create a file in the .cache folder. git-bash couldn't find the rev command

Our next action should be to write the batch script instead but it would mean we would need a batch version for all the .sh scripts so far.

Let me know which way you would like to go with it.

julianghionoiu commented 5 years ago

Hi Mani,

I appreciate that this is automating the "self-test" which covers the recorder and the JRE.

We need to also get coverage on the runner itself and it might be a manual process at this stage. From the candidate perspective, we need to prove that using the provided bundle, a candidate can successfully connect to the challenge server. (using a credential file)

This covers:

the bundle contains all the needed bits for running a challenge
the recorder runs and the runner connects to it
when the recorder stops, the sourcecode and video are uploaded

Let's divide up the work and run through a manual check. I can test all the languages on MacOs.

neomatrix369 commented 5 years ago

Thanks for clarifying, I can do the manual parts for Linux and Windows but this script does help me along the way, it will serve me as a checklist otherwise doing these manual tests can be inconsistent.

I'll ensure I have video and source code snippets saved to verify they have been upload.

Please go ahead with your MacOS tests.

neomatrix369 commented 5 years ago

Whats the oneline command that runs some recorder jar that can check for the integrity of the video (at the moment I'm opening the video in a player and seeing it run.

Also, one line command to list the content of the .srcs file created.

Just now with the help of the script I can see the following files are created:

screencast_20190108T214137.mp4
sourcecode_20190108T214137.srcs
sourcecode_20190108T214202.srcs

Recorder is capturing video and source files. But it will make my manual tests more effective if I can have something that already runs on these files before I open them up to verify.

Logs are showing that these files are being uploaded:

INFO  [SourceRec]  - Recording stopped
INFO  [SourceRec]  - Closing the source code stream
INFO  [main]       - Stopped ServerConnector@6b8d96d9{HTTP/1.1,[http/1.1]}{localhost:41375}
INFO  [main]       - Upload remaining parts and finalise recording session
INFO  [main]       - Uploading file run_tmp/accelerate_runner/record/localstore/sourcecode_20190108T214202.srcs
INFO  [Metrics]    - Uploaded 97.1% of 1.41 MB at 1.261 MB/sec
INFO  [main]       - Finished uploading file run_tmp/accelerate_runner/record/localstore/sourcecode_20190108T214202.srcs
INFO  [main]       - Uploading file run_tmp/accelerate_runner/record/localstore/record-and-upload-20190108T214215.log
INFO  [main]       - Finished uploading file run_tmp/accelerate_runner/record/localstore/record-and-upload-20190108T214215.log
INFO  [main]       - Uploading file run_tmp/accelerate_runner/record/localstore/record-and-upload-20190108T214201.log
INFO  [main]       - Finished uploading file run_tmp/accelerate_runner/record/localstore/record-and-upload-20190108T214201.log
INFO  [main]       - ~~~~~~ Stopped ~~~~~~

neomatrix369 commented 5 years ago

The script is also helping me gather all these logs, video and source files in a directory structure:

linux/java
linux/python ...

I can upload it to you so you can see you have received them on your end in the s3 bucket - if needed.

neomatrix369 commented 5 years ago

Results of my Linux test:

test-results/linux/csharp:
4.0K Jan  8 22:24 .
4.0K Jan  8 22:26 ..
 19K Jan  8 22:24 record-and-upload-20190108T221718.log
1.9K Jan  8 22:24 record-and-upload-20190108T222216.log
6.7K Jan  8 22:24 record-and-upload-20190108T222235.log
1.1K Jan  8 22:24 record-and-upload-20190108T222407.log
 15M Jan  8 22:24 screencast_20190108T221719.mp4
1.4M Jan  8 22:24 sourcecode_20190108T221719.srcs
1.4M Jan  8 22:24 sourcecode_20190108T222235.srcs

test-results/linux/fsharp:
total 5.6M
4.0K Jan  8 22:25 .
4.0K Jan  8 22:26 ..
4.5K Jan  8 22:25 record-and-upload-20190108T222425.log
1.4K Jan  8 22:25 record-and-upload-20190108T222518.log
2.0K Jan  8 22:25 record-and-upload-20190108T222526.log
1.1K Jan  8 22:25 record-and-upload-20190108T222533.log
2.8M Jan  8 22:25 screencast_20190108T222425.mp4
1.4M Jan  8 22:25 sourcecode_20190108T222425.srcs
1.4M Jan  8 22:25 sourcecode_20190108T222526.srcs

test-results/linux/java:
4.0K Jan  8 22:02 .
4.0K Jan  8 22:26 ..
 14K Jan  8 22:02 record-and-upload-20190108T215733.log
1.5K Jan  8 22:02 record-and-upload-20190108T220059.log
7.1K Jan  8 22:02 record-and-upload-20190108T220115.log
1.1K Jan  8 22:02 record-and-upload-20190108T220255.log
3.9M Jan  8 22:02 screencast_20190108T215734.mp4
1.5M Jan  8 22:02 sourcecode_20190108T215734.srcs
1.5M Jan  8 22:02 sourcecode_20190108T220116.srcs

test-results/linux/python:
4.0K Jan  8 22:06 .
4.0K Jan  8 22:26 ..
3.8K Jan  8 22:06 record-and-upload-20190108T220429.log
1.4K Jan  8 22:06 record-and-upload-20190108T220508.log
5.4K Jan  8 22:06 record-and-upload-20190108T220520.log
1.1K Jan  8 22:06 record-and-upload-20190108T220628.log
987K Jan  8 22:06 screencast_20190108T220430.mp4
1.4M Jan  8 22:06 sourcecode_20190108T220430.srcs
1.4M Jan  8 22:06 sourcecode_20190108T220521.srcs

test-results/linux/ruby:
4.0K Jan  8 22:16 .
4.0K Jan  8 22:26 ..
7.3K Jan  8 22:16 record-and-upload-20190108T221438.log
1.4K Jan  8 22:16 record-and-upload-20190108T221617.log
2.7K Jan  8 22:16 record-and-upload-20190108T221626.log
1.1K Jan  8 22:16 record-and-upload-20190108T221641.log
3.9M Jan  8 22:16 screencast_20190108T221439.mp4
1.4M Jan  8 22:16 sourcecode_20190108T221439.srcs
1.4M Jan  8 22:16 sourcecode_20190108T221627.srcs

test-results/linux/scala:
4.0K Jan  8 22:14 .
4.0K Jan  8 22:26 ..
 12K Jan  8 22:14 record-and-upload-20190108T220958.log
1.6K Jan  8 22:14 record-and-upload-20190108T221255.log
5.5K Jan  8 22:14 record-and-upload-20190108T221304.log
1.1K Jan  8 22:14 record-and-upload-20190108T221416.log
6.1M Jan  8 22:14 screencast_20190108T220958.mp4
1.4M Jan  8 22:14 sourcecode_20190108T220958.srcs
1.4M Jan  8 22:14 sourcecode_20190108T221305.srcs

test-results/linux/vbnet:
4.0K Jan  8 22:26 .
4.0K Jan  8 22:26 ..
4.2K Jan  8 22:26 record-and-upload-20190108T222556.log
1.4K Jan  8 22:26 record-and-upload-20190108T222644.log
1.9K Jan  8 22:26 record-and-upload-20190108T222650.log
1.1K Jan  8 22:26 record-and-upload-20190108T222655.log
2.6M Jan  8 22:26 screencast_20190108T222557.mp4
1.4M Jan  8 22:26 sourcecode_20190108T222557.srcs
1.4M Jan  8 22:26 sourcecode_20190108T222651.srcs

Can you please check if you have received he respective video and source files on your end?

FYI by the time I receached fsharp and vbnet the challenges were completed, but you can see the videos of the activity on the screen and the small file changes it must have uploaded.

neomatrix369 commented 5 years ago

Will perform the Windows test tomorrow, let me know if you are satisfied with the Linux test - I will add some more comments to the script so you know the steps taken when doing the test

neomatrix369 commented 5 years ago

I think this manual testing can be made a bit more efficient but for our current iteration it may not be necessary

neomatrix369 commented 5 years ago

One feedback I would like to give, I noticed while testing the new generated bundles:

the challenges folder was usually empty, on starting the challenge it would still remain empty, unless I invoked the deploy command, got a penalty and then the challenge description would be downloaded
the CLI interface emulated where we can type start, deploy or pause can be a bit jittery for java or scala or python, gotta always remember to run it after I have confirmed my changes and mentally ready to deploy

Let me know if you notice something similar when performing your tests on the MacOS. I'll look out for them on Windows.

julianghionoiu commented 5 years ago

Hi Mani, No need to upload. Would you also be able to show me the sizes of the generated files? I am concerned that the sourcecode capture might be capturing binary files.

neomatrix369 commented 5 years ago

@julianghionoiu Just updated the list above with file sizes - I hope this confirms your guess

julianghionoiu / tdl-lord-of-runners