Run tests in separate thread

[ashashwat]

Topcoder runs tests in separate threads. Using globals though a bad practice is pretty common in TC contests. So we need to write extra code to clear globals.

Is it possible to have a testing template so that we don't need to clear globals. It is not a mandatory feature but a nice to have.

[vexorian]

I am not sure about the new filetest tester template that is used by default in Greed 2.0. But the old "test" template supports this by default. You just need to change the way in which you call the program.

./program 1 runs only test case 1. ./program 2 runs only test case 2. Etc.

I am sure you can come up with a script that runs the program with different arguments in sequence.

[ashashwat]

The behavior is retained in the new filetester template. This is what I do currently, using ./a.out [arg] that is.

On Thu, Dec 5, 2013 at 8:52 PM, vexorian notifications@github.com wrote:

I am not sure about the new filetest tester template that is used by default in Greed 2.0. But the old "test" template supports this by default. You just need to change the way in which you call the program.

./program 1 runs only test case 1. ./program 2 runs only test case 2. Etc.

I am sure you can come up with a script that runs the program with different arguments in sequence.

— Reply to this email directly or view it on GitHubhttps://github.com/shivawu/topcoder-greed/issues/90#issuecomment-29906306 .

[zen0wu]

I'm not sure how to do this and how the TC tests our program. Since the generated code is only a file, when you compile and run it, you have to clean all the global state for each testcase. Running in separate thread may not change this, unless you have an external tester which runs your program at the process level, this will initialize the global variables each time. However, I do think it's a good idea to clean the globals in your program, for example, the arrays in C++ is initialized to uncertain content if you don't clear them.

[vexorian]

It is possible to make a c++ template program that calls itself for each test case. But imho that is better left as an exercise for user to make with the template engine, as it has too many specifics regarding OS and other things.

[ashashwat]

Modifying template program will supposedly work. Even a script [1] will work.

However greed has jumped in popularity recently and not everybody is comfortable with playing with templates. They either go with default template or the one @vexorian linked to in his blog post.

The motivation of the bug report was due to a chat in TC arena.

I am thinking of using Testerdream once again Greed is good, but i really need it to clear global variables

Since the generated code is only a file, when you compile and run it, you have to clean all the global state for each testcase.

I'm not sure if separate threads share memory. IIRC, it was mentioned somewhere in TC forum about the TC-tester using threads. Changing the template such that you call the executable itself rather then instances or an external script running the executable with arguments seems two options to me, both of which IMHO seems hackish option. I am not even sure how threads are supposed to manage it, if there is a global and say 5 different threads are manipulating it at the same time, will it be thread safe ? Isn't the whole point of semaphores are to protect from the same issue.

, as it has too many specifics regarding OS and other things.

I don't think there is any OS specific requirement. C++, Java, Python, all of them are cross platform.

[1] https://gist.github.com/ashashwat/7837708

[vexorian]

External script is the most logical choice and it is very easy. Whoever is running greed has to eventually run the code manually anyway.

You wouldn't really use globals in Java or Python. Java and Python also have exception handling, so the other advantage for using a different process is also void in those languages. C++ is the language that needs this and c++ has the most issues with doing OS stuff in a cross platform way. I am not sure if system() works in Visual C++ compiler, I think you'd need to use winapi there.. Anyway, if you can make a template that can do this, that's fine.

As hacky as making a source file that runs itself sounds like, that's exactly what TesterDream does. In fact, it appears that the only code that is needed for current template to work is something like this:

    if (argc == 1) 
    {
        cout << "(First message)" << endl << endl;
        for (int i = 0; i < 20; i++)
        {
            ostringstream s; s << argv[0] << " " << i;
            int exitCode = system(s.str().c_str());
            if (exitCode)
                cout << "#" << i << ": Runtime Error" << endl;
        }
        int T = time(NULL)-1386418445;
        double PT = T/60.0, TT = 75.0;
        cout.setf(ios::fixed,ios::floatfield);
        cout.precision(2);
        cout << endl;
        cout << "Time  : " << T/60 << " minutes " << T%60 << " secs" << endl;
        cout << "Score : " << 950.0*(.3+(.7*TT*TT)/(10.0*PT*PT+TT*TT)) << " points" << endl;
    }

[vexorian]

As usual, github ate some of the text I pasted.

    if (argc == 1) 
    {
        (do stuff to do before running test cases)
        for (int i = 0; i < 20; i++)
        {
            ostringstream s; s << argv[0] << " " << i;
            int exitCode = system(s.str().c_str());
            if (exitCode)
                cout << "#" << i << ": Runtime Error" << endl;
        }
        int T = time(NULL)-1386418445;
        double PT = T/60.0, TT = 75.0;
        cout.setf(ios::fixed,ios::floatfield);
        cout.precision(2);
        cout << endl;
        cout << "Time  : " << T/60 << " minutes " << T%60 << " secs" << endl;
        cout << "Score : " << 950.0*(.3+(.7*TT*TT)/(10.0*PT*PT+TT*TT)) << " points" << endl;
    }

Incidentally, I had to set up TesterDream to verify this and the configuration process was just an awful experience I would not like to repeat. Go greed.

[vexorian]

Made a PR #92 no idea if this should be default behavior.

[zen0wu]

I think this is resolved with the wonderful patch from @vexorian in #93 and #92.

[ashashwat]

Thanks @vexorian for the patch.

@shivawu Since there has been lot of changes post 2.0 beta, are you planning to release Greed 2.0 RC before releasing Greed 2.0 ?