rethink constants

[mzuenni]

Right now we replace constants at the following places:

1. problem statements
2. input and output validators
3. included code
4. example submissions
5. testdata.yaml

There are a few things that I want to mention.

1. "Replacement in example submissions and included code" What is the use case here and why do we differentiate between example submissions and other submissions? (Note that I don't want to argue that we should do replacement for all submissions, rather I think that we should not touch any code at all).

2. In most cases text replacement is just inconvenient. We already restrict validators to c++ and python in both cases there are nicer ways to automagically introduce constants. For example in c++ we could add constexpr <std::string_view,long long,double> <constant name> = <constant value> and something similar in python. In latex, we could also introduce the constants as macros. I know that this makes constants more complicated to implement on the tooling side but IMO it is much more convenient for the user. (For markdown and testdata.yaml i have no better idea...)

[evouga]

The use case is stuff like

#define MAXN 100000
int dp[MAXN];

in judge solutions, which can now be

int dp[{{MAXN}}];

I agree that this is not as crucial as constant expansion in validators and problem statements, but it’s nice to be able to tweak problem bounds without needing to go through and fix RTEs in judge solutions.

Expanding constants in contestant submissions seems to me a bad idea; for one thing contestants don’t know what constants are available, and for another we should not be doing any messing around with submission source code that could accidentally break a solution that works locally for the student.

I’m not sure why text replacement is “just inconvenient”?

[RagnarGrootKoerkamp]

Text replacement badly messes up editor highlighting and autoformatting and such. Wouldn't it be nicer in your example if we did a #define MAX_N 1000 in the background? And similarly we could \newcommand{\maxn}{1000} for latex.

[evouga]

Is there concrete evidence that the proposed balanced-braces “badly” messes up anything in practice?

I think it is good to make it obvious which variables are externally defined problem constants and which are statically scoped, and to make it more or less impossible to accidentally shadow the problem constants

[mzuenni]

regarding the change in judge submissions: I am not sure if this really makes things much easier since you still need to change all the test data anyway (note that we don't define generators). And distinguishing between user submissions and jury submissions makes stuff more complicated than necessary. Further, we don't don't allow a constant to write solutions like "do random stuff until you are out of time" for which automatic updates would be useful.

regarding messing stuff up: my problem is more that this messes up standard build processes. For c++ I can't compile stuff anymore but if it's done with macros or as constant I can simply tell my compiler what macros exist. Same for latex. The curly bracket thing breaks latexmk rebuild whereas with macros this can not happen.

[Tagl]

Yeah I think defines and environment variables might be better approaches here.

[evouga]

For LaTeX, don't you already need a script that does preprocessing before being able to compile the problem statement? You need to wrap the problem statement in a document environment, define all of the problem-package-specific macros, etc.

I agree that it's annoying that C++ code cannot be compiled locally using e.g. -D{{maxn}}=100000, and that it may be quite difficult to convince my judges to use constants in the judge solutions for this reason.

So:

• we could try to come up with some language-specific way to include the constants (note that this would need to include all valid input validator languages, including stuff like CheckTestData, not just Python, TeX, and C++).
• we could remove judge submissions from the list of preprocessed files, if this is the main sticking point. I am OK with this, though I don't really see the harm---judge teams who can't be bothered to deal with problem constants during local compilation don't have to use them.

But we had long discussions already about these issues at Lund, and considered alternatives like __maxN__ and XXMAXNXX and other schemes that could be implemented as macros, and the status quo was a compromise balancing factors such as simplicity, flexibility across multiple languages, readability, and not-breaking-syntax-highlighting-too-badly. I don't think the current proposal fits everybody's workflow perfectly, but I'm also hesitant to relitigate the feature from scratch, unless there is a major undiscovered problem with the proposed approach.

[mzuenni]

For LaTeX, don't you already need a script that does preprocessing before being able to compile the problem statement?

you can make latexmk do all that. it just cant preprocess the tex file with text replacement... (especially not with live pdf generation)

we could try to come up with some language-specific way to include the constants (note that this would need to include all valid input validator languages, including stuff like CheckTestData, not just Python, TeX, and C++).

we only allow ctd, viva, c++ and python as checkers

[evouga]

we only allow ctd, viva, c++ and python as checkers

Yes, you are right. So we could specify different mechanisms for these languages (plus yaml and markdown) and remove support for constants in included code and submissions.

you can make latexmk do all that. it just cant preprocess the tex file with text replacement... (especially not with live pdf generation)

And it cannot be configured to sed the input files? (Though this is a moot point if you have a nice solution along the lines of the above).

[mzuenni]

you can make latexmk do all that. it just cant preprocess the tex file with text replacement... (especially not with live pdf generation)

And it cannot be configured to sed the input files? (Though this is a moot point if you have a nice solution along the lines of the above).

I see no good way to do that. but if we add a mechanism for c++ etc. then one for latex is also fine i.e. if you have the constant blub we define the latex macro \blub or \constant_blub or something like this.

[niemela]

1. why do we differentiate between example submissions and other submissions?

"example submissions" are the only kind of submissions in the problem packages.

Expanding constants in contestant submissions seems to me a bad idea; [...]

Yes, definitely. I don't think anybody has suggested or want that.

regarding the change in judge submissions: I am not sure if this really makes things much easier since you still need to change all the test data anyway (note that we don't define generators).

I agree that constants are significantly less needed for (judge) submissions, simply because the agreement between submissions and statement/data/validators are checked. If there is a change of a limit that breaks a submission, then installation will fail.

The reason why we wanted something like constants to begin with is that we verify that submissions agree with data/validators, but we have to trust problem authors to make sure that the statement also agrees. In fact, the most common error with problem packages is exactly this.

And distinguishing between user submissions and jury submissions makes stuff more complicated than necessary.

I don't understand what this means? There are no "user submissions" in a problem package. Clearly (?) constants are not meant to apply outside of the package, i.e. they are absolutely not meant to be applied to submissions sent to some system that has installed the package.

Further, we don't don't allow a constant to write solutions like "do random stuff until you are out of time" for which automatic updates would be useful.

I can't quite parse this. Is "contsant" supposed to be "contestant"? If so, why would constants help with writing "do stuff until time runs out" style submissions?

we only allow ctd, viva, c++ and python as checkers

Yes, you are right. So we could specify different mechanisms for these languages (plus yaml and markdown) and remove support for constants in included code and submissions.

A minor point is that this could make it harder to add additional languages in the future. That said, we could always say that constants are not avaialble for some languages (and then we would be not worse off than if we didn't define constants).

[mzuenni]

1. why do we differentiate between example submissions and other submissions?

"example submissions" are the only kind of submissions in the problem packages.

yes but in the end the problem gets uploaded to a a system that handles both the example submissions and participants submissions. Now it has to distinguish them (which will likely also make it impossible to upload the example submission alone).

Further, we don't don't allow a constant to write solutions like "do random stuff until you are out of time" for which automatic updates would be useful.

I can't quite parse this. Is "contsant" supposed to be "contestant"? If so, why would constants help with writing "do stuff until time runs out" style submissions?

Regarding use cases for constants in submissions: I think inserting something like the time-limit into a submission would be more useful than inserting constants but we already decided that that won't be possible.

Overall i am just in favor of not having constants in submissions and use the for their main purpose, to make checkers and statements consistent.

we only allow ctd, viva, c++ and python as checkers

Yes, you are right. So we could specify different mechanisms for these languages (plus yaml and markdown) and remove support for constants in included code and submissions.

A minor point is that this could make it harder to add additional languages in the future. That said, we could always say that constants are not avaialble for some languages (and then we would be not worse off than if we didn't define constants).

that is true, but I would prefer to change the constants like this:

• If its c++ do X (for example macros)
• If its python do Y (for example insert a variable with that value)
• If its tex do Z (for example introduce a macro)
• Else, replace {{constant}} with its value (we don't (yer?) have better ideas for ctd checkers, viva checkers, markdown or testdata.yaml).

[niemela]

"example submissions" are the only kind of submissions in the problem packages.

yes but in the end the problem gets uploaded to a a system that handles both the example submissions and participants submissions. Now it has to distinguish them (which will likely also make it impossible to upload the example submission alone).

I disagree with that claim. A system does not have to distinguish submissions in the problem package after installation, from any user submissions. To upload package submissions that are using constants you would of course have to apply the constant replacements first, but that is exactly why the target system does not have to even know that our constants exist.

Overall i am just in favor of not having constants in submissions and use the for their main purpose, to make checkers and statements consistent.

The consensus certainly seem to be that statements and validators are the more important use case. Is there anybody who sees a good use for constants in submissions (and included code)?

[...] I would prefer to change the constants like this:

• If its c++ do X (for example macros)
• If its python do Y (for example insert a variable with that value)
• If its tex do Z (for example introduce a macro)
• Else, replace {{constant}} with its value (we don't (yer?) have better ideas for ctd checkers, viva checkers, markdown or testdata.yaml).

Ok, that would work, but:

• It's more complicated
• It still has the issue you are complaining about. The progams wont work until after you have made the replacements. If you're willing to do the replacements first (as is the intent) with these rules, why wont it work with the simpler rules?

[mzuenni]

"example submissions" are the only kind of submissions in the problem packages.

yes but in the end the problem gets uploaded to a a system that handles both the example submissions and participants submissions. Now it has to distinguish them (which will likely also make it impossible to upload the example submission alone).

I disagree with that claim. A system does not have to distinguish submissions in the problem package after installation, from any user submissions. To upload package submissions that are using constants you would of course have to apply the constant replacements first, but that is exactly why the target system does not have to even know that our constants exist.

Well, I think the system you upload the package to has to do the replacement (so for example domjudge)? Anyway, if I want to upload an example submission independently it won't work. As a problem setter, this makes it unlikely for me to use this feature for submissions. I get highlighting errors, I get syntax errors from my ide compiler, and I can't upload my submission and that is only for the small benefit of making changing one or two constants automatic?

Ok, that would work, but:

• It's more complicated
• It still has the issue you are complaining about. The progams wont work until after you have made the replacements. If you're willing to do the replacements first (as is the intent) with these rules, why wont it work with the simpler rules?

Yes, but I rarely run validators on my own (unlike submissions)? And with what I propose at least I don't get errors from my ide while writing those validators. Further, for c++ my ide (and even plain g++ with -D) can do macro insertion. Same for latex (-usepretex). But none of my tools can directly do the replacement we currently specify. So I would need to write some script to add an extra build step that somehow does the replacement and in case of latexmk -pvc (continual rebuild) even that won't be sufficient... Overall, this does not solve all problems especially if the language in question does not have a more convenient way of insertion but IMO in the cases where it is possible, we should use those features.

Also, yes what I propose makes it harder for us as tool authors but it makes it more convenient for the people who use our tools? In the end, our goal should be that constants are used as much as possible for the purpose of consistency i.e. ensuring that validators and therefore test cases match what is specified in the problem statement. Therefore, my aim is to make it as convenient as possible.

[niemela]

Well, I think the system you upload the package to has to do the replacement (so for example domjudge)?

Yes, true, but only when installing the problem (or, depending on how you look at it, before installing them). And that point user submissions do not exist, so I would disagree with the statement that "it has to distinguish them".

Also, yes what I propose makes it harder for us as tool authors but it makes it more convenient for the people who use our tools? In the end, our goal should be that constants are used as much as possible for the purpose of consistency i.e. ensuring that validators and therefore test cases match what is specified in the problem statement. Therefore, my aim is to make it as convenient as possible.

Good point.

[niemela]

I'm (again) hearing strong consensus for constants in statement and validators, some good arguments against constants in submissions, and (so far) no-one that explicitly wants constants in submissions (which is a lower bar than having a good argument for it), only that it could in theory maybe be useful (which is an even lower bar).

If you really think we should allow constants in submissions (and included code), please speak up.