InseeFr / Trevas

Transformation engine and validator for statistics.
MIT License
10 stars 5 forks source link

Usage of scalars #355

Open noahboerger opened 1 week ago

noahboerger commented 1 week ago

Scalars can currently not be used in Trevas in VTL expressions where constants / literals are allowed. As an example the following VTL code is failing

ds1 := simple_data;
my_scalar := 3;

ds_r := ds1[filter val = my_scalar];

e.g. for simple_data being the dataset

id_1 id_2 val
IDENTIFIER IDENTIFIER MEASURE
INTEGER INTEGER INTEGER
1 2 3
4 5 6

with the exception

Occured error ### Exception ``` fr.insee.vtl.engine.exceptions.UndefinedVariableException: undefined variable my_scalar at fr.insee.vtl.engine.visitors.expression.VarIdVisitor.visitVarID(VarIdVisitor.java:42) at fr.insee.vtl.engine.visitors.expression.VarIdVisitor.visitVarID(VarIdVisitor.java:23) at fr.insee.vtl.parser.VtlParser$VarIDContext.accept(VtlParser.java:9572) at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:18) at fr.insee.vtl.engine.visitors.expression.ExpressionVisitor.visitVarID(ExpressionVisitor.java:106) at fr.insee.vtl.engine.visitors.expression.ExpressionVisitor.visitVarID(ExpressionVisitor.java:41) at fr.insee.vtl.parser.VtlParser$VarIDContext.accept(VtlParser.java:9572) at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:46) at fr.insee.vtl.parser.VtlBaseVisitor.visitVarIdExpr(VtlBaseVisitor.java:48) at fr.insee.vtl.parser.VtlParser$VarIdExprContext.accept(VtlParser.java:478) at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:18) at fr.insee.vtl.engine.visitors.expression.ComparisonVisitor.visitComparisonExpr(ComparisonVisitor.java:142) at fr.insee.vtl.engine.visitors.expression.ComparisonVisitor.visitComparisonExpr(ComparisonVisitor.java:28) at fr.insee.vtl.parser.VtlParser$ComparisonExprContext.accept(VtlParser.java:586) at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:18) at fr.insee.vtl.engine.visitors.expression.ExpressionVisitor.visitComparisonExpr(ExpressionVisitor.java:202) at fr.insee.vtl.engine.visitors.expression.ExpressionVisitor.visitComparisonExpr(ExpressionVisitor.java:41) at fr.insee.vtl.parser.VtlParser$ComparisonExprContext.accept(VtlParser.java:586) at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:18) at fr.insee.vtl.engine.visitors.ClauseVisitor.visitFilterClause(ClauseVisitor.java:167) at fr.insee.vtl.engine.visitors.ClauseVisitor.visitFilterClause(ClauseVisitor.java:33) at fr.insee.vtl.parser.VtlParser$FilterClauseContext.accept(VtlParser.java:1743) at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:46) at fr.insee.vtl.parser.VtlBaseVisitor.visitDatasetClause(VtlBaseVisitor.java:230) at fr.insee.vtl.parser.VtlParser$DatasetClauseContext.accept(VtlParser.java:1494) at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:18) at fr.insee.vtl.engine.visitors.expression.ExpressionVisitor.visitClauseExpr(ExpressionVisitor.java:357) at fr.insee.vtl.engine.visitors.expression.ExpressionVisitor.visitClauseExpr(ExpressionVisitor.java:41) at fr.insee.vtl.parser.VtlParser$ClauseExprContext.accept(VtlParser.java:683) at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:18) at fr.insee.vtl.engine.visitors.AssignmentVisitor.visitAssignment(AssignmentVisitor.java:51) at fr.insee.vtl.engine.visitors.AssignmentVisitor.visitTemporaryAssignment(AssignmentVisitor.java:59) at fr.insee.vtl.parser.VtlParser$TemporaryAssignmentContext.accept(VtlParser.java:372) at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:18) at fr.insee.vtl.engine.VtlScriptEngine.evalStream(VtlScriptEngine.java:263) at fr.insee.vtl.engine.VtlScriptEngine.eval(VtlScriptEngine.java:282) at java.scripting/javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:262) at fr.insee.trevas.jupyter.VtlKernel.eval(VtlKernel.java:305) at io.github.spencerpark.jupyter.kernel.BaseKernel.handleExecuteRequest(BaseKernel.java:334) at io.github.spencerpark.jupyter.channels.ShellChannel.lambda$bind$0(ShellChannel.java:64) at io.github.spencerpark.jupyter.channels.Loop.lambda$new$0(Loop.java:21) at io.github.spencerpark.jupyter.channels.Loop.run(Loop.java:78) ```

Such VTL statements are out of our point of view especially useful when the scalar value is set in the bindings outside of the script itself to parametrize the whole script execution.

Is that something that also needs to be adressed in the Task force Repo?

NicoLaval commented 1 week ago

Thanks @noahboerger. It's a Trevas issue, not a VTL issue. You are at the right place.