Rel server protocol error: expected 'E' or 'X', but got '<'.

[ancailliau]

While executing the tests, I got the following error:

Tuple({:puzzle => "S6", :exemplar => "suppliers-and-parts", :dataset => "empty", :answer1 => "c17e5350-e9d1-0133-5dbc-6c40089dd66a", :submission1 => "c17c15e0-e9d1-0133-5dbc-6c40089dd66a", :language => "tutorial-d", :expression1 => "PARTS WHERE CITY = 'London' OR CITY = 'Paris'", :answer2 => "c014a460-e9d1-0133-5dba-6c40089dd66a", :submission2 => "c0131c70-e9d1-0133-5dba-6c40089dd66a", :language2 => "tutorial-d", :expression2 => "PARTS WHERE CITY='London' OR CITY='Paris'"})
PARTS WHERE CITY = 'London' OR CITY = 'Paris'
Unable to test submissions
  Failure: Failed to parse input on line 1 at offset 5
ERROR: Rel server protocol error: expected 'E' or 'X', but got '<'.

     ^

I'm not sure on why it fails or what extra information is required to reproduce the bug.

[ancailliau]

Got it four times in total -not the same student, not the same question-

Tuple({:puzzle => "M2", :exemplar => "suppliers-and-parts", :dataset => "empty", :answer1 => "c1829920-e9d1-0133-5dbc-6c40089dd66a", :submission1 => "c17c15e0-e9d1-0133-5dbc-6c40089dd66a", :language => "tutorial-d", :expression1 => "(SHIPMENTS JOIN SUPPLIERS WHERE STATUS >= 20){PID}", :answer2 => "c0b9d7d0-e9d1-0133-5dba-6c40089dd66a", :submission2 => "c0b7feb0-e9d1-0133-5dba-6c40089dd66a", :language2 => "tutorial-d", :expression2 => "PARTS {PID} JOIN (SHIPMENTS {PID} MATCHING (SUPPLIERS WHERE STATUS >= 20))"})
(SHIPMENTS JOIN SUPPLIERS WHERE STATUS >= 20){PID}
Unable to test submissions
  Failure: Failed to parse input on line 1 at offset 5
ERROR: Rel server protocol error: expected 'E' or 'X', but got '<'.

Tuple({:puzzle => "H2", :exemplar => "suppliers-and-parts", :dataset => "empty", :answer1 => "c1838230-e9d1-0133-5dbc-6c40089dd66a", :submission1 => "c17c15e0-e9d1-0133-5dbc-6c40089dd66a", :language => "tutorial-d", :expression1 => "(SUMMARIZE (SHIPMENTS MATCHING (SUPPLIERS WHERE STATUS <= 20)) PER (SHIPMENTS{PID}):{n:=COUNT()} WHERE n >= COUNT((SUPPLIERS WHERE STATUS <= 20))){PID}", :answer2 => "c05850c0-e9d1-0133-5dba-6c40089dd66a", :submission2 => "c0549d80-e9d1-0133-5dba-6c40089dd66a", :language2 => "tutorial-d", :expression2 => "(SUMMARIZE (SHIPMENTS JOIN SUPPLIERS WHERE STATUS <= 20) BY {PID}:{\n    number:=COUNT()} WHERE number = COUNT(SUPPLIERS WHERE STATUS <= 20)){PID}"})
(SUMMARIZE (SHIPMENTS MATCHING (SUPPLIERS WHERE STATUS <= 20)) PER (SHIPMENTS{PID}):{n:=COUNT()} WHERE n >= COUNT((SUPPLIERS WHERE STATUS <= 20))){PID}
Unable to test submissions
  Failure: Failed to parse input on line 1 at offset 5
ERROR: Rel server protocol error: expected 'E' or 'X', but got '<'.

Tuple({:puzzle => "E3", :exemplar => "suppliers-and-parts", :dataset => "empty", :answer1 => "c17e8b30-e9d1-0133-5dbc-6c40089dd66a", :submission1 => "c17c15e0-e9d1-0133-5dbc-6c40089dd66a", :language => "tutorial-d", :expression1 => "RELATION {c INTEGER} { TUPLE { c COUNT((SHIPMENTS WHERE SID = S#('S3')){PID})}}", :answer2 => "bf967a60-e9d1-0133-5dba-6c40089dd66a", :submission2 => "bf943eb0-e9d1-0133-5dba-6c40089dd66a", :language2 => "tutorial-d", :expression2 => "COUNT (\n  (SHIPMENTS WHERE SID = S#('S3'))\n  {PID} )"})
RELATION {c INTEGER} { TUPLE { c COUNT((SHIPMENTS WHERE SID = S#('S3')){PID})}}
Unable to test submissions
  Failure: Failed to parse input on line 1 at offset 5
ERROR: Rel server protocol error: expected 'E' or 'X', but got '<'.

[blambeau]

@ancailliau strange. Could you send me Quentin's submission and one submission that fails as above? I'll try on my computer and log what is actually sent to Rel.

[ancailliau]

What's funny is that if I clear the database, re-import and re-run tests, I also fail, but not on the same submissions. But I got more occurences of the bug this time.

872/5096: Tuple({:exemplar => "suppliers-and-parts", :dataset => "empty", :answer1 => "f65c1150-e9d4-0133-5dce-6c40089dd66a", :answer2 => "f58b27e0-e9d4-0133-5dcc-6c40089dd66a", :outcome => "parsing error", :details => "Failed to parse input on line 1 at offset 5\nERROR: Encountered \" \")\" \") \"\" at line 2, column 110.\n\n     ^"})
Tuple({:puzzle => "D6", :exemplar => "suppliers-and-parts", :dataset => "base", :answer1 => "f65c5180-e9d4-0133-5dce-6c40089dd66a", :submission1 => "f654ada0-e9d4-0133-5dce-6c40089dd66a", :language => "tutorial-d", :expression1 => "WITH (HEAVIEST := PARTS WHERE WEIGHT = MIN(PARTS, WEIGHT)) :\n(HEAVIEST WHERE PID = MAX(HEAVIEST, PID)) { PID }", :answer2 => "f4e48a50-e9d4-0133-5dcc-6c40089dd66a", :submission2 => "f4e21cc0-e9d4-0133-5dcc-6c40089dd66a", :language2 => "tutorial-d", :expression2 => "(PARTS WHERE PID = MAX((PARTS WHERE WEIGHT= MIN(PARTS, WEIGHT)), PID)){PID}"})
WITH (HEAVIEST := PARTS WHERE WEIGHT = MIN(PARTS, WEIGHT)) :
(HEAVIEST WHERE PID = MAX(HEAVIEST, PID)) { PID }
Unable to test submissions
  Failure: Failed to parse input on line 1 at offset 5
ERROR: Rel server protocol error: expected 'E' or 'X', but got '<'.

872/5096: Tuple({:exemplar => "suppliers-and-parts", :dataset => "empty", :answer1 => "f65c1150-e9d4-0133-5dce-6c40089dd66a", :answer2 => "f58b27e0-e9d4-0133-5dcc-6c40089dd66a", :outcome => "parsing error", :details => "Failed to parse input on line 1 at offset 5\nERROR: Encountered \" \")\" \") \"\" at line 2, column 110.\n\n     ^"})
Tuple({:puzzle => "D4", :exemplar => "suppliers-and-parts", :dataset => "empty", :answer1 => "f65c1150-e9d4-0133-5dce-6c40089dd66a", :submission1 => "f654ada0-e9d4-0133-5dce-6c40089dd66a", :language => "tutorial-d", :expression1 => "((SUPPLIERS{SID, CITY} RENAME {CITY AS SCITY} JOIN PARTS{PID, CITY} RENAME {CITY AS PCITY}) WHERE SCITY <> PCITY){SID, PID}", :answer2 => "f595aab0-e9d4-0133-5dcc-6c40089dd66a", :submission2 => "f5930cf0-e9d4-0133-5dcc-6c40089dd66a", :language2 => "tutorial-d", :expression2 => "((SUPPLIERS{SID, CITY} JOIN PARTS{PID}) NOT MATCHING PARTS {PID, CITY}) {SID, PID}"})
((SUPPLIERS{SID, CITY} RENAME {CITY AS SCITY} JOIN PARTS{PID, CITY} RENAME {CITY AS PCITY}) WHERE SCITY <> PCITY){SID, PID}
Unable to test submissions
  Failure: Failed to parse input on line 1 at offset 5
ERROR: Rel server protocol error: expected 'E' or 'X', but got '<'.

216/4223: Tuple({:exemplar => "suppliers-and-parts", :dataset => "empty", :answer1 => "f6572230-e9d4-0133-5dce-6c40089dd66a", :answer2 => "f58a5ac0-e9d4-0133-5dcc-6c40089dd66a", :outcome => "parsing error", :details => "Failed to parse input on line 1 at offset 5\nERROR: Encountered \" \")\" \") \"\" at line 2, column 26.\n\n     ^"})
Tuple({:puzzle => "S8", :exemplar => "suppliers-and-parts", :dataset => "empty", :answer1 => "f6572230-e9d4-0133-5dce-6c40089dd66a", :submission1 => "f654ada0-e9d4-0133-5dce-6c40089dd66a", :language => "tutorial-d", :expression1 => "(PARTS WHERE CITY = 'London'){PID}", :answer2 => "f594a8f0-e9d4-0133-5dcc-6c40089dd66a", :submission2 => "f5930cf0-e9d4-0133-5dcc-6c40089dd66a", :language2 => "tutorial-d", :expression2 => "(PARTS  WHERE CITY = 'London') {PID}"})
(PARTS WHERE CITY = 'London'){PID}
Unable to test submissions
  Failure: Failed to parse input on line 1 at offset 5
ERROR: Rel server protocol error: expected 'E' or 'X', but got '<'.

1166/4006: Tuple({:exemplar => "suppliers-and-parts", :dataset => "empty", :answer1 => "f65758e0-e9d4-0133-5dce-6c40089dd66a", :answer2 => "f4645280-e9d4-0133-5dcc-6c40089dd66a", :outcome => "parsing error", :details => "Failed to parse input on line 1 at offset 5\nERROR: Encountered \" <IDENTIFIER> \"NO \"\" at line 2, column 7.\n\n     ^"})
Tuple({:puzzle => "E5", :exemplar => "suppliers-and-parts", :dataset => "empty", :answer1 => "f65758e0-e9d4-0133-5dce-6c40089dd66a", :submission1 => "f654ada0-e9d4-0133-5dce-6c40089dd66a", :language => "tutorial-d", :expression1 => "PARTS NOT MATCHING SHIPMENTS", :answer2 => "f45a33f0-e9d4-0133-5dcc-6c40089dd66a", :submission2 => "f45848b0-e9d4-0133-5dcc-6c40089dd66a", :language2 => "tutorial-d", :expression2 => "PARTS NOT MATCHING SHIPMENTS"})
PARTS NOT MATCHING SHIPMENTS
Unable to test submissions
  Failure: Failed to parse input on line 1 at offset 5
ERROR: Rel server protocol error: expected 'E' or 'X', but got '<'.

341/2839: Tuple({:exemplar => "suppliers-and-parts", :dataset => "empty", :answer1 => "f65c6de0-e9d4-0133-5dce-6c40089dd66a", :answer2 => "f5205490-e9d4-0133-5dcc-6c40089dd66a", :outcome => "parsing error", :details => "Failed to parse input on line 1 at offset 5\nERROR: Encountered \" \"WHERE\" \"WHERE \"\" at line 2, column 17.\n\n     ^"})
Tuple({:puzzle => "H2", :exemplar => "suppliers-and-parts", :dataset => "empty", :answer1 => "f65c6de0-e9d4-0133-5dce-6c40089dd66a", :submission1 => "f654ada0-e9d4-0133-5dce-6c40089dd66a", :language => "tutorial-d", :expression1 => "(SUMMARIZE (SHIPMENTS MATCHING (SUPPLIERS WHERE STATUS <= 20)) PER (SHIPMENTS{PID}):{n:=COUNT()} WHERE n >= COUNT((SUPPLIERS WHERE STATUS <= 20))){PID}", :answer2 => "f52fd980-e9d4-0133-5dcc-6c40089dd66a", :submission2 => "f52d2960-e9d4-0133-5dcc-6c40089dd66a", :language2 => "tutorial-d", :expression2 => "(SUMMARIZE (SHIPMENTS JOIN SUPPLIERS WHERE STATUS <= 20) BY {PID}:{\n    number:=COUNT()} WHERE number = COUNT(SUPPLIERS WHERE STATUS <= 20)){PID}"})
(SUMMARIZE (SHIPMENTS MATCHING (SUPPLIERS WHERE STATUS <= 20)) PER (SHIPMENTS{PID}):{n:=COUNT()} WHERE n >= COUNT((SUPPLIERS WHERE STATUS <= 20))){PID}
Unable to test submissions
  Failure: Failed to parse input on line 1 at offset 5
ERROR: Rel server protocol error: expected 'E' or 'X', but got '<'.

118/2497: Tuple({:exemplar => "suppliers-and-parts", :dataset => "empty", :answer1 => "f65b3bb0-e9d4-0133-5dce-6c40089dd66a", :answer2 => "f419f300-e9d4-0133-5dcc-6c40089dd66a", :outcome => "parsing error", :details => "Failed to parse input on line 1 at offset 5\nERROR: Encountered \" \"RENAME\" \"RENAME \"\" at line 2, column 111.\n\n     ^"})
Tuple({:puzzle => "E6", :exemplar => "suppliers-and-parts", :dataset => "empty", :answer1 => "f65b3bb0-e9d4-0133-5dce-6c40089dd66a", :submission1 => "f654ada0-e9d4-0133-5dce-6c40089dd66a", :language => "tutorial-d", :expression1 => "(SHIPMENTS {SID, PID} JOIN SUPPLIERS{SID, CITY} RENAME {CITY AS SLOC}) JOIN PARTS{PID, NAME, WEIGHT} RENAME {NAME AS PNAME, WEIGHT AS PWEIGHT}", :answer2 => "f429c480-e9d4-0133-5dcc-6c40089dd66a", :submission2 => "f4262d80-e9d4-0133-5dcc-6c40089dd66a", :language2 => "tutorial-d", :expression2 => "((SUPPLIERS {ALL BUT NAME} JOIN SHIPMENTS) JOIN (PARTS {ALL BUT CITY})) {SID, CITY, PID , NAME, WEIGHT} RENAME {CITY AS SLOC, NAME AS PNAME, WEIGHT AS PWEIGHT}"})
(SHIPMENTS {SID, PID} JOIN SUPPLIERS{SID, CITY} RENAME {CITY AS SLOC}) JOIN PARTS{PID, NAME, WEIGHT} RENAME {NAME AS PNAME, WEIGHT AS PWEIGHT}
Unable to test submissions
  Failure: Failed to parse input on line 1 at offset 5
ERROR: Rel server protocol error: expected 'E' or 'X', but got '<'.

932/2378: Tuple({:exemplar => "suppliers-and-parts", :dataset => "no-shipments", :answer1 => "f65b3bb0-e9d4-0133-5dce-6c40089dd66a", :answer2 => "f419f300-e9d4-0133-5dcc-6c40089dd66a", :outcome => "parsing error", :details => "Failed to parse input on line 1 at offset 5\nERROR: Encountered \" \"RENAME\" \"RENAME \"\" at line 2, column 111.\n\n     ^"})
Tuple({:puzzle => "E6", :exemplar => "suppliers-and-parts", :dataset => "no-shipments", :answer1 => "f65b3bb0-e9d4-0133-5dce-6c40089dd66a", :submission1 => "f654ada0-e9d4-0133-5dce-6c40089dd66a", :language => "tutorial-d", :expression1 => "(SHIPMENTS {SID, PID} JOIN SUPPLIERS{SID, CITY} RENAME {CITY AS SLOC}) JOIN PARTS{PID, NAME, WEIGHT} RENAME {NAME AS PNAME, WEIGHT AS PWEIGHT}", :answer2 => "f40f99a0-e9d4-0133-5dcc-6c40089dd66a", :submission2 => "f40d5c10-e9d4-0133-5dcc-6c40089dd66a", :language2 => "tutorial-d", :expression2 => "((SHIPMENTS JOIN SUPPLIERS) RENAME {CITY AS sloc} JOIN (PARTS RENAME {NAME AS pname, WEIGHT AS pweight})){SID, PID, sloc, pname, pweight}"})
(SHIPMENTS {SID, PID} JOIN SUPPLIERS{SID, CITY} RENAME {CITY AS SLOC}) JOIN PARTS{PID, NAME, WEIGHT} RENAME {NAME AS PNAME, WEIGHT AS PWEIGHT}
Unable to test submissions
  Failure: Failed to parse input on line 1 at offset 5
ERROR: Rel server protocol error: expected 'E' or 'X', but got '<'.

[blambeau]

@ancailliau On my machine, everything works fine with the submissions you gave me. Possibly we do not have the same Rel version?

[blambeau]

@ancailliau OK, it is clearly when a given query had a syntax error, then the next query fails with the "E, X, <" message. I would need more data to investigate, especially a submission with parse errors...

[blambeau]

@ancailliau underterministic problem because of underterministic order in which tests are ran.

The good news is: you can simply keep executing bin/rqp2 test until no error shows and have the guarantee that everything has been ran and is then ok because of the way rqp2 has been built (especially to handle this kind of problems).

Still, I'd like to have all submissions, to debug the stuff.

[blambeau]

For the record: it is most probably due to a kind of "injection attack" of Rel's text protocol. The protocol expects something like

E valid a query text here
<EOT>

And it seems that here, the <EOT> keeps being pending for the next query, while it shouldn't. Without having a deterministic way of reproducing this, it's rather hard to verify that assumption.