Another way to compare result file

[jiacai2050]

Describe This Problem

In current usage pattern, sqlness will write query response to .out file, and it's used to be compared with corresponding .result, while this works for basic case, it becomes verbose when we change some behaviors of server, and lots of .result file need to be updated ONE AFTER ONE.

Proposal

I suggest another pattern to compare result:

Write query result directly to result file

It works like this

• When a SQL test file is first added, write result file directly can save us one mv command
• When execute testcases next time, save original result file in Vec or tempfile before write query response, then compare it with result file,
  • If updated result file is what we expected, just check into git with git add ...
  • if wrong, discard result file with git checkout ..., update testcase and run again

I think this usage pattern is more efficient than before.

Additional Context

No response

[jiacai2050]

@ShiKaiWi @waynexia How do you guys think about this?

[waynexia]

In short, this ticket suggests getting rid of .out file and writing the execution result directly into .result file. Then make use of git to tracking/comparing/reverting the new result, right?

[jiacai2050]

Exactly

[waynexia]

Makes sense to me 💯 I was trying to find a way to better integrate git and diff into sqlness' workflow, but haven't thought of just dropping .out file and letting git track 🥲.

[waynexia]

save original result file in Vec or tempfile

Do we have to keep the previous result before overwriting? If the diff comes from different behavior of code changes, the original result is likely to be tracked in git.

[jiacai2050]

Yes, but in some environment there maybe not have git info, such as download src via zip....

I just think git is not a must dependency for sqlness to work.

[waynexia]

Yes, but in some environment there maybe not have git info, such as download src via zip....

I just think git is not a must dependency for sqlness to work.

Great 👍 . Remind me CI is such an environment that doesn't have git.

[jiacai2050]

😁 I will implement this feature soon.