enso-org / enso

Hybrid visual and textual functional programming.
https://enso.org
Apache License 2.0
7.31k stars 317 forks source link

Initial separation of Snowflake_Dialect from Postgres_Dialect #10266

Closed radeusgd closed 2 weeks ago

radeusgd commented 2 weeks ago

Pull Request Description

Important Notes

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

radeusgd commented 2 weeks ago

Trying to run the Snowflake tests. At first it was failing miserably and completely crashing mid-way.

After some fixes I'm able to run the suite, although not everything works. Still I think it's enough to integrate - we can work on fixing tests later.

We won't be running these on CI yet - they are far too slow and need a separate workflow, probably not run for most PRs. So it's ok to keep them failing - for many tests it's similar amount of work to mark them pending as to fix them.

Current status:

The biggest issue is the execution time of tests. Running the whole suite takes 45 minutes (!!!). In comparison running SQLite or Postgres tests takes ~2.5 minutes each.

I tried writing some profiling code and it turns out the problem is that we are running lots of queries (as we are testing lots of stuff). The Snowflake query latency is actually very nice - averaging at 187 ms per operation - for a remote Database operation that's not too bad. However, in local instances of Postgres or SQLite it's just around 1 ms - so the difference is huge - roughly 100x. Given that even the lightening-fast local tests run for about 2 minutes, it is not a surprise that the 100x slower remote tests take up to an hour... This may be a problem for other remote backends as well.

radeusgd commented 2 weeks ago

image