search

EvgSkv / logica

Logica is a logic programming language that compiles to SQL. It runs on DuckDB, Google BigQuery, PostgreSQL and SQLite.
https://logica.dev
Apache License 2.0
1.88k stars 95 forks source link

Databricks SQL support #236

Open samuell opened 1 year ago

samuell commented 1 year ago

Based on recent work for clients, I for one would think support for Databricks SQL would be a huge deal. Seems to be used a little bit of everywhere.

I recently learned Databricks has teamed up with dbt to enable using their SQL templating tech for simplifying writing transformation logic, but Logica would be on a completely different level in my opinion.

EvgSkv commented 1 year ago

Thanks for filing the issue!

I didn't work with Databricks before. I briefly looked at documentation.

The good news is that Databricks has structs and arrays, so adding this as a Logica engine appears doable.

Unfortunately it looks like Databricks doesn't have a free tier. The free trial is limited to 14 days. So to implement it we'd need somebody who already pays for Databricks.

Drop a note in this issue if you are interested in contributing to this.

rvalek commented 1 year ago

Hi, @samuell and @EvgSkv

My team and I also looked into DatabricksSQL support, and we're doing some work right now to target Databricks at least partially. Some of it might be a bit crude, but we'd be happy to raise a PR to see if it could be useful to others once we're done.

I'll follow up on this within several weeks.

EvgSkv commented 1 year ago

Hi @rvalek , that's great to hear! Adding nuances of Databricks to dialects (under compile folder) should be straightforward and safe. If deeper changes are needed (e.g. in universe.py or expr_translate.py), I'm happy to work with you to get them done. Looking forward!

rvalek commented 1 year ago

Hi! We are a little behind on this promise due to some unrelated technical issues, but we are close to resolving them. Just wanted update here that we are still interested in sharing our work on the Databricks dialect. We're planning to raise the PR in early-mid September.

KantorSerhiy commented 1 year ago

hi @EvgSkv! can you suggest in which branch it is best to open a PR for this?

EvgSkv commented 1 year ago

Hi @KantorSerhiy ! Sorry, missed the comment. Sending PR to main is just fine! Thank you!

EvgSkv commented 1 year ago

@KantorSerhiy , btw, I did respond in the PR, it looks great! I left just a couple of requests to comment and it's good to be merged after that.

KantorSerhiy commented 1 year ago

hey @EvgSkv! yes, I saw your comments and will definitely deal with them. but it takes for me 1-2 weeks=( Thank you!

EvgSkv commented 1 year ago

@KantorSerhiy great! No worries! Just wanted to make sure your progress isn't blocked on me :-)

rvalek commented 1 year ago

Hi @samuell , we merged the baseline databricks dialect a couple of weeks ago. Feel free to try it out!