Data source challenge: SAP HANA

[tomtom215]

Due to it being a proprietary DB that doesn't speak mysql or Postgres dialects, and is a columnar store - I'd love to see the ability to use Quokka to query SAP HANA

[marsupialtail]

@tomtom215 I though SAP HANA supports JDBC?

[tomtom215]

It does, I opened the issue because I didn't see any JDBC config details in the documentation and saw on your roadmap you were working on a SQL interface: https://marsupialtail.github.io/quokka/search.html?q=jdbc

In your opinion - if a DB supports JDBC, do feel it should be supported by Quokka?

[marsupialtail]

So if a DB supports JDBC, can't you just query it directly with SQL? Sorry I am misunderstanding something here.

[tomtom215]

Yes and no due to SQL dialects. For example DataGrip is a great Database IDE but you can't just drop in the SAP HANA JDBC driver and everything works, support had to be added in by Jetbrains. Similar in that SQLalchemy will get you only so far for certain DB's with special SQL dialects. However, if you think I'm way off and you believe everything should work, I'll see if I can setup a proof of concept in the next month or so (holidays makes it hard to schedule) with Quokka

• https://intellij-support.jetbrains.com/hc/en-us/community/posts/360010694680-DataGrip-connecting-to-SAP-Hana-via-JDBC-
• https://youtrack.jetbrains.com/issue/DBE-630
• https://github.com/SAP/sqlalchemy-hana

[tomtom215]

Another reason why I wanted to ask is the example of Clickhouse JDBC Bridge. In theory, it should allow you to connect to any JDBC capable DB, but it has clear limitations:

• Connection issue like jdbc-bridge is not running or connect timed out - see performance section and https://github.com/ClickHouse/ClickHouse/issues/9609 for details
• Complex data types like Array and Tuple are currently not supported - they're treated as String
• Pushdown is not supported and query may execute twice because of type inferring
• Mutation is not fully supported - only insertion in simple cases

[marsupialtail]

Let me do more research into JDBC and its limitations. In an ideal world everything will be Arrow Flight based and life will be amazing but we are quite far off from that. We can keep discussing in this thread.

Adding a JDBC driver is a great way to contribute to Quokka. Happy to work on it with you.

[marsupialtail]

Thank you for all the information.

[marsupialtail]

@tomtom215 do you know if JDBC tells you how big a SAP HANA table is? I imagine you can query select count (*) to figure that out but I wonder how performant that is. That should return instantly in a good implementation but perhaps not.

This is quite important in Quokka as Quokka is moving towards an architecture where all the data sources are bounded, so we can do better progress tracking during execution.

[marsupialtail]

Quokka will support ODBC soon.