Weekly Status Log: Query Interfaces

[domoritz]

This is for https://github.com/uwdata/mosaic/issues/483, https://github.com/uwdata/mosaic/issues/510, and https://github.com/uwdata/mosaic/issues/506. Code is in https://github.com/cmudig/mosaic/tree/prepare.

[dorianpan]

September 20, 2024

– should try a different way of extracting the params (visitor pattern as a ref) different from the one in toString() – server implementation (python first) – possibly try to refactor toString() into toSql() and have two versions for prepared and non-prepared

Goal next week: – layout implementation plan for adding the prepare statement – have an initial implementation based on the plan

[dorianpan]

September 25th, 2024

Discussion today – rename toString() function – safety issue with current implementation -- ensure the approach is robust – use a helper function probably? – better logic for adding “?” https://docs.google.com/document/d/1RiO9qCwJS333HMGcDEAKb7xV80oZwGfYoFzIyARqeZg/edit

[dorianpan]

September 27th, 2024

Discussion today: 9.27 meeting — front end sends the generic string & param list (still buggy but should be done soon) — server almost the same as the old implementation (already passed the test) — Dorian’s next step:

• Make sure the toSql() function works
• Test it with current backend implementation — Plan to be completed (doug)
• Node connector
• Update json format in data-server.js
• Add prepared config param
• arrowBuffer (DuckDBStatement): this.statement instead of this.con?
• Add tests
• cache: update cache to include prepared params, etc, update calls to cacheKey
• prepare(sql): add map to check for prepared
• What is bundle?
• Python connector
• What are handlers doing? (what is app?)

[domoritz]

What is bundle?

Bundles are prepared sets of queries that can be loaded into the cache. There is a demo app in https://github.com/uwdata/mosaic/blob/main/dev/bundle.html.

[dorianpan]

October 2, 2024

Douglas: updated the node server code to accommodate for the update on the sql builder; started python server update

-- todo: update python server code to prepare query in the sample script format

Dorian: the toSql function is able to return an object containing the param array and the query string with ? marks

-- todo: need to support both prepared and non-prepared versions by having an indicator variable as an input

Next step: test the implementation together & make changes if anything doesn't work

[dorianpan]

October 4, 2024

Pushed all changes to server and sql builder to remote branch prepare Modified test cases for Python server so now able to run test locally

Pending: need to find a way to connect to the python server and run test with both sql builder and the python server; maybe write a script that make requests to localhost?

[dorianpan]

October 9, 2024

meeting updates Douglas -- updated python and node server so now they are able to handle prepared statements

Dorian -- implemented the query generator to support generating both prepared and non-prepared query strings (buggy now needs a closer look)

goals next week: -- figure out if prepared statement is per session -- able to enable & disable prepared statement (mostly client side) -- create unit tests and start to think about how to modify the interface so there will be an option to select whether the statement is prepared or not

[dorianpan]

October 23, 2024

meeting updates

Douglas: -- added prepared statement support for bundles -- added more test cases -- updated query manager so that it's able to disable/enable prepared statement

Dorian: -- rewrote the SQL builder to implement the visitor pattern logic to generate query strings -- renamed the toString() method of the Queryclass to toSql() and modified it to accept visitors as inputs -- added test cases for the updated Queryclass

goals next week:

Douglas: -- figure out prepared logic: per connection / per query -- look into rust server and make updates -- start integration with the updated toSQL() function

Dorian: -- add more test cases for the updated toSQL() function -- change all other files that uses Query.toString() to Query.toSQL() (might include implicit toString calls) -- refine the visitor logic if needed

[domoritz]

Relevant comment: https://github.com/uwdata/mosaic/issues/567#issuecomment-2444898957

[dorianpan]

November 6, 2024

Updates

Douglas: looked into rust server, questions related to built-in cache for the rust server

merged current work with Dorian's branch, backend support for prepared statements ready for node.js server & python server

Dorian: updated toSQL() function according to the comments from last meeting

updated code in core package related to the implicit use of toString; replaced them with non-prepared toSQL() function for now, passed all unit tests

merged code with Douglas' branch

Next step:

Douglas: implement backend support for rust

Dorian: implement prepared version of toSQL() with updated backend support

write unit tests for prepared statements

Both: address any remaining merge conflicts or obsolete code caused by updated return type of toSQL()