Weekly Status Log: Query Interfaces

domoritz commented 2 months ago

This is for https://github.com/uwdata/mosaic/issues/483, https://github.com/uwdata/mosaic/issues/510, and https://github.com/uwdata/mosaic/issues/506. Code is in https://github.com/cmudig/mosaic/tree/prepare.

dorianpan commented 1 month ago

September 20, 2024

– should try a different way of extracting the params (visitor pattern as a ref) different from the one in toString() – server implementation (python first) – possibly try to refactor toString() into toSql() and have two versions for prepared and non-prepared

Goal next week: – layout implementation plan for adding the prepare statement – have an initial implementation based on the plan

dorianpan commented 1 month ago

September 25th, 2024

Discussion today – rename toString() function – safety issue with current implementation -- ensure the approach is robust – use a helper function probably? – better logic for adding “?” https://docs.google.com/document/d/1RiO9qCwJS333HMGcDEAKb7xV80oZwGfYoFzIyARqeZg/edit

dorianpan commented 1 month ago

September 27th, 2024

Discussion today: 9.27 meeting — front end sends the generic string & param list (still buggy but should be done soon) — server almost the same as the old implementation (already passed the test) — Dorian’s next step:

Make sure the toSql() function works
Test it with current backend implementation — Plan to be completed (doug)
Node connector
Update json format in data-server.js
Add prepared config param
arrowBuffer (DuckDBStatement): this.statement instead of this.con?
Add tests
cache: update cache to include prepared params, etc, update calls to cacheKey
prepare(sql): add map to check for prepared
What is bundle?
Python connector
What are handlers doing? (what is app?)

domoritz commented 1 month ago

What is bundle?

Bundles are prepared sets of queries that can be loaded into the cache. There is a demo app in https://github.com/uwdata/mosaic/blob/main/dev/bundle.html.

dorianpan commented 1 month ago

October 2, 2024

Douglas: updated the node server code to accommodate for the update on the sql builder; started python server update

-- todo: update python server code to prepare query in the sample script format

Dorian: the toSql function is able to return an object containing the param array and the query string with ? marks

-- todo: need to support both prepared and non-prepared versions by having an indicator variable as an input

Next step: test the implementation together & make changes if anything doesn't work

dorianpan commented 1 month ago

October 4, 2024

Pushed all changes to server and sql builder to remote branch prepare Modified test cases for Python server so now able to run test locally

Pending: need to find a way to connect to the python server and run test with both sql builder and the python server; maybe write a script that make requests to localhost?

dorianpan commented 1 month ago

October 9, 2024

meeting updates Douglas -- updated python and node server so now they are able to handle prepared statements

Dorian -- implemented the query generator to support generating both prepared and non-prepared query strings (buggy now needs a closer look)

goals next week: -- figure out if prepared statement is per session -- able to enable & disable prepared statement (mostly client side) -- create unit tests and start to think about how to modify the interface so there will be an option to select whether the statement is prepared or not

dorianpan commented 3 weeks ago

October 23, 2024

meeting updates

Douglas: -- added prepared statement support for bundles -- added more test cases -- updated query manager so that it's able to disable/enable prepared statement

Dorian: -- rewrote the SQL builder to implement the visitor pattern logic to generate query strings -- renamed the toString() method of the Queryclass to toSql() and modified it to accept visitors as inputs -- added test cases for the updated Queryclass

goals next week:

Douglas: -- figure out prepared logic: per connection / per query -- look into rust server and make updates -- start integration with the updated toSQL() function

Dorian: -- add more test cases for the updated toSQL() function -- change all other files that uses Query.toString() to Query.toSQL() (might include implicit toString calls) -- refine the visitor logic if needed

domoritz commented 2 weeks ago

Relevant comment: https://github.com/uwdata/mosaic/issues/567#issuecomment-2444898957

dorianpan commented 1 week ago

November 6, 2024

Updates

Douglas: looked into rust server, questions related to built-in cache for the rust server

merged current work with Dorian's branch, backend support for prepared statements ready for node.js server & python server

Dorian: updated toSQL() function according to the comments from last meeting

updated code in core package related to the implicit use of toString; replaced them with non-prepared toSQL() function for now, passed all unit tests

merged code with Douglas' branch

Next step:

Douglas: implement backend support for rust

Dorian: implement prepared version of toSQL() with updated backend support

write unit tests for prepared statements

Both: address any remaining merge conflicts or obsolete code caused by updated return type of toSQL()

cmudig / mosaic