2020-11-02 - KDB Project Call Meeting Minutes

alvin-c-shih commented 4 years ago

Date

Monday, November 2nd, 2020 - 10am EST / 3pm BST

// First Monday of every month

Attendees

Fullname	Affiliation	GitHub Username
Aitana Myohl	FINOS	@aitana16
Aaron Davies	Morgan Stanley	@adavies42
Alvin Shih	Morgan Stanley	@alvin-c-shih
Peter Gyorok	Morgan Stanley	@gyorokpeter
Jonny Press	AquaQ Analytics	@jonnypress
Richard Wagener	Codethink	@RichardWagener365
Latchezar Nikolov	Morgan Stanley

Minutes

(5 min) Convene, roll call, welcome new people.
(5 min) Approve previous meeting minutes.
- Minutes approved by acclamation.
(5 min) Kdb Studio updates.
- font size change from the UI
- release process actions create a Zipfile but bakes in "java" instead of "javaw"
- to be investigated
(3 min) Question about submitting pull requests from home computer.
- Safer to have an Outside Business Interest filed with employer for that.
(3 min) Kdb Studio README.
- Peter will update the blurb now that new icons have been contributed.
- Will also update download instructions.
(2 min) Updates on best practices to contribute upstream?
- Still some mechanics to discuss with Open Source governance in Morgan Stanley.
(2 min) Saving CSV files with compound columns: https://github.com/finos/kdb/pull/32
- .h.tx[`csv]:{.q.csv 0: .finos.html.stringifyCompoundColumns x}
- no one had occasion to play with it
(5 min) Allow multi-line functions to be pasted into the console.
- https://github.com/alvin-c-shih/kdb/blob/master/misc/mline.q
- Does not support \d.
- Also doesn't appear to support block comments.
- Best that everyone lobbies Kx to expose their multi-line parser instead of making us re-invent it.
(5 min) PyQ interop topics for future meetings:
- PyQ doesn't handle nested values very well
- issues with q.lj from PyQ?
- best practices for:
- enumerated columns
- loading HDBs?
- dealing with nulls / infinities
- q('table') doesn't work for splays
- attributes
- link columns
(5 min) Alternatives to PyQ?
- Vaex might have decent enough performance for certain use cases.
- https://www.aquaq.co.uk/q/comparing-columnar-data-formats-arrow-vaex-and-kdb/
- Doesn't include map-reduce out of the box.
- kdb interface to BigQuery?
- Hybrid model with on-prem datasets in kdb, history in the cloud.
- Refinitiv historical data accessed through BigQuery.
- https://developers.refinitiv.com/en/article-catalog/article/big-data-tick-history-google-bigquery
(5 min) AOB, Q&A, Adjourn.