2021-06-07 - KDB Project Call Meeting Minutes

alvin-c-shih commented 3 years ago

Date

Monday, June 7th, 2021 - 9:30am EST / 2:30pm BST (first Monday of every month)

Attendees

Fullname	Affiliation	GitHub Username
Colin Eberhardt	Scott Logic	@ColinEberhardt
Alvin Shih	Morgan Stanley	@alvin-c-shih
Jane Gavronsky	FINOS	@jgavronsky
Aaron Davies	Morgan Stanley	@adavies42
James McLeod	FINOS	@mcleo-d
Jonny Press	AquaQ Analytics	@jonnypress
IvanR

Meeting notices

FINOS Project leads are responsible for observing the FINOS guidelines for running project meetings. Project maintainers can find additional resources in the FINOS Maintainers Cheatsheet.
All participants in FINOS project meetings are subject to the LF Antitrust Policy, the FINOS Community Code of Conduct and all other FINOS policies.
FINOS meetings involve participation by industry competitors, and it is the intention of FINOS and the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws. Please contact legal@finos.org with any questions.
FINOS project meetings may be recorded for use solely by the FINOS team for administration purposes. In very limited instances, and with explicit approval, recordings may be made more widely available.

Agenda

(5 min) Convene, roll call, welcome new people.
[x] Display FINOS Antitrust Policy summary slide
[x] Review Meeting Notices (see above)
(2 min) Approve previous meeting minutes.
(15 min) Idea : kdb Studio as a VS Code plugin?
- @ihull from Scott Logic investigating options for both plugin and standalone.
- Knows Kdb Studio users. Doesn't know kdb users on VS Code.
- One participant asked about other IDEs like IntelliJ / Sublime Text.
- Different plugin architecture, plus a lot of momentum behind VS Code right now.
- VS Code support for "development in the cloud" capturing some imaginations.
- https://devblogs.microsoft.com/visualstudio/introducing-visual-studio-codespaces/
- Support for Kerberos authentication (via plugin API) required for adoption in Morgan Stanley.
- JavaScript implementation of q client API is here: https://github.com/KxSystems/kdb/blob/master/c/c.js
- Reiteration of limitations of NetBeans snapshot we currently have:
- Difficult to support drag-and-drop.
- Keybindings don't match modern conventions.
(5 min) qPython updates.
- Repository is set up.
- Initial hacking on pure Python implementation works, but on the slow side.
- Work on updating fastutils.pyx underway.
- Some questions about memory consumption when receiving a > 2GB payload.
(5 min) dep.q PR sitting since March 5th : https://github.com/finos/kdb/pull/44
- Seeking volunteers to help write unit tests.
- Suggestion about selecting a framework in place and some examples to make it easier to get started.
(5 min) Unique Set Size updates?
- Pure q implementation PR should be out soon.
- Uses FIFOs to read special /proc files, so Linux-only for now.
(5 min) Asof joins for Vaex?
- Vaex doesn't seem to have anything at all.
- https://vaex.io/docs/search.html?q=asof
- Have to convert to Pandas:
- https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.merge_asof.html
- Need to pass temporal column as on parameter(s) and indicate related entries using by parameter(s).
- kdb queries across multiple partitions catenate in RAM.
- But map-reduce over partitions is using mmap.
(5 min) AOB, Q&A, Adjourn.