2020-12-07 - KDB Project Call Meeting Minutes

alvin-c-shih commented 3 years ago

Date

Monday, December 7, 2020 - 10am EST / 3pm BST

// First Monday of every month

Attendees

Fullname	Affiliation	GitHub Username
Timofey Beletskiy	JP Morgan
Aaron Davies	Morgan Stanley	@adavies42
Alvin Shih	Morgan Stanley	@alvin-c-shih
Peter Gyorok	Morgan Stanley	@gyorokpeter
Rob Underwood	FINOS	@brooklynrob
Jonny Press	AquaQ Analytics	@jonnypress
Alok Dutt	Citi

Minutes

(5 min) Convene, roll call, welcome new people.
(5 min) Approve previous meeting minutes.
- Approved by acclamation.
(5 min) Scheduling tweaks.
- Moving time of meeting to 9:30 approved by acclamation.
(5 min) Kdb Studio updates.
- Updated README about icons.
- Download instructions clarified.
- Small dataset to replicate graph for screen shot added.
- Still looking for help with generating launcher using javaw.exe instead of java.exe.
(5 min) dep.q as a first iteration of FINOS module loader. https://github.com/finos/kdb/pull/33
- GitHub project will be reorganized to have a q/ directory for compatible modules.
- More clearly distinguish from other contributions like technical-documentation/.
- Won't worry about layout for C/C++ plugins until some are contributed.
- Look to Python / Perl for inspiration when the time comes.
(10 min) PyQ wrapper.
- Several hassles related to HDBs / splayed tables.
- Suggestions on how attributes / link columns can be meaningful in Python?
- Will be easier if we have a script to generate sample HDB that makes the issues reproducible.
- Start with a finos.q wrapper for pyq.q that detects HDB tables and reformulates as functional form of select.
- Will check with Aaron Williamson whether can fork Bottleneck which is licensed under BSD-2.
- It's what Pandas uses for NaN handling. Create a variant that also understands kdb nulls and infinities.
- https://pypi.org/project/Bottleneck/
- https://github.com/pydata/bottleneck/issues/193
(5 min) Vaex mentioned both by AquaQ and Kx.
- https://www.aquaq.co.uk/q/comparing-columnar-data-formats-arrow-vaex-and-kdb/
- https://www.linkedin.com/pulse/python-data-analysis-really-simple-ferenc-bodon-ph-d-/
- Vaex currently V3.0.0. V4.0.0 currently in Alpha.
- https://pypi.org/project/vaex/
- https://github.com/vaexio/vaex/releases/tag/core-v4.0.0-alpha.9
- No push from users as of yet. Revisit later.
(5 min) BigQuery
- Some looking at hybrid model for 2021 : shallow history kdb / deep history BigQuery.
- Was hoping for transfers of gigabytes / second from cloud provider, but seems to max out at 300 megabytes / second.
- Small tables don't work well due to slow metadata queries.
(5 min) AOB, Q&A, Adjourn.
- Stories about Lustre filesystem for kdb on premises and in cloud welcome.
- Tools for migrating to kdb+ 4.0?
- Interoperability with Hadoop?
- Utilities for accessing REST data sources?
- Abstraction that might also work for on-disk and in-memory data?

WebEx info

Webex

https://finos.webex.com/finos/j.php?MTID=mfc9190e818fd85f9e3ece5166f23d920

Dial-in and Project Materials

US: +1-415-655-0003 US Toll
UK: +44-20319-88141 UK Toll
Access code: 663 746 304
GitHub Repos:
- https://github.com/finos/kdb
- https://github.com/finos/kdb-studio
Wiki Page:
- https://finosfoundation.atlassian.net/wiki/spaces/DT/pages/329383945/kdb+Project
KDB Metrics on FINOS
- https://metrics.finos.org/goto/033b15c80f0d08cab70f40f4358932f6

brooklynrob commented 3 years ago

I am here.

brooklynrob commented 3 years ago

Sounds like decision was made to permanently move time to 9:30am ET, correct @alvin-c-shih? (FYI @mcleo-d @aitana16 @maoo)

adavies42 commented 3 years ago

Aaron Davies, Morgan Stanley.

brooklynrob commented 3 years ago

Here are the people on the call

tbel commented 3 years ago

Tim Beletskiy, JPMC (logged in anyway)

alvin-c-shih commented 3 years ago

Sounds like decision was made to permanently move time to 9:30am ET, correct @alvin-c-shih? (FYI @mcleo-d @aitana16 @maoo)

Yes. Thanks!

brooklynrob commented 3 years ago

Here's the issue I crated related to evaluating how to use/incorporate Bottleneck within the Kdb project --> https://github.com/finos/kdb/issues/37

agitana commented 3 years ago

Sounds like decision was made to permanently move time to 9:30am ET, correct @alvin-c-shih? (FYI @mcleo-d @aitana16 @maoo)

Yes. Thanks!

Thanks, I've adjusted the calendar invite and webex details accordingly. The next meeting will be on January 4th at 9:30am ET. / cc @brooklynrob @alvin-c-shih

finos / kdb