2021-01-04 - KDB Project Call Meeting Minutes

[alvin-c-shih]

Date

Monday, January 4th, 2021 - 9:30am EST / 2:30pm BST

// First Monday of every month

Attendees

Fullname	Affiliation	GitHub Username
James McLeod	FINOS	@mcleo-d
Aaron Davies	Morgan Stanley	@adavies42
Jonny Press	AquaQ Analytics	@jonnypress
Peter Gyorok	Morgan Stanley	@gyorokpeter
Timofey Beletskiy	JP Morgan
Alvin Shih	Morgan Stanley	@alvin-c-shih
Latchezar Nikolov	Morgan Stanley	@latchezar-nikolov
Gary Davies	AquaQ Analytics	@drgdavies

Agenda

• (5 min) Convene, roll call, welcome new people.
• (5 min) Approve previous meeting minutes.
  • Approved by acclamation.
• (5 min) Kdb Studio updates?
  • Grouped Undo feature involved hacking on the old NetBeans code included with Kdb Studio.
  • Could use some testing.
  • Might be worth looking for latest NetBeans library.
• (5 min) dep.q / q/ reorganization complete.
  • Now easier for modules to depend on each other.
  • "try2" hook selectively allows protected eval to be disabled.
  • https://github.com/finos/kdb/blob/master/q/module/include.q#L20
  • Local to module instead of global to isolate its effect.
  • (5 min) PyQ utilities?
  • There are multiple ways to convert between DataFrames and q tables.
  • Some ways are much faster than others.
  • Will try to get a fast implementation contributed at some point.
• (5 min) kdb+ 4.0 migration utility ideas.
  • Really more a matter of any backwards-incompatible file format change.
  • 2.8 => 3.0, 3.5 => 3.6
  • Kx has a proof-of-concept HDB rewriter it gives to some clients.
  • Probably not production quality.
  • If code appends to files, problems can arise later.
  • General utility to update q code too much work.
  • Especially hard for 2.8 => 3.0 and many have already made that migration.
  • kdb 4.0 behaves differently with duplicate column names?
  • Seems worth developing a integration test suite for one's system to be ready for any future NUCs.
• (5 min) Prototype REST service to develop against?
  • Nothing available as of yet.
  • Jonathan McMurray has wrappers for q HTTP which could be helpful developing connectors to REST services.
  • https://github.com/jonathonmcmurray/reQ
  • Some limitations. See comments.
  • Need to put in a feature request to Kx to fix this limitation mentioned here:
  • https://code.kx.com/q/ref/dotq/#qhg-http-get
  • "N.B. HTTPS is not supported across proxies which require CONNECT." which is a problem for the kinds of proxies used by financial services firms.
• (5 min) kdb-related cloud benchmarks?
  • One caller in the midst of synthetic benchmarks of RDB on AWS.
  • STAC M3 Shasta appears to be a benchmark of this type:
  • https://stacresearch.com/news/2019/05/02/KDB190403 (Azure in this case)
• (5 min) AOB, Q&A, Adjourn.

WebEx info

Webex

• https://finos.webex.com/finos/j.php?MTID=mfc9190e818fd85f9e3ece5166f23d920

Dial-in and Project Materials

• US: +1-415-655-0003 US Toll
• UK: +44-20319-88141 UK Toll
• Access code: 663 746 304
• GitHub Repos:
  • https://github.com/finos/kdb
  • https://github.com/finos/kdb-studio
• Wiki Page:
  • https://finosfoundation.atlassian.net/wiki/spaces/DT/pages/329383945/kdb+Project
• KDB Metrics on FINOS
  • https://metrics.finos.org/goto/033b15c80f0d08cab70f40f4358932f6

[mcleo-d]

Hello 👋

[gyorokpeter]

hello

[alvin-c-shih]

Greetings!

[jonnypress]

hello

[adavies42]

hello

[latchezar-nikolov]

hello

[mcleo-d]

The following repo was shared on the call by @jonnypress:

reQ

HTTP requests library in kdb+/q

kdb+ has built in functions for HTTP requests such as .Q.hg (GET) and .Q.hp(POST). However, these functions are somewhat limited by several factors. For example, using these functions you cannot supply custom HTTP headers within the requests (for example, authorization tokens that are required by many APIs, a user agent etc.). In addition, in case of an HTTP redirect response, .Q.hg/.Q.hp will fail.

https://github.com/jonathonmcmurray/reQ

[mcleo-d]

@adavies42 shared the following link on the call for the following point noted on the page N.B. HTTPS is not supported across proxies which require CONNECT.

https://code.kx.com/q/ref/dotq/#qhg-http-get

[drgdavies]

Hello