2023-07-10 - KDB Project Call Meeting Minutes

alvin-c-shih commented 12 months ago

Date

Monday, July 10, 2023 - 9:30am ET / 2:30pm UK

// Second Monday of every month

Attendees

Fullname	Affiliation	GitHub Username
James McLeod	FINOS	@mcleo-d
Alvin Shih	Morgan Stanley	@alvin-c-shih
Aaron Davies	Morgan Stanley	@adavies42
Peter Gyorok	KX Systems	@gyorokpeter
Gary Davies	Data Intellect	@drgdavies
Ross Duffy	Morgan Stanley	@rdffy
Rimma Perelmuter	FINOS
Simon Frazer	Data Intellect	@swfrazer
Connor Gervin	KX Systems	@Connor-Gervin

Meeting notices

FINOS Project leads are responsible for observing the FINOS guidelines for running project meetings. Project maintainers can find additional resources in the FINOS Maintainers Cheatsheet.
All participants in FINOS project meetings are subject to the LF Antitrust Policy, the FINOS Community Code of Conduct and all other FINOS policies.
FINOS meetings involve participation by industry competitors, and it is the intention of FINOS and the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws. Please contact legal@finos.org with any questions.
FINOS project meetings may be recorded for use solely by the FINOS team for administration purposes. In very limited instances, and with explicit approval, recordings may be made more widely available.

Agenda

(5 min) Convene, roll call, welcome new people.
[ ] Display FINOS Antitrust Policy summary slide
[ ] Review Meeting Notices (see above)
(5 min) Approve previous meeting minutes.
- Approved by acclamation.
(30 min) KX Con 23 Discussion Continued
- KX Con [23] Recordings
- https://kx.com/kx-con-23-recordings/
- AWS FinSpace
- https://kx.com/videos/kx-con-23-amazon-finspace-with-managed-kdb-insights/
- Provides an environment for migrating one's traditional Tick Architecture to AWS
  - Ticker Plant, RDBs, HDBs
  - Bring your own code.
  - Integrates with Amazon APIs.
  - .z baseline implementations
  - Everything starts in S3.
  - Hydrates disks as compute is spun up.
  - Auto scaling.
  - Can specify what to cache.
- One caller mentioned that their current PoC is kdb Insights on Azure.
  - SecArch has not looked at AWS FinSpace.
- Azure Managed kdb Insights has a different audience
  - Azure more targeted at Python developers.
  - FinSpace more about shifting existing kdb plant to the cloud.
- Scaling Bottlenecks? eg. Load Balancer RAM?
  - 20 people on AWS side who have kdb experience.
  - Feeding back ideas to AWS Core.
  - KX Insights has a streaming load balancer in Java.
- PyKX on Snowflake
- https://kx.com/videos/kx-con-23-kx-and-snowflake/
- NYSE TAQ available on Snowflake
  - Snowpark avoids having to export data from provider to consumer.
- Query into DataFrames.
  - q.so queries against the DataFrames.
  - Then converts back to Pandas.
- Can be 80x faster on certain algorithms.
  - Network overhead can drastically reduce effective throughput.
  - Need to be thoughtful on shifting the balance.
- Large RAM available on Snowflake nodes?
- May have to partition and distribute over tickers, etc.
- Benchmarks for AWS FinSpace to assure that it's competitive with on-prem?
- Being investigated.
- PyKX on Showflake sounds great for systems that consume from numerous vendors.
- MBS files are free to download but not distribute.
  - eg. Millions and millions of single family loans for the last 20 years.
- Would avoid having to develop own ingest, reconciliation, back fill, jobs.
  - Could probably do with lighter data quality checks.
- Vendors in the space: S&P, BlackKnight, Bloomberg, CoreLogic, MSCI, Markit, Refinitiv, etc.
- Allows users to have more direct control over query plan.
- Snowflake data mostly T+1.
(5 min) Zenith SIG and LLMs.
- https://github.com/finos/zenith/issues/33
- Next Zenith SIG call is on July 13th at 10 AM EDT.
(5 min) AOB, Q&A, Adjourn.
- A few attendees recommended Erin Stanton's talk at KX Con.
- https://kx.com/videos/kx-con-23-accelerating-machine-learning-with-kx/

WebEx info

Webex

https://finos.webex.com/finos/j.php?MTID=me61719ca3c3295a7cc76762d78392bc3

Dial-in and Project Materials

US: +1-415-655-0003 US Toll
UK: +44-20319-88141 UK Toll
Access code: 2704 417 0390
GitHub Repos:
- https://github.com/finos/kdb
- https://github.com/finos/kdb-studio
Wiki Page:
- https://finosfoundation.atlassian.net/wiki/spaces/DT/pages/329383945/kdb+Project
KDB Metrics on FINOS
- https://metrics.finos.org/goto/033b15c80f0d08cab70f40f4358932f6

adavies42 commented 12 months ago

hello!

gyorokpeter commented 12 months ago

Peter Gyorok / KX

mcleo-d commented 12 months ago

James McLeod / FINOS

alvin-c-shih commented 12 months ago

Alvin Shih / Morgan Stanley

drgdavies commented 12 months ago

Gary Davies / Data Intellect

swfrazer commented 12 months ago

Simon Frazer / Data Intellect

rdffy commented 12 months ago

Ross Duffy / Morgan Stanley

Connor-Gervin commented 12 months ago

Connor Gervin / KX

jshinonome commented 11 months ago

Hi @alvin-c-shih I would like represent kuki, the K Ultimate pacKage Installer in the next meeting. Do you mind allocate 20mins for this?

gyorokpeter commented 11 months ago

@jshinonome I would certainly be interested, although it's an unfortunate name choice for Hungarian speakers.

jshinonome commented 11 months ago

@jshinonome I would certainly be interested, although it's an unfortunate name choice for Hungarian speakers.

Ah, naming is so difficult... please think it as cookie

alvin-c-shih commented 11 months ago

I’ll definitely slot into the agenda!

Thanks!

ACS

From: Jo Shinonome @.> Sent: Thursday, August 03, 2023 5:39 AM To: finos/kdb @.> Cc: Shih, Alvin (Technology) @.>; Mention @.> Subject: [EXTERNAL] Re: [finos/kdb] 2023-07-10 - KDB Project Call Meeting Agenda (Issue #94)

@jshinonomehttps://github.com/jshinonome I would certainly be interested, although it's an unfortunate name choice for Hungarian speakers.

Ah, naming is so difficult... please think it as cookie

— Reply to this email directly, view it on GitHubhttps://github.com/finos/kdb/issues/94#issuecomment-1663638409, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AE7MRELEV42KBBY6MYS45DTXTNWTZANCNFSM6AAAAAA2C7R24E. You are receiving this because you were mentioned.Message ID: @.***>

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. By communicating with Morgan Stanley you acknowledge that you have read, understand and consent, (where applicable), to the Morgan Stanley General Disclaimers found at http://www.morganstanley.com/disclaimers/terms. The entire content of this email message and any files attached to it may be sensitive, confidential, subject to legal privilege and/or otherwise protected from disclosure.

jshinonome commented 11 months ago

Thank you @alvin-c-shih !

finos / kdb