2022-11-14- KDB Project Call Meeting Agenda

alvin-c-shih commented 1 year ago

Date

Monday, November 14, 2022 - 9:30am ET / 2:30pm UK

// Second Monday of every month

Attendees

Fullname	Affiliation	GitHub Username
James McLeod	FINOS	@mcleo-d
Aaron Davies	Morgan Stanley	@adavies42
Alvin Shih	Morgan Stanley	@alvin-c-shih
Peter Gyorok	KX Systems	@gyorokpeter
Jonny Press	AquaQ Analytics	@jonnypress
Ross Duffy	Morgan Stanley

Meeting notices

FINOS Project leads are responsible for observing the FINOS guidelines for running project meetings. Project maintainers can find additional resources in the FINOS Maintainers Cheatsheet.
All participants in FINOS project meetings are subject to the LF Antitrust Policy, the FINOS Community Code of Conduct and all other FINOS policies.
FINOS meetings involve participation by industry competitors, and it is the intention of FINOS and the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws. Please contact legal@finos.org with any questions.
FINOS project meetings may be recorded for use solely by the FINOS team for administration purposes. In very limited instances, and with explicit approval, recordings may be made more widely available.

Agenda

(5 min) Convene, roll call, welcome new people.
[ ] Display FINOS Antitrust Policy summary slide
[ ] Review Meeting Notices (see above)
(5 min) Approve previous meeting minutes.
- Approved by acclamation.
(5 min) Ideas for bringing together an ML Ops Project in FINOS.
- FINOS-BLUG joint event about AI & Open Source Ethics coming up.
- https://www.finos.org/featured-events/2022-12-01-blug-general-ai-open-source-ethics
- NatWest presented an AI/ML bot at meetup in Bristol.
- Synechron's Compliance Horizon Scanning incorporates ML.
- Will sync up with Goldman Sachs OSPO at OSFF.
(15 min) Discussion about HDBs partitioned on smaller intervals than days.
- Supported partitions inferred by length of directory name:
- date is 10 characters, month is 7 characters, year is 4 characters, anything else is a long int
- Examples of systems that aren't partitioned by date:
- FX ticks partitioned by hour.
- Risk system that makes a snapshot every 10 minutes.
- Equity tick volumes have increased since the early days of kdb+ 25+ years ago.
- Options tick volumes are huge and hard to manage whole day partitions.
- Question about whether a non-daily roll is driven by the ticker plant or independent of it.
- Current workarounds convert timestamps to long int, but it makes it harder for the users to query.
- Some hacking on q.k to support timestamp revealed:
- Colons in timestamps cause problems when trying to load.
- q can parse timestamps with the colons omitted.
- Should be a modest change to kdb to support this.
(10 min) Discussion about loading large files into RDB intra day.
- One example cites a table that requires 15 minutes to load.
- qclone could create a child process to parse input files and send to parent via IPC.
- Might want try splayed table or AnyMap tables that load quickly due to mmap.
- No easy a way for child to inject new table into parent memory.
(5 min) kdb+ Project gets a Project Expo booth at Open Source in Finance Forum NYC on December 8th.
- https://www.finos.org/hosted-events/2022-12-08-open-source-in-finance-forum-new-york
- https://events.linuxfoundation.org/open-source-finance-forum-new-york/register/
- Stop by if you're around.
(5 min) AOB, Q&A, Adjourn.
- No activity on kdb-studio in last 4 months.
- Some interest to hear about experiences with PyKX.