2023-03-13 - KDB Project Call Meeting Minutes

alvin-c-shih commented 1 year ago

Date

Monday, March 13, 2023 - 9:30am ET / 2:30pm UK

// Second Monday of every month

Attendees

Fullname	Affiliation	GitHub Username
Richard Dunn	Anaconda	@RichardDunn23
Alvin Shih	Morgan Stanley	@alvin-c-shih
Fara Manjili	Anaconda	@fmanjili
Peter Gyorok	KX Systems	@gyorokpeter
Ross Duffy	Morgan Stanley	@rdffy
Connor Gervin	KX Systems
Jonny Press	AquaQ Analytics	@jonnypress
Thomas Tran
James McLeod	FINOS	@mcleo-d
Ian O'Dwyer	KX Systems	@iodwyer
Aaron Davies	Morgan Stanley	@adavies42

Meeting notices

FINOS Project leads are responsible for observing the FINOS guidelines for running project meetings. Project maintainers can find additional resources in the FINOS Maintainers Cheatsheet.
All participants in FINOS project meetings are subject to the LF Antitrust Policy, the FINOS Community Code of Conduct and all other FINOS policies.
FINOS meetings involve participation by industry competitors, and it is the intention of FINOS and the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws. Please contact legal@finos.org with any questions.
FINOS project meetings may be recorded for use solely by the FINOS team for administration purposes. In very limited instances, and with explicit approval, recordings may be made more widely available.

Agenda

(5 min) Convene, roll call, welcome new people.
[ ] Display FINOS Antitrust Policy summary slide
[ ] Review Meeting Notices (see above)
(5 min) Approve previous meeting minutes.
- Approved by acclamation.
(10 min) Next steps to get more KX offerings on Anaconda.
- KX has conversations in flight with Anaconda.
- KX has an Anaconda channel from years ago
- See what from KX Insights can be published that way.
- Initial discussions are around publishing of KX artifacts which have an internal packaging system.
- A few on the call thought it would make for a more vibrant ecosystem if developers could build on that.
- dep.q has pluggable back end, which might smooth over difference between KX assumptions and IT infrastructure.
- Would be good to have at least 3 organizations participate in module loader discussions to ensure sufficient flexibility / reusability.
- Hope to get someone from KX Insights Core Team on future call to hear a bit more about their module loader.
- Questions from the Anaconda side about platforms KX customers are using.
- Many are familiar with Docker.
- Others looking to Podman as that's what RedHat 8 supports.
- James will have a call with Anaconda folks to see if other FINOS teams interested in engaging.
- See if it makes it easier to onboard larger FINOS offerings like Legend.
- Also see if Anaconda can get a chance to meet folks at KX Con.
- https://kx.com/events/kx-con-2023/
(10 min) Options for backing out failed DB maintenance.
- KX Insights has its own symlink "forest".
- Writes to temporary directory and then swaps to achieve some level of atomicity.
- KX Insights does not support appending to splayed tables.
- Doesn't seem as though anyone uses FUSE for this, but some will read up on it.
- https://thepythoncorner.com/posts/2017-02-27-writing-a-fuse-filesystem-in-python/
- Ragged table detection and full versioning didn't seem to be big concerns.
(5 min) verbose.q / PyKX alignment.
- PyKX has a shim on q.
- q functions and namespace visible in Python with a similar structure.
- Pandas-style API.
- Hope to get some PyKX developers on a future call.
(5 min) KX Insights care and feeding Q&A.
- Uploading non-time series data like product master / account master?
- Look for "reference table" in the docs:
  - https://code.kx.com/insights/1.2/platform/features/reference-data.html
  - Example shows splayed and partitioned table types.
  - Support for q binary / Anymap?
  - https://code.kx.com/q/releases/ChangesIn3.6/#anymap
    - "Symbol vectors/atoms are automatically enumerated against file## and de-enumerated (and therefore always copied) on access."
- Questions about controlling garbage collection in KX Insights.
- Compressed string columns create a lot of garbage that's not collected in -g 1 mode.
  - -g 1 weaker than .Q.gc[].
  - A -g 2 mode that is as strong as .Q.gc[] would be useful.
- Apparently 4.1t has new GC modes that can be specified by passing a long to .Q.gc[...], but it's not clear when / if it will be released.
  - Fate of 4.1t to be announced at KX Con in May.
(5 min) RASP (Restricted Access Sequence Processing Language) could be useful for studying AI security / safety.
- "Perplexity.ai prompt leakage"
- https://news.ycombinator.com/item?id=34482318
- "I think running simple string searches is a reasonable and cheap defense. Of course, the attacker can still request the prompt in French, or with meaningless emojis after every word, or Base64 encoded."
- Something resembling RASP for q will be presented at KX Con.
(5 min) AOB, Q&A, Adjourn.
- unzip.q PR coming soon.
- https://github.com/finos/kdb/commit/a4317d171c3f47f932e72c0f75a85e69e95ec252
- ChatGPT identified compression algorithm in c.java as LZ77 with 256 byte window.