2023-02-13 - KDB Project Call Meeting Minutes

alvin-c-shih commented 1 year ago

Date

Monday, February 13, 2023 - 9:30am ET / 2:30pm UK

// Second Monday of every month

Attendees

Fullname	Affiliation	GitHub Username
Richard Dunn	Anaconda	@RichardDunn23
Alvin Shih	Morgan Stanley	@alvin-c-shih
James McLeod	FINOS	@mcleo-d
Fara Manjili	Anaconda	@fmanjili
Peter Gyorok	KX Systems	@gyorokpeter
Ross Duffy	Morgan Stanley	@rdffy
Connor Gervin	KX Systems
Ian O'Dwyer	KX Systems	@iodwyer
Callum McCrea	Morgan Stanley

Meeting notices

FINOS Project leads are responsible for observing the FINOS guidelines for running project meetings. Project maintainers can find additional resources in the FINOS Maintainers Cheatsheet.
All participants in FINOS project meetings are subject to the LF Antitrust Policy, the FINOS Community Code of Conduct and all other FINOS policies.
FINOS meetings involve participation by industry competitors, and it is the intention of FINOS and the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws. Please contact legal@finos.org with any questions.
FINOS project meetings may be recorded for use solely by the FINOS team for administration purposes. In very limited instances, and with explicit approval, recordings may be made more widely available.

Agenda

(5 min) Convene, roll call, welcome new people.
[ ] Display FINOS Antitrust Policy summary slide
[ ] Review Meeting Notices (see above)
(5 min) Approve previous meeting minutes.
- Approved by acclamation.
(5 min) kdb Studio Updates
- PC hang when restoring tabs not easily reproduced.
- Perhaps an interaction with other software or shared drives.
- Users provided with instructions on how to capture diagnostic info.
(20 min) Anaconda for kdb Ecosystem.
- https://www.anaconda.com/blog/anaconda-and-kx-systems-partner-to-deliver-kdb-database-system-and-related-machine-learning-libraries
- Currently base q package available.
- KX pushed new binaries in December, including Mac ARM(!).
- Nick Psaris lectures at CMU and recommends his students use Anaconda to install q.
- Variety of artifacts could be published to jump start new developers:
- KX Developer
- Runnable Ticker Plant
- KX Accelerators (examples)
- KX Dashboards
- Jupyter Notebooks
- Some review of how financial services didn't participate much in OSS until FINOS.
- Discussion on whether packages need to be built under Anaconda.
- Current focus is on script-based libraries that don't need building.
- Seems like a good fit since Anaconda's already used for Python and KX has various Python integration offerings.
- More important to have a central repository and easy-to-install packages.
(5 min) verbose.q starting point: https://github.com/finos/kdb/pull/87
- PyKX has a Python abstraction layer along similar lines.
- See about how to create consistency for ease of learning.
- Newbie-friendly functions like readcsv instead of 0:.
- Purists in the kdb community may be resistant.
- Note that q itself is syntactic sugar on top of k.
- Would confer the ability to distinguish read-only operations from others which could impact stability / security.
- ? operator can be used for select and symbol enumeration which is very different!
- https://github.com/finos/kdb/blob/main/enterprise-best-practices/enterprise-kdb-vs-small-team-kdb.md
- Could write in verbose form and then use search / replace for performance-critical code.

(15 min) "Thinking Like Transformers" / RASPy analog for k/q ?

https://arxiv.org/pdf/2106.06981.pdf
https://srush.github.io/raspy/
Challenges - implement: sort, moving average, moving volatility.
Example of analyzing a Transformers Neural Network that "learned" how to do modular addition (as a rotation!):
https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking#Modular_Addition

To learn about the Transformers model this series is highly recommended: https://karpathy.ai/zero-to-hero.html

// The primitives of RASP (Restricted Access Sequence Processing Language): 
tok:{x}
ind:{til count x}
len:{(count x)#count x}  // no atoms in RASP
sel:{if[0>type x;x:(count y)#x];if[0>type y;y:(count x)#y];x z\:/:y}
agg:{0^avg each y where each x}

// Higher level operations expressible with the primitives:
selector_width:{sum each x y}
Flip:{sel[ind x;len[x]-ind[x]+1;=]}
rev:{agg[Flip x;tok x]}
same_tok:{sel[tok x;tok x;=]}
select_all:{sel[(count x)#1;(count x)#1;=]}

// Example output:
q)rev"foo"
111 111 102f
q)"c"$rev"foo"
"oof"
q)Flip"foo"
001b
010b
100b
q)tok"foo"
"foo"
q)ind"foo"
0 1 2

def sort (vals ,keys , assume_bos=False ) {
smaller = select (keys ,keys ,<) or
( select (keys ,keys ,==) and
  select ( indices , indices ,<) ) ;
num_smaller =
selector_width ( smaller ,
    assume_bos=assume_bos ) ;
target_pos = num_smaller if
            not assume_bos else
(0 if indices==0 else ( num_smaller +1) ) ;
sel_new =
select ( target_pos , indices ,==) ;
sort = aggregate ( sel_new , vals ) ;
}

(5 min) AOB, Q&A, Adjourn.
- https://github.com/finos/community/issues/231 👀
- Reminder that you may need to supply a PAT (Personal Authentication Password) when pushing and amend the commit to use your GitHub anonymized ID.
- https://docs.github.com/en/enterprise-server@3.4/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token
- https://stackoverflow.com/questions/43378060/meaning-of-the-github-message-push-declined-due-to-email-privacy-restrictions
- Check with your local GitHub gurus about additional steps required for corporate environments (location of tools, proxy settings, etc.).