2021-04-05 - KDB Project Call Meeting Minutes

alvin-c-shih commented 3 years ago

Date

Monday, April 05, 2021 - 9:30am EST / 2:30pm BST

// First Monday of every month

FINOS Project leads are responsible for observing the FINOS guidelines for running project meetings. Project maintainers can find additional resources in the FINOS Maintainers Cheatsheet.
All participants in FINOS project meetings are subject to the LF Antitrust Policy, the FINOS Community Code of Conduct and all other FINOS policies.
FINOS meetings involve participation by industry competitors, and it is the intention of FINOS and the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws. Please contact legal@finos.org with any questions.
FINOS project meetings may be recorded for use solely by the FINOS team for administration purposes. In very limited instances, and with explicit approval, recordings may be made more widely available.

(5 min) Convene, roll call, welcome new people.
- [ ] Display FINOS Antitrust Policy summary slide
- [ ] Review Meeting Notices (see above)
(5 min) Approve previous meeting minutes.
- It was pointed out that the WebEx link on the Wiki needs updating.
- https://finosfoundation.atlassian.net/wiki/spaces/DT/pages/329383945/kdb+Project
(5 min) Wrapper of psutil (Python) in Appendix B of Minutes for 2021-03-01.
- Please test and see if you'd like to turn it into a full-fledged module for FINOS.
(15 min) Shall we hard fork qPython? https://github.com/exxeleron/qPython
- Only supports protocol 3 whereas we have users who want >2 GB payloads, which implies at least protocol 5.
- https://code.kx.com/q/basics/ipc/
- No appetite from the qPython maintainers.
- https://github.com/exxeleron/qPython/issues/58
- qPython and c.java read the entire payload into a buffer before decompressing.
- https://github.com/exxeleron/qPython/blob/master/qpython/fastutils.pyx
- Significant rework to decompress on the fly.
(10 min) Synthetic data for microbenchmarks?
- SciDB makes use of sample TAQ data.
- https://github.com/Paradigm4/TAQ
- See: https://www.nyse.com/market-data/historical/daily-taq
- Is there a particular subset of interest?
(5 min) AOB, Q&A, Adjourn.
- Ideas for future calls
- entitlements
- tools for setting up multiple cooperating q processes
- spreadsheet / MarkDown file comparing compression schemes

psutil / Unique Set Size
- Attendees expressed preference for implementation that doesn't depend on Python.
qPython hard fork
- Agreement to move forward.
- Suggested name for fork is qPython64.
- Aim for most tactical fix to get >2 GB payloads.
- Use cases that would benefit include:
- gateways that combine RDB+HDB results
- heavily traded names such as QQQ
benchmarking
- MIght want to look at TPC-DS.
- http://www.tpc.org/information/benchmarks5.asp
- STAC is aware of our interest and will discuss with members.
- Kdb's Data At Rest Encryption activated in a manner similar to on-disk compression
- https://code.kx.com/q/kb/dare/
```
-36!(`:testkey.key;"mypassword")
(`:etest;20;16;0)set 100000000?10000
system"ts max get`:etest"
```