2021-05-03 - KDB Project Call Meeting Minutes

alvin-c-shih commented 3 years ago

Date

Monday, May 3rd, 2021 - 9:30am EST / 2:30pm BST (first Monday of every month)

Attendees

Fullname	Affiliation	GitHub Username
Aitana Myohl	FINOS	@aitana16
Erik Friis
Alvin Shih	Morgan Stanley	@alvin-c-shih
Aaron Davies	Morgan Stanley	@adavies42
Peter Gyorok	Morgan Stanley	@gyorokpeter
Alok Dutt	Citi

Meeting notices

FINOS Project leads are responsible for observing the FINOS guidelines for running project meetings. Project maintainers can find additional resources in the FINOS Maintainers Cheatsheet.
All participants in FINOS project meetings are subject to the LF Antitrust Policy, the FINOS Community Code of Conduct and all other FINOS policies.
FINOS meetings involve participation by industry competitors, and it is the intention of FINOS and the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws. Please contact legal@finos.org with any questions.
FINOS project meetings may be recorded for use solely by the FINOS team for administration purposes. In very limited instances, and with explicit approval, recordings may be made more widely available.

Agenda

(5 min) Convene, roll call, welcome new people.
[x] Display FINOS Antitrust Policy summary slide
[x] Review Meeting Notices (see above)
(10 min) Eric Friis looking for collaborators for a book, "Q in Depth".
- Has published in SIGAPL and IBM Journal.
- Highlights:
- formalization of a q grammar
- how ? (find) works on lists with mixed rank
- 450 pages so far?!?
- Hoping a co-author / significant contributor can help with:
  - readability / diagrams
  - exercises for the reader
  - adapt to k9
(5 min) Approve previous meeting minutes.
- Approved by acclamation.
(10 min) Discussion of qclone for scalability. https://github.com/finos/kdb/tree/master/q/qclone
- Use cases supported "out of the box".
- Contrast with https://code.kx.com/q/kb/multithreaded-input/ .
- Encourage more people to try it.
(5 min) Pure q implementation of Unique Set Size?
- For Linux, can be done with /proc .
- Not sure how to achieve with out a plugin on Windows.
- Needs a bit of cleanup before it's ready as a PR.
(10 min) Handler library?
- Stacking handlers as projections gets clumsy after a certain point.
- Awkward to remove handlers in the middle of the chain.
- A few implementations have been sketched out at Morgan Stanley.
- See if the ideas from those can be brought together.
(5 min) qPython64
- GitHub project for it is being set up.
- Would be good if Kx can provide details on how to decompress >2 GB payloads.
- If not, time to start experimenting.
- fastutils.pyx decompressor to stay in Pyrex or move to C++ / cppimport?
- https://github.com/exxeleron/qPython/blob/master/qpython/fastutils.pyx
- https://www.csse.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/About.html
- https://github.com/tbenthompson/cppimport
(5 min) Package repository for kdb modules / plugins? (vs CPAN, CRAN, Anaconda, etc.)
- If we want to avoid centralized infrastructure, could look at Emacs "Straight" package manager which downloads straight from GitHub:
- https://www.reddit.com/r/emacs/comments/6di4oy/straightel_source_based_package_manager_for_emacs/
(5 min) AOB, Q&A, Adjourn.

WebEx info

Webex

https://finos.webex.com/finos/j.php?MTID=mfc9190e818fd85f9e3ece5166f23d920

Dial-in and Project Materials

US: +1-415-655-0003 US Toll
UK: +44-20319-88141 UK Toll
Access code: 663 746 304
GitHub Repos:
- https://github.com/finos/kdb
- https://github.com/finos/kdb-studio
Wiki Page:
- https://finosfoundation.atlassian.net/wiki/spaces/DT/pages/329383945/kdb+Project
KDB Metrics on FINOS
- https://metrics.finos.org/goto/033b15c80f0d08cab70f40f4358932f6

adavies42 commented 3 years ago

hi

agitana commented 3 years ago

hi

ColinEberhardt commented 3 years ago

Hi @alvin-c-shih - there was a brief mention on the All Community call last week that the KDB group were interested in contribution towards a new UI? I think someone mentioned that KDB Studio is a NetBeans app and that you wanted something more modern?

We might be able to help with this - could you perhaps elaborate a little?

(I was going to attend the call today but it is a bank holiday In the UK!)

gyorokpeter commented 3 years ago

@ColinEberhardt Yes, the motivation is that the current editor backend is very dated so features commonly found in modern editors are missing. So either it could be migrated to the latest version of the NetBeans codebase or any other modern Java-based UI framework. This would affect the majority of the code in the project as it was basically forked from NetBeans at some point, and the KDB-specific parts are relatively small.

ColinEberhardt commented 3 years ago

Thanks for the info @gyorokpeter - does it have to be a Java-UI? We do very little Java-based UI, the vast majority is JavaScript / TypeScript, using React or Angular.

alvin-c-shih commented 3 years ago

Main points:

kdb+ client API. Would have to be able to port this without significant runtime performance penalty (say under 2x): https://github.com/finos/kdb-studio/blob/master/src/main/kx/c.java
Pluggable authentication mechanism. Each environment might use something slightly different: https://github.com/finos/kdb-studio/blob/master/src/main/studio/core/IAuthenticationMechanism.java
Export results to Excel in a way that doesn't trigger warnings.

As long as these requirements are met (plus being reasonably modern, of course), almost anything goes. :-)

gyorokpeter commented 3 years ago

Additionally the aim of this project is to be a desktop client, rather than another browser tab. This is the very reason I'm still using this program instead of Kx Analyst which is available with the enterprise KDB license (or Kx Developer if you don't have that). So the result of doing such a drastic change would be a completely new project and not a new version of KDB Studio.

ColinEberhardt commented 3 years ago

Thanks @gyorokpeter and @alvin-c-shih - I certainly don't want to push you down a technology route that you are not happy with. For a bit of context, I am the CTO of Scott Logic, a UK-based technology consultancy. I'm always on the look-out for interesting open source projects that our consultancy can potentially work on between client work. When I heard about this one in the All Community call it sounded interesting.

We're pretty flexible about which technology we use, but it does have to be something that reflects the type of work we typically do for a client.

With that in mind, is it worth us doing a little bit of a feasibility study to see whether using JavaScript / React / Electron (making it a desktop app), is a viable option - meeting the performance, auth and Excel integration requirements mentioned above?

psmulovics commented 3 years ago

Have you considered doing it as a vscode plugin? Thst way, you can use JavaScript and node to develop, and you still have a desktop app. Many similar things, let it be jupiter notebooks or python support, is already there, so it could be a nice addition. Also, distribution is provided free by the marketplace.

ColinEberhardt commented 3 years ago

Hi @psmulovics - completely agree, VSCode is a very good option for this sort of environment. Have you seen this?

https://github.com/jshinonome/vscode-q

@alvin-c-shih @gyorokpeter - before we invest any time in exploring this further (i.e. looking at technology options, planning / sizing the project), I do want to confirm that you think it is worthwhile? We don't want to invest time looking into an option that isn't right for this project!

finos / kdb