New Functionality

[jasraj]

• file:
  • New .file.replace (delete + move)
  • New optimised list element size function
  • New par.txt wrapper function
• rand: Random data generator for tables
• require: Support for slf4j-style parameterised messages in basic log messages

[jasraj]

.file.kdb.getLength

Performance comparison - single-typed lists (18 total, 1 per type) with 10 million elements in the list. ~99% performance improvement

Method	Compressed?	\ts Output
.file.kdb.getLength	NO	1 1216
count get	NO	439 138412688
.file.kdb.getLength	YES	2 667184
count get	YES	516 142575952

Code:

/ Random data doesn't compress well
q) .require.lib`rand;
q) { set[`$":/tmp/kdb-types/",x; 10000000?y] }./: flip (key;value)@\:enlist[" "] _ .rand.charTypes;
q) { set[(`$":/tmp/kdb-types-comp/",x),.compress.defaults`lz4hc; 10000000?y] }./: flip (key;value)@\:enlist[" "] _ .rand.charTypes

q) srcs:.file.listFolderPaths `$":/tmp/kdb-types";
q) compSrcs:.file.listFolderPaths `$":/tmp/kdb-types-comp";

q) (.file.kdb.getLength each srcs) = count each get each srcs
111111111111111111b

q) \ts count each get each srcs
439 138412688
q) \ts .file.kdb.getLength each srcs
1 1216

q)(.file.kdb.getLength each compSrcs) = count each get each compSrcs
111111111111111111b

q)\ts count each get each compSrcs
516 142575952
q)\ts .file.kdb.getLength each compSrcs
2 667184