dolphindb / DolphinDBPlugin

Apache License 2.0
49 stars 33 forks source link

[New Feature] Development List TO DO #20

Open peeyee opened 1 year ago

peeyee commented 1 year ago

To do list for new Contributors

Contributions are welcome, here is a list for new contributors:

No | Plugin | Content | Priority | Difficulty | Contributer | Status | Issue/PR -- | -- | -- | -- | -- | -- | -- | -- 1 | clickhouse |   | low | medium |   |   |   2 | hdf5 | add docs about the perfect way of loading data and use cases. | medium | low |   |   |   3 | httpClient | support for windows | high | medium |   |   |   4 | kafka | support for kerberos | low | medium |   |   |   5 | kafka | support for windows | high | medium |   |   |   6 | kdb | support for using loadTextEx | medium | medium |   |   |   7 | kdb | support for reading  compressed data of type q IPC (1), snappy (3), L4Z (4) | high | high |   |   |   8 | mqtt | support for arm | low | medium |   |   |   9 | mqtt | publish stream engine calculation results directly to the mqtt server | medium | medium |   |   |   10 | mysql | add docs about return type | medium | low |   |   |   11 | mysql | support for type blob | high | high |   |   |   12 | mysql | support for type decimal | high | high |   |   |   13 | oracle |   | high | high |   |   |   14 | parquet | support for reading column of  type repeated | high | high |   |   |   15 | parquet | support for windows | high | medium |   |   |   16 | signal | support for rfft, irfft | high | high |   |   |   17 | xgboost | upgrade the library of xgboost to version 1.7 | medium | medium |   |   |   18 | zip | support for arm | low | medium |   |   |   19 | zip | support for compressing folders | high | medium |   |   |  
FlyingOE commented 1 year ago

Item No.6 kdb+'s tasks: Is it referring to direct file access (similar to loadFile), bypassing q process (similar to loadTable)?

peeyee commented 1 year ago

Yes, the way of loadTable/File is totally loading the data into DolphinDB's in-memory table. Thus comes a problem if we do not have enough memory,especially for community users. So we want to have a method loadTextEx that directly loading data into DolphinDB's DFS table, meanwhile reduce the memory usage, for example, each batch costs 512MB.