search

singlestore-labs / singlestoredb-python

Python SDK for the SingleStore database and workspace management APIs
Apache License 2.0
22 stars 17 forks source link

WIP: Load data from memory #16

Closed pmishchenko-ua closed 6 months ago

pmishchenko-ua commented 10 months ago

Using LOAD DATA internally can significantly speed up (around 2x) ingestion of the data which is in memory. The PR in its current form is not to be merged - the current API is not particularly user-friendly, so it should be discussed first.

kesmit13 commented 6 months ago

I used some of these ideas and implemented streams in LOCAL INFILE in another commit. It's available in v1.3.0 of the SDK. To use it, you specify :stream: as the LOCAL INFILE source and pass infile_stream= to the execute call. The infile stream can be an iterable, queue, io.TextIO or io.BytesIO object. If you want to use a queue or generator to push data while the INFILE statement is running, the execute should be run in a separate thread.