Bulk data import

[bluestreak01]

Summary

More often the not our users have an existing data in either existing database or proprietary data format. They would like to import this data efficiently and painlessly into QuestDB.

Currently QuestDB imposes hard requirement that user data has to be ordered before import. This is usually hard to achieve for very large data sets. Think 2TB or more, basically anything that does not fit RAM. The existing QuestDB ingestion mechanism (ILP) also leans on RAM to sort data.

Desired outcome

• Require user to export data as CSV
• Have no limits on CSV size and no absurd RAM requirements
• Import CSV into QuestDB quickly and efficiently without requiring data to be pre-sorted

[bluestreak01]

implementation detail: https://github.com/questdb/questdb/issues/2143