search

dlt-hub / dlt

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
https://dlthub.com/docs
Apache License 2.0
2.26k stars 148 forks source link

Add sqlite as a dlt destination #1627

Open Vasilije1990 opened 1 month ago

Vasilije1990 commented 1 month ago

Feature description

sqlite can be added as dlt destination

Are you a dlt user?

Yes, I'm already a dlt user.

Use case

I need an in-memory database with async support

Proposed solution

Adding sqlite as a destination

Related issues

https://news.ycombinator.com/item?id=39527441

rudolfix commented 1 month ago

@Vasilije1990 (and anyone else interested) - is there anything special that you need:

  1. what does async support mean? we do not have async destinations right now. they operate in batch mode. you could write custom async destination (reverse ETL) but this is not probably what you want
  2. why SQL lite and not DuckDB?
  3. sql lite is so different from other engines that we planned to do this in #21 via sql alchemy. that would also support 50+ other destinations. the blocker is lack of resources and until now there was not much interest in supporting regular dbmses as destinations
Vasilije1990 commented 1 month ago
  1. DuckDB is not async, and I am using the destination as a metadata store for other services that use for example FastAPI Users as one
  2. FastAPI Users requires async to work
  3. That would work
rudolfix commented 2 weeks ago

should be solved by #21