Arp-G / csv2sql

A blazing fast fully-automated CSV to database importer
MIT License
54 stars 4 forks source link

Postgres support ETA? #3

Closed bvilnis closed 3 years ago

bvilnis commented 4 years ago

Hey,

Loving this project. It's an absolute gem. I can see in your readme that you have plans to add support for postgres and can see a branch for it that looks promising. Just wondering if there is an ETA on postgres support as this is exactly what I need to import time-series CSV from S3 buckets into TimescaleDB?

Keep up the top work!

Arpan-Kreeti commented 4 years ago

Hi thanks, for the support. I had started working on Postgres support but faced some issues when handling different data types. I haven't been able to work on this project lately, since I am busy with some other work. So I can't promise an ETA for now, but I will work on it whenever I have time.

The application has some issues when dealing with different data types like integer, date, etc for Postgres however it can handle strings. So, if for your use case your CSV only has strings it should work, I am attaching the binary script for that, however as I said it will only work for CSVs where the data will be treated as a string when trying to insert into the database.

So, for ids or date-time data, it will fail. Also, make sure the database is created before running it. I haven't tested it extensively so if you plan to use it, do it at your own risk.

You can run it by: ./csv2sql --source-csv-directory "/home/user/temp" --db-connection-string "postgres:username:password@localhost/db_name"

Lastly, if anyone has time and knowledge of elixir feel free to add Postgres support or additional features if you want, I will be happy to review and merge the PRs. 😊

csv2sql.zip

Arp-G commented 4 years ago

Hi, @bvilnis

I just added two new features to the application, which includes support for the PostgreSQL database and a web interface to use the application.

It would be very nice if you could have and look and provide some feedback.

Also, please feel free to report any bugs, suggestions, or advice.

Thanks again for giving this app a try. 😀