gm3dmo / syslog-to-csv

Turn log files into data.
GNU General Public License v3.0
15 stars 7 forks source link

Append csv file to a sqlite table #19

Closed gm3dmo closed 1 year ago

gm3dmo commented 2 years ago

https://stackoverflow.com/questions/46981839/is-it-possible-to-import-a-csv-file-to-an-existing-table-without-the-headers-bei

gm3dmo commented 2 years ago
sqlite3 logs.db << EOF
.separator ","
.import --csv github-logs/unicorn.log.1.csv unicorn
.import --csv --skip 1 github-logs/unicorn.log.csv unicorn
EOF

returned:

Usage: .import FILE TABLE
Usage: .import FILE TABLE
gm3dmo commented 2 years ago

This was more successful for me:

sqlite3 logs.db << EOF
.separator ","
.import  github-logs/unicorn.log.1.csv unicorn
.import "|tail -n +2 github-logs/unicorn.log.csv" unicorn
EOF

Total count of lines:

$ wc -l github-logs/unicorn.log github-logs/unicorn.log.1
    6768968 github-logs/unicorn.log
    6851163 github-logs/unicorn.log.1
   13620131 total
[Bundle #85327] 85327 $ sqlite3 logs.db
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> select count() from unicorn;
13619869
gm3dmo commented 1 year ago

I had to add something which squishes invalid sql table names to valid names.