search

simonw / csvs-to-sqlite

Convert CSV files into a SQLite database
Apache License 2.0
879 stars 69 forks source link

Ignore lines are start and end of file #72

Open psychemedia opened 4 years ago

psychemedia commented 4 years ago

Lots of CSV files have informally declared metadata in the first few lines of the file, and footnotes at the end. It would be useful to have an explicit way of declaring that N lines at the start of the file and M lines at the end are to be ignored.

ryancheley commented 2 years ago

I was working with someone using datasette and caves-to-SQLite just yesterday and came across this exact problem. I wrote myself a note to write up an issue, but it seems it’s already here.

@simonw I can take a try at writing a fix for this if that’s something you think would be helpful. One thing I think may be blocking implementation though would be #69 as I think that the engine for importing the data uses pandas and the idea I have for a fix would involve that, BUT if the import part is going to switch to SQLite-utils then maybe this issue can be incorporate into that update?