search

lukasmartinelli / pgfutter

Import CSV and JSON into PostgreSQL the easy way
MIT License
1.33k stars 128 forks source link

Does not accept wildcards in file names #8

Open houshuang opened 8 years ago

houshuang commented 8 years ago

When I try to run pgfutter on all the csv files in my current directory, using *.csv as the file name specification, it only runs on the first file it encounters, and stops. I had to use gnu parallels to run it on all files (which it did beautifully, so not a big issue, but still counter-intuitive).

lukasmartinelli commented 8 years ago

Nice idea! Should be doable https://golang.org/pkg/path/filepath/#Glob

I always used a bash loop until now but that use case occurs quite often. But I would probably implement the matching files serially at first.

If I support parallel imports I would have to disable the progress indicator for multiple imports or switch the library to support multiple progress bars. I'm not sure whether it really improves performance to open another PostgreSQL connection and stream data.

houshuang commented 8 years ago

Thanks! No need to do serially, I'm just used to the syntax of parallel, that's why I used it instead of xargs etc. Anyway it's very quick (for my purposes).

On Sat, Dec 12, 2015 at 11:50 AM, Lukas Martinelli <notifications@github.com

wrote:

Nice idea! Should be doable https://golang.org/pkg/path/filepath/#Glob

I always used a bash loop until now but that use case occurs quite often. But I would probably implement the matching files serially at first.

If I support parallel imports I would have to disable the progress indicator for multiple imports or switch the library to support multiple progress bars. I'm not sure whether it really improves performance to open another PostgreSQL connection and stream data.

— Reply to this email directly or view it on GitHub https://github.com/lukasmartinelli/pgfutter/issues/8#issuecomment-164167661 .

http://reganmian.net/blog -- Random Stuff that Matters