search

turbot / steampipe-plugin-csv

Use SQL to instantly query data from CSV files. Open source CLI. No DB required.
https://hub.steampipe.io/plugins/turbot/csv
Apache License 2.0
19 stars 4 forks source link

Gather the errors and print logs before terminate the plugin. #43

Closed daeho-ro closed 1 year ago

daeho-ro commented 2 years ago

Discussed in the issue #31, and this is how I pass the malformed csv files. It may confuse to user why some tables are not created and should find the reason from the log file.

cbruno10 commented 2 years ago

Hey @daeho-ro , thanks for raising this PR!

I like the idea of making the CSV plugin easier to use, but as you had stated in this PR, I do have concerns that it may result in users not understanding why certain CSV files were not loaded. I think that before adding this behaviour, I would really like the ability to send warning messages in the terminal as per https://github.com/turbot/steampipe/issues/1090.

@kaidaguerre I'm not sure the level of effort required, so would you be able to estimate how long a feature like this would take? I think it would be useful not only in this case, but also in other plugins where being able to send a message in the terminal would be much more preferable to throwing errors/failing like we do today.

daeho-ro commented 2 years ago

I have added an option to skip those errors, so user can select it.

Furthermore, I have made another changes that is for the errors. Until now, when you met the error in the tableCSV, it is hard to find which CSV files are bad because the plugin only shows the error for a CSV. So, I collect the errors into a slice, and raise the error later which makes you can find all the errors from the log file.

Of course, the steampipe CLI can only shows an error and still need to be added the feature what you mention, but at least I want to add the logs for all bad CSV files.

cbruno10 commented 2 years ago

@daeho-ro I've merged in https://github.com/turbot/steampipe-plugin-csv/pull/42, so I'm not sure if we need this config argument and feature, as I believe most of the file loading errors people experience today are due to incompatible headers.

Are there other common cases you've encountered where you've had issues loading invalid CSVs?

daeho-ro commented 2 years ago

I have simplified the PR to be more convenient for debugging. There could be other cases such as the file is actually not a csv, or reading error and so on. This will help you to find which file is not suitable by running steampipe query just once.

github-actions[bot] commented 1 year ago

'This PR is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.'

github-actions[bot] commented 1 year ago

'This PR is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.'

github-actions[bot] commented 1 year ago

This PR was closed because it has been stalled for 90 days with no activity.