CSV Plugin Crashes if any files in path are in invalid format

[psolomon-opploans]

Describe the bug CSV Plugin Crashes if any files in path are in invalid format

Steampipe version (steampipe -v) failed to start plugin 'hub.steampipe.io/plugins/turbot/csv@latest': myfile.xlsx - Detailed Findings (1).csv header row has empty value in field 1

Plugin version (steampipe plugin list)

$ steampipe --version
steampipe version 0.16.0
OPL-M-PSOLOMON4:Downloads psolomon$ steampipe plugin list
failed to start plugin 'hub.steampipe.io/plugins/turbot/csv@latest': myfile.xlsx - Detailed Findings (1).csv header row has empty value in field 1
+--------------------------------------------------+---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Name                                             | Version | Connections                                                                                                                                                           |
+--------------------------------------------------+---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| hub.steampipe.io/plugins/turbot/aws@latest       | 0.74.1  | <LIST_REDACTED> |
|                                                  |         |  |
|                                                  |         |                                                                                                                                                        |
| hub.steampipe.io/plugins/turbot/csv@latest       | 0.3.2   |                                                                                                                                                                       |
| hub.steampipe.io/plugins/turbot/datadog@latest   | 0.1.0   | datadog                                                                                                                                                               |
| hub.steampipe.io/plugins/turbot/finance@latest   | 0.2.1   | finance                                                                                                                                                               |
| hub.steampipe.io/plugins/turbot/github@latest    | 0.19.0  | github                                                                                                                                                                |
| hub.steampipe.io/plugins/turbot/jira@latest      | 0.5.0   | jira                                                                                                                                                                  |
| hub.steampipe.io/plugins/turbot/net@latest       | 0.7.0   | net                                                                                                                                                                   |
| hub.steampipe.io/plugins/turbot/pagerduty@latest | 0.1.0   | pagerduty                                                                                                                                                             |
| hub.steampipe.io/plugins/turbot/slack@latest     | 0.8.0   | slack                                                                                                                                                                 |
| hub.steampipe.io/plugins/turbot/terraform@latest | 0.1.0   | terraform                                                                                                                                                             |
| hub.steampipe.io/plugins/turbot/whois@latest     | 0.5.0   | whois                                                                                                                                                                 |
| hub.steampipe.io/plugins/turbot/zoom@latest      | 0.4.0   | zoom                                                                                                                                                                  |
+--------------------------------------------------+---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+

To reproduce Create a CSV with an invalid header column (i.e. blank)

Expected behavior Instead of completely crashing the plugin, just print warnings of the bad files.

Additional context None at this time.

[cbruno10]

@psolomon-opploans Thanks for opening this issue! I agree, having the plugin fail to load on just 1 bad table is not a great experience. It would be much better if instead we warned you what files we failed to load and proceeded with the rest; however, I don't think there's a way we can print warnings or messages to the terminal today. Instead, we can send messages to the plugin logs, e.g., ~/.steampipe/logs/plugin-2022-08-28.log, but those feel a bit hidden.

I currently have an issue open to add support for these types of messages, but it's still open.

So based on your initial issue, one possible solution is that after the Steampipe session starts, valid CSV files will be loaded and invalid ones will not be, but there won't be any warnings or messages printed in the terminal indicating they were not loaded (these messages will be in the plugin logs at a warn or error level).

Interested in hearing your thoughts on the proposal above (feel free to add any other comments/questions you have as well), thanks!

[github-actions[bot]]

'This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.'

[gedw99]

https://github.com/turbot/steampipe/issues/1090 Looks like the best solution.

It should possibly the default that a generic logging API and debug API ( of sorts ) is designed to allow plugins to return errors etc to the cli and cloud gui ?

[daeho-ro]

@cbruno10 I prefer to filter the invalid tables because the plugin should be more stable. When I fix or remove the wrong csv, another warning or error occur next time again and again. To avoid this, the bast use case is that just move the specific csv to the empty folder and run query. That is really mess the use experience.

[cbruno10]

Hey @psolomon-opploans , we just released a new version, v0.5.0, which introduces a header config arg (thanks to @daeho-ro for the addition!). I believe this should help your use case, in particular if you leave the header config arg value as auto (which is the default), the CSV files you mentioned should now load in with generic column names, e.g., "a", "b".

If you're still seeing issues, feel free to re-open this issue. Thanks!