search

singer-io / tap-s3-csv

GNU Affero General Public License v3.0
27 stars 52 forks source link

More expressive error message for invalid regex #19

Closed timvisher closed 5 years ago

timvisher commented 5 years ago

Motivation

Errors with the regex were not immediately obvious to an end user.

timvisher commented 5 years ago 
2019-04-05 00:03:31,687Z    tap - CRITICAL ('search_pattern for table `t_tran_log_prod` is not a valid regular expression. See https://docs.python.org/3.5/library/re.html#regular-expression-syntax', '*.csv')
2019-04-05 00:03:31,688Z    tap - Traceback (most recent call last):
2019-04-05 00:03:31,689Z    tap -   File "/usr/local/share/virtualenvs/tap-s3-csv/bin/tap-s3-csv", line 11, in <module>
2019-04-05 00:03:31,689Z    tap -     load_entry_point('tap-s3-csv', 'console_scripts', 'tap-s3-csv')()
2019-04-05 00:03:31,689Z    tap -   File "/usr/local/share/virtualenvs/tap-s3-csv/lib/python3.5/site-packages/singer/utils.py", line 192, in wrapped
2019-04-05 00:03:31,690Z    tap -     return fnc(*args, **kwargs)
2019-04-05 00:03:31,690Z    tap -   File "/opt/code/tap-s3-csv/tap_s3_csv/__init__.py", line 79, in main
2019-04-05 00:03:31,690Z    tap -     do_discover(args.config)
2019-04-05 00:03:31,691Z    tap -   File "/opt/code/tap-s3-csv/tap_s3_csv/__init__.py", line 18, in do_discover
2019-04-05 00:03:31,691Z    tap -     streams = discover_streams(config)
2019-04-05 00:03:31,691Z    tap -   File "/opt/code/tap-s3-csv/tap_s3_csv/discover.py", line 8, in discover_streams
2019-04-05 00:03:31,692Z    tap -     schema = discover_schema(config, table_spec)
2019-04-05 00:03:31,692Z    tap -   File "/opt/code/tap-s3-csv/tap_s3_csv/discover.py", line 13, in discover_schema
2019-04-05 00:03:31,692Z    tap -     sampled_schema = s3.get_sampled_schema_for_table(config, table_spec)
2019-04-05 00:03:31,693Z    tap -   File "/opt/code/tap-s3-csv/tap_s3_csv/s3.py", line 82, in get_sampled_schema_for_table
2019-04-05 00:03:31,693Z    tap -     samples = [sample for sample in sample_files(config, table_spec, s3_files_gen)]
2019-04-05 00:03:31,693Z    tap -   File "/opt/code/tap-s3-csv/tap_s3_csv/s3.py", line 82, in <listcomp>
2019-04-05 00:03:31,693Z    tap -     samples = [sample for sample in sample_files(config, table_spec, s3_files_gen)]
2019-04-05 00:03:31,694Z    tap -   File "/opt/code/tap-s3-csv/tap_s3_csv/s3.py", line 146, in sample_files
2019-04-05 00:03:31,694Z    tap -     for s3_file in itertools.islice(s3_files, max_files):
2019-04-05 00:03:31,694Z    tap -   File "/opt/code/tap-s3-csv/tap_s3_csv/s3.py", line 167, in get_input_files_for_table
2019-04-05 00:03:31,695Z    tap -     pattern) from None
2019-04-05 00:03:31,696Z    tap - ValueError: ('search_pattern for table `t_tran_log_prod` is not a valid regular expression. See https://docs.python.org/3.5/library/re.html#regular-expression-syntax', '*.csv')
2019-04-05 00:03:31,745Z   main - INFO Tap exited abnormally with status 1
2019-04-05 00:03:31,747Z   main - WARNING Not publishing metrics because some environment variables are not configured
2019-04-05 00:03:31,747Z   main - INFO No tunnel subprocess to tear down
2019-04-05 00:03:31,748Z   main - INFO Exit status is: Discovery failed with code 1 and error message: "('search_pattern for table `t_tran_log_prod` is
not a valid regular expression. See https://docs.python.org/3.5/library/re.html#regular-expression-syntax', '*.csv')".
timvisher commented 5 years ago

Merged https://github.com/singer-io/tap-s3-csv/commit/640cf8327cc7586b43e536754911a0b89942af56