Configure Log Level From CLI

[DarshanSP19]

Adding an argument --log-level to configure Log Level for current execution in CLI.

Partially resolves #135 for weather-mv. Same can be done for weather-dl and weather-sp.

Also removing a file level logger.setLevel(..) calls as package level log severity is configured in pipeline.py.

[google-cla[bot]]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[DarshanSP19]

We have 2 log at debug level. logger.debug('Created the BigQuery table with schema...') logger.debug(f'\n{pformat(table_schema)}')

Running the pipeline with --log-level 2 (INFO log level).

INFO:loader_pipeline.sinks:opened dataset size: 3122828
INFO:loader_pipeline.bq:Inferring schema from data.
INFO:loader_pipeline.util:target data_vars empty; using whole dataset; size: 3122828
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['true']
INFO:root:Default Python SDK image for environment is apache/beam_python3.8_sdk:2.40.0

Running the pipeline with --log-level 3 (DEBUG log level).

INFO:loader_pipeline.sinks:opened dataset size: 3122828
INFO:loader_pipeline.bq:Inferring schema from data.
INFO:loader_pipeline.util:target data_vars empty; using whole dataset; size: 3122828
DEBUG:loader_pipeline.bq:Created the BigQuery table with schema...
DEBUG:loader_pipeline.bq:
[SchemaField('d2m', 'FLOAT64', 'NULLABLE', None, (), None),
 SchemaField('u10', 'FLOAT64', 'NULLABLE', None, (), None),
 SchemaField('v10', 'FLOAT64', 'NULLABLE', None, (), None),
 SchemaField('longitude', 'FLOAT64', 'NULLABLE', None, (), None),
 SchemaField('latitude', 'FLOAT64', 'NULLABLE', None, (), None),
 SchemaField('time', 'TIMESTAMP', 'NULLABLE', None, (), None),
 SchemaField('data_import_time', 'TIMESTAMP', 'NULLABLE', None, (), None),
 SchemaField('data_uri', 'STRING', 'NULLABLE', None, (), None),
 SchemaField('data_first_step', 'TIMESTAMP', 'NULLABLE', None, (), None),
 SchemaField('geo_point', 'GEOGRAPHY', 'NULLABLE', None, (), None)]
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['true']
INFO:root:Default Python SDK image for environment is apache/beam_python3.8_sdk:2.40.0

[alxmrs]

Thanks! Sorry to ask for one more thing: Can you apply this setting to the two other tools? weather-dl & weather-sp?

[DarshanSP19]

@alxmrs Can you please re-review the changes. As you asked I've added the same configuration for weather-dl and weather-sp.