Create BigQuery plugin

[amedhat3]

Create a wrapper around BigQuery, it invokes the query function and waits for the result to be returned. This plugin helps us to monitor the functional metrics of the applications, like conversion rate and successful searches, which can't be monitored from the application logs/metrics API, fixes https://github.com/zalando-zmon/zmon-worker/issues/115.

A decrypted service key need to be sent to the wrapper to be able to access BigQuery

Ran the tests using an external docker file, thanks for the hint (Alexey Ermakov)

---------- coverage: platform linux2, python 2.7.15-final-0 ----------
Name                                                                    Stmts   Miss  Cover   Missing
-----------------------------------------------------------------------------------------------------
zmon_worker_monitor/__init__.py                                             1      1     0%   1
zmon_worker_monitor/__main__.py                                             3      1    67%   4
zmon_worker_monitor/adapters/__init__.py                                    3      0   100%
zmon_worker_monitor/adapters/ibase_plugin.py                               11      2    82%   30, 38
zmon_worker_monitor/adapters/ifunctionfactory_plugin.py                    21      1    95%   25
zmon_worker_monitor/builtins/__init__.py                                    1      0   100%
zmon_worker_monitor/builtins/plugins/__init__.py                            1      0   100%
zmon_worker_monitor/builtins/plugins/appdynamics.py                       135      0   100%
zmon_worker_monitor/builtins/plugins/aws_common.py                          4      0   100%
zmon_worker_monitor/builtins/plugins/bigquery.py                           36      7    81%   65-109
zmon_worker_monitor/builtins/plugins/cassandra_wrapper.py                  43      1    98%   47
zmon_worker_monitor/builtins/plugins/checkldap.py                         218    155    29%   111, 127-148, 151, 154-159, 162-168, 175-188, 193-197, 211-249, 261-268, 271-295, 331-362, 388-399, 402-407
zmon_worker_monitor/builtins/plugins/cloudwatch.py                        132      0   100%
zmon_worker_monitor/builtins/plugins/counter.py                            43     25    42%   42-44, 49-51, 56-67, 72, 76-83
zmon_worker_monitor/builtins/plugins/datapipeline.py                       36      2    94%   60, 71
zmon_worker_monitor/builtins/plugins/distance_to_history.py                85     72    15%   15-29, 35-39, 47-63, 72-80, 87-94, 101-113, 122, 133-139, 143-168
zmon_worker_monitor/builtins/plugins/dns_.py                               38      2    95%   50, 58
zmon_worker_monitor/builtins/plugins/ebs.py                                31      1    97%   70
zmon_worker_monitor/builtins/plugins/elasticsearch.py                      90      8    91%   133, 176, 183, 196-211
zmon_worker_monitor/builtins/plugins/entities_wrapper.py                   65      6    91%   57, 67, 112-117
zmon_worker_monitor/builtins/plugins/history.py                            74     14    81%   59-97, 113, 127, 165, 170-179
zmon_worker_monitor/builtins/plugins/http.py                              218      5    98%   53-55, 296, 380
zmon_worker_monitor/builtins/plugins/jmx.py                                69     33    52%   45-52, 72-73, 87-88, 91-95, 98-121, 127-130
zmon_worker_monitor/builtins/plugins/kairosdb.py                           62      2    97%   126, 129
zmon_worker_monitor/builtins/plugins/kubernetes.py                        105      0   100%
zmon_worker_monitor/builtins/plugins/memcached.py                          49      8    84%   108-120
zmon_worker_monitor/builtins/plugins/mongodb.py                            28     11    61%   40-41, 44-56
zmon_worker_monitor/builtins/plugins/ping_.py                              18      7    61%   31-43, 47-48
zmon_worker_monitor/builtins/plugins/redis_wrapper.py                      65     20    69%   92, 95, 101, 104, 107, 110, 113, 116, 119, 122, 125, 155, 162-174
zmon_worker_monitor/builtins/plugins/s3.py                                 80      1    99%   115
zmon_worker_monitor/builtins/plugins/scalyr.py                            107      4    96%   179, 203-206
zmon_worker_monitor/builtins/plugins/sql_mysql.py                          96     14    85%   54, 136, 159-175
zmon_worker_monitor/builtins/plugins/sql_oracle.py                         91     67    26%   41-43, 47-53, 62-80, 83-84, 89-107, 112-127, 132-145
zmon_worker_monitor/builtins/plugins/sql_postgresql.py                    121     11    91%   169-171, 175-176, 208, 241-245
zmon_worker_monitor/builtins/plugins/tcp.py                                38     23    39%   34-35, 38-51, 54-58, 62-65
zmon_worker_monitor/builtins/plugins/time_.py                              28      0   100%
zmon_worker_monitor/builtins/plugins/whois_.py                             23     10    57%   32-36, 39-47
zmon_worker_monitor/builtins/plugins/zmon_.py                              26     10    62%   39-43, 50-56
zmon_worker_monitor/builtins/plugins/zomcat.py                            104     70    33%   64, 72-80, 84, 89-102, 107-124, 129-151, 163-167, 173-197
zmon_worker_monitor/emu_kombu.py                                           12      1    92%   34
zmon_worker_monitor/eventloghttp.py                                        21      1    95%   24
zmon_worker_monitor/flags.py                                               19      6    68%   40-43, 47, 59-60
zmon_worker_monitor/main.py                                                73     11    85%   49-52, 78, 82-83, 136-139, 145
zmon_worker_monitor/plugin_manager.py                                     189     40    79%   52, 60-65, 80, 82-85, 97-98, 152, 217-220, 228-231, 238-243, 252-256, 265-268, 288, 314
zmon_worker_monitor/process_controller.py                                 577     89    85%   67-71, 80, 89, 97, 100, 103, 106, 109, 112, 115, 118-129, 132, 135, 141, 324, 345, 359, 518-520, 527-529, 570, 652, 666-668, 678-679, 685-689, 694, 701, 710, 714-718, 721-722, 725-727, 730-733, 769, 836-838, 844-845, 855, 866-868, 880-883, 903-910, 919-923, 934-935, 951
zmon_worker_monitor/redis_context_manager.py                              138     36    74%   92-93, 105, 126-132, 144, 151-153, 159-160, 164, 167, 170, 173-180, 185, 192, 197-200, 214, 220-229
zmon_worker_monitor/rpc_client.py                                          59     41    31%   51, 54-57, 68, 72, 76, 96, 100-130, 135-149
zmon_worker_monitor/rpc_server.py                                          23      3    87%   44, 58-60
zmon_worker_monitor/rpc_utils.py                                           59     40    32%   28-30, 34, 38-42, 46, 50, 53, 57-77, 86, 99-116
zmon_worker_monitor/settings.py                                            13      1    92%   33
zmon_worker_monitor/settings_pro.py                                         3      0   100%
zmon_worker_monitor/tasks.py                                               20      5    75%   31-34, 38-40
zmon_worker_monitor/web_server/__init__.py                                  0      0   100%
zmon_worker_monitor/web_server/rest_api/__init__.py                         0      0   100%
zmon_worker_monitor/web_server/rest_api/api_v2.py                          71     39    45%   28, 45-51, 57-59, 88-98, 108-113, 121-130, 142-149, 155
zmon_worker_monitor/web_server/rest_api/commons.py                         20     10    50%   22-27, 31, 44-47
zmon_worker_monitor/web_server/rest_api/errors.py                          14      6    57%   12-15, 18, 27
zmon_worker_monitor/web_server/start.py                                    10      7    30%   8-11, 26-30
zmon_worker_monitor/web_server/web.py                                      25     17    32%   23, 27-33, 37-56
zmon_worker_monitor/worker.py                                              17      0   100%
zmon_worker_monitor/workflow.py                                           278     87    69%   71-96, 193, 209-210, 222-225, 227-229, 234-243, 276-279, 316-321, 402-403, 411-417, 425-440, 451-467, 470-473, 481-483, 509, 520-521, 537-539
zmon_worker_monitor/zmon_worker/__init__.py                                 0      0   100%
zmon_worker_monitor/zmon_worker/common/__init__.py                          0      0   100%
zmon_worker_monitor/zmon_worker/common/eval.py                             52      1    98%   192
zmon_worker_monitor/zmon_worker/common/http.py                              5      0   100%
zmon_worker_monitor/zmon_worker/common/mathfun.py                          36      2    94%   38, 49
zmon_worker_monitor/zmon_worker/common/time_.py                            36      0   100%
zmon_worker_monitor/zmon_worker/common/tracing.py                          12      4    67%   13-14, 18-19
zmon_worker_monitor/zmon_worker/common/utils.py                            75      4    95%   109-113
zmon_worker_monitor/zmon_worker/encoder.py                                 21      2    90%   29, 35
zmon_worker_monitor/zmon_worker/errors.py                                  42      4    90%   34, 39-40, 43
zmon_worker_monitor/zmon_worker/notifications/__init__.py                   0      0   100%
zmon_worker_monitor/zmon_worker/notifications/google_hangouts_chat.py      42      4    90%   88-91
zmon_worker_monitor/zmon_worker/notifications/hipchat.py                   47     35    26%   22-77
zmon_worker_monitor/zmon_worker/notifications/http.py                      62      9    85%   49-51, 61-63, 91-93
zmon_worker_monitor/zmon_worker/notifications/hubot.py                     39     26    33%   26-63, 68-75
zmon_worker_monitor/zmon_worker/notifications/mail.py                     116     25    78%   56-58, 69, 122, 133-143, 149-154, 160-163, 167, 183-184
zmon_worker_monitor/zmon_worker/notifications/notification.py              59     13    78%   18, 68, 79-93
zmon_worker_monitor/zmon_worker/notifications/opsgenie.py                  85      0   100%
zmon_worker_monitor/zmon_worker/notifications/pagerduty.py                 52      0   100%
zmon_worker_monitor/zmon_worker/notifications/push.py                      46     34    26%   14-23, 31-79, 83-86
zmon_worker_monitor/zmon_worker/notifications/slack.py                     38      0   100%
zmon_worker_monitor/zmon_worker/notifications/sms.py                       50      5    90%   79-96
zmon_worker_monitor/zmon_worker/notifications/twilio.py                    47     33    30%   32-86
zmon_worker_monitor/zmon_worker/tasks/__init__.py                           3      0   100%
zmon_worker_monitor/zmon_worker/tasks/main.py                             981    250    75%   243-244, 349, 504, 532, 613, 630, 689-691, 797-799, 820, 845, 963-966, 974-976, 1044, 1048-1051, 1073-1104, 1108-1181, 1184-1190, 1193-1202, 1217-1221, 1230-1232, 1246-1248, 1297-1298, 1302-1314, 1337, 1339, 1360-1382, 1437, 1441-1443, 1454-1455, 1472-1473, 1488-1491, 1526, 1537-1538, 1544-1553, 1632-1634, 1640-1643, 1711-1717, 1742-1747, 1764, 1798-1799, 1809, 1814-1815, 1821-1825, 1829-1849, 1855-1904, 1916-1917, 1950-1955, 1959-1961
-----------------------------------------------------------------------------------------------------
TOTAL                                                                    5986   1485    75%

============== 557 passed, 3 skipped, 1 warnings in 37.84 seconds ==============

[codecov[bot]]

Codecov Report

Merging #423 into master will increase coverage by 0.04%. The diff coverage is 82.35%.

@@            Coverage Diff            @@
##           master    #423      +/-   ##
=========================================
+ Coverage   75.15%   75.2%   +0.04%     
=========================================
  Files          80      81       +1     
  Lines        5954    5988      +34     
=========================================
+ Hits         4475    4503      +28     
- Misses       1479    1485       +6

Impacted Files	Coverage Δ
zmon_worker_monitor/builtins/plugins/bigquery.py	82.35% <82.35%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update ce70713...a14fa1e. Read the comment docs.

[vetinari]

LGTM

[vetinari]

:+1:

[mohabusama]

👍