Enhance the robustness for LIST METHOD

[OmegaVVeapon]

The LIST method is a one-time run that occurs in order to load up the datasources as part of the initContainer.

It was added because:

1. The Grafana Helm chart needs it
2. Grafana itself does NOT support dynamic loading of datasources (like they do with dashboards)

However, if the connection with the k8s control plane fails, it's possible for the initContainer to fail to come up.

Need to add additional logging and retry handling to make this area more robust.

Here's one such situation of that issue:

Setting USER environment variable to 472
2021-05-19 16:53:06,618 Using the LIST METHOD
2021-05-19 16:53:06,618 The default FOLDER to write files to is /etc/grafana/provisioning/datasources
2021-05-19 16:53:06,619 FOLDER_ANNOTATION not set. Defaulting to look for 'k8s-sidecar-target-directory'
2021-05-19 16:53:06,619 Looking for resources with LABEL 'grafana_datasource'
2021-05-19 16:53:06,619 Monitoring configmap and secret resources for changes
2021-05-19 16:53:06,619 Looking for resources in the entire cluster
2021-05-19 16:54:19,920 [CREATE:ConfigMap] Writing content to file /etc/grafana/provisioning/datasources/datasource.yaml
Traceback (most recent call last):
  File "/.venv/lib/python3.8/site-packages/urllib3/response.py", line 697, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/.venv/lib/python3.8/site-packages/urllib3/response.py", line 438, in _error_catcher
    yield
  File "/.venv/lib/python3.8/site-packages/urllib3/response.py", line 764, in read_chunked
    self._update_chunk_length()
  File "/.venv/lib/python3.8/site-packages/urllib3/response.py", line 701, in _update_chunk_length
    raise InvalidChunkLength(self, line)
urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/.venv/lib/python3.8/site-packages/requests/models.py", line 753, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/.venv/lib/python3.8/site-packages/urllib3/response.py", line 572, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/.venv/lib/python3.8/site-packages/urllib3/response.py", line 793, in read_chunked
    self._original_response.close()
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/.venv/lib/python3.8/site-packages/urllib3/response.py", line 455, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "sidecar.py", line 109, in <module>
    main()
  File "sidecar.py", line 104, in main
    one_run()
  File "/list_mode.py", line 54, in one_run
    for secret in secrets:
  File "/.venv/lib/python3.8/site-packages/pykube/query.py", line 195, in __iter__
    return iter(self.query_cache["objects"])
  File "/.venv/lib/python3.8/site-packages/pykube/query.py", line 185, in query_cache
    cache["response"] = self.execute().json()
  File "/.venv/lib/python3.8/site-packages/pykube/query.py", line 159, in execute
    r = self.api.get(**kwargs)
  File "/.venv/lib/python3.8/site-packages/pykube/http.py", line 431, in get
    return self.session.get(*args, **self.get_kwargs(**kwargs))
  File "/.venv/lib/python3.8/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/.venv/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/.venv/lib/python3.8/site-packages/requests/sessions.py", line 697, in send
    r.content
  File "/.venv/lib/python3.8/site-packages/requests/models.py", line 831, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/.venv/lib/python3.8/site-packages/requests/models.py", line 756, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[OmegaVVeapon]

Currently testing a fix. Relax stalebot.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[OmegaVVeapon]

Had to make a few more changes but got a promising fix in https://github.com/OmegaVVeapon/kopf-k8s-sidecar/pull/22...

[OmegaVVeapon]

Fixed by #22.

Closing.