Permission to send service logs to external server

[canasdiaz]

Hi @hmitsch in order to browse the Bitergia Analytics logs in a more efficient way we would like to send them to one of our servers. The data sent is available in the VM in the file /data/docker/logs/mordred/mozilla/all.log and would be sent via logstash using a secure connection. There is no personal data in the logs, the contributor data is not logged.

The type of data logged is:

• timestamps
• module names
• repository names
• the ES endpoint (which can not be accessed from internet)
• stack traces / python tracebacks

Find an example below:

2018-04-19 07:46:26,035 - grimoire_elk.elk - INFO - Starting study: <bound method GitHubEnrich.enrich_onion of <grimoire_elk.enriched.github.GitHubEnrich object at 0x7f5097c31278>> (no_incremental False)
2018-04-19 07:46:26,035 - grimoire_elk.enriched.enrich - INFO - [Onion] Starting study
2018-04-19 07:46:26,036 - grimoire_elk.enriched.enrich - INFO - [Onion] Creating out ES index
2018-04-19 07:46:26,036 - urllib3.util.retry - DEBUG - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
2018-04-19 07:46:26,036 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): search-bitergia-metrics-es-****.es.amazonaws.com
2018-04-19 07:46:26,057 - urllib3.connectionpool - DEBUG - https://search-bitergia-metrics-es-****.es.amazonaws.com:443 "HEAD /github_issues_onion-enriched HTTP/1.1" 200 0
2018-04-19 07:46:26,057 - elasticsearch - INFO - HEAD https://search-bitergia-metrics-es-****.es.amazonaws.com:443/github_issues_onion-enriched [status:200 request:0.021s]
2018-04-19 07:46:26,057 - elasticsearch - DEBUG - > None
2018-04-19 07:46:26,057 - elasticsearch - DEBUG - < 
2018-04-19 07:46:26,057 - grimoire_elk.enriched.ceres_base - INFO - Deleting index github_issues_onion-enriched
2018-04-19 07:46:26,058 - urllib3.util.retry - DEBUG - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
2018-04-19 07:46:26,135 - urllib3.connectionpool - DEBUG - https://search-bitergia-metrics-es-****.es.amazonaws.com:443 "DELETE /github_issues_onion-enriched HTTP/1.1" 200 21
2018-04-19 07:46:26,136 - elasticsearch - INFO - DELETE https://search-bitergia-metrics-es-****.es.amazonaws.com:443/github_issues_onion-enriched [status:200 request:0.078s]
2018-04-19 07:46:26,136 - elasticsearch - DEBUG - > None
2018-04-19 07:46:26,136 - elasticsearch - DEBUG - < {"acknowledged":true}
2018-04-19 07:46:26,136 - urllib3.util.retry - DEBUG - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
2018-04-19 07:46:26,228 - mordred.task_identities - DEBUG - [load identities] Enrich tasks active: 3
2018-04-19 07:46:26,299 - perceval.backends.core.git - DEBUG - 
2018-04-19 07:46:26,300 - perceval.backends.core.git - DEBUG - Git https://github.com/mozillahispano/sleep-webapp.git repository has 4 objects
2018-04-19 07:46:26,300 - perceval.backends.core.git - DEBUG - Running command git show-ref --heads --tags (cwd: /home/bitergia/.perceval/repositories/https://github.com/mozillahispano/sleep-webapp.git-git, env: {'HOME': '/home/bitergia', 'PAGER': '', 'LANG': 'C'})
2018-04-19 07:46:26,708 - urllib3.connectionpool - DEBUG - https://search-bitergia-metrics-es-****.es.amazonaws.com:443 "PUT /github_issues_onion-enriched HTTP/1.1" 200 48

Are we allowed to send this data to our node?

[canasdiaz]

Hi @hmitsch Let me know if we can do something to speed up this. We need it to improve the service :nerd_face:

Best!

[hmitsch]

On it ... Please give me 24-48 hours. Sorry for dropping that ball.

[hmitsch]

@sanacl we got clearance. Please make sure the log data is handled properly (retention, etc.). If you have any policies on that, we'd be happy to learn about them.

[canasdiaz]

@hmitsch our currently policy in data retention is dropping every log older than 2 weeks, we are just using these logs to identify issues in the latest release so we don't need older logs. Is this the info you were expecting?

[hmitsch]

Yes, thank you. Please go ahead with the log shipping.

[canasdiaz]

As soon as I resize one of our ES servers, this will be done. ETA: May 30th

[canasdiaz]

This is finally done. See an example of the data we are using in the screenshot below