search

operate-first / support

This repo should serve as a central source for users to raise issues/questions/requests for Operate First.
GNU General Public License v3.0
15 stars 25 forks source link

Export log data sample for anomaly detection system #387

Closed dhonza closed 2 years ago

dhonza commented 3 years ago

Requested actions

I kindly ask you to provide an export of a sample of your cluster log files (we have discussed that with @durandom via call). I lead a small Log Anomaly Detection team at AIC, CTU in Prague (see https://www.aic.fel.cvut.cz/). We experiment with NLP-based methods to analyze raw log files. Lately, two of my students defended their theses:

Martin Koryťák: "Anomaly Detection Methods for Log Files": https://dspace.cvut.cz/handle/10467/95290

Prokop Černý: "Contextual Embeddings for Anomaly Detection in Log Files": https://dspace.cvut.cz/handle/10467/95324?locale-attribute=en

It would be great to have a sample of your log files (let's say few gigabytes), so we can try how the methods we developed work on other datasets. The best possible form of the data will involve cluster node ids and/or process ids.

Best

Jan Drchal

durandom commented 3 years ago

@4n4nd please link to documentation on how to access our cluster logs

sesheta commented 2 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

sesheta commented 2 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

4n4nd commented 2 years ago

/remove-lifecycle rotten

4n4nd commented 2 years ago

@dhonza we have documentation available here that you could use to get access to our Loki instance, there you can access all the available cluster logs.

sesheta commented 2 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

/close

sesheta commented 2 years ago

@sesheta: Closing this issue.

In response to [this](https://github.com/operate-first/support/issues/387#issuecomment-1045069306): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.