Test NLP Text Analytics App on Splunkbase - $500 #7

Closed evgenydmitriev closed 3 years ago

In GitLab by @evgenydmitriev on Apr 12, 2019, 11:17

Produce a comparative report on the capabilities of the NLP Text Analytics App (Blog post) vs Yachay (NTerminal's NLP module). The report should be supplemented with the results of the real-world NTerminal data and should include the following sections:

Data enrichment (named entities, sentiment, etc.)
Analytics
Visualisations
System load (used resources such as CPU and RAM)

Resources:

All NLP data from NTerminal
Developer Splunk search head with MLTK and NLP Text Analytics App

Inca Team:

Infrastructure Engineer (@anshlykov)
NLP Engineer (@myrmecophagous)
ML and Data Engineer (@alina.park)

In GitLab by @evgenydmitriev on Apr 12, 2019, 11:22

changed the description

In GitLab by @evgenydmitriev on Apr 12, 2019, 11:27

changed the description

In GitLab by @evgenydmitriev on Apr 17, 2019, 09:41

changed the description

In GitLab by @evgenydmitriev on Apr 22, 2019, 02:17

Email from Alexey:

Коллеги, приветствую.

Да, понятно, что нужно тестировать на боевых данных, но т.к. для меня приложение не известно и иных доступов нет, то понять принцип работы команд/алгоритмов прогонял на тестовых данных из коробки apps. Андрей, Егор, как можем начать тестировать промышленные данные? У вас есть dev среда (если пром.данные на неё заходят) или возможно достаточно выложить dump данных (обезличенных конечно если требуется, чтобы подать на вход NLP app's и протестировать на локалке)?

Какой вариант для вас предпочтительный?

Собрания и/или другие варианты общений по этой задаче будут проводиться? Какой предпочтительный формат общения?

@anshlykov

[ ] Set up an account for Alexey in dev search head
[ ] Install the NLP app in dev search head

In GitLab by @evgenydmitriev on Apr 24, 2019, 13:43

assigned to @alina.park

In GitLab by @evgenydmitriev on Apr 24, 2019, 13:43

changed due date to May 31, 2019

In GitLab by @evgenydmitriev on Apr 24, 2019, 13:45

New email from Alexey

Mon, Apr 22, 1:21 AM

Коллеги, приветствую.

Да, понятно, что нужно тестировать на боевых данных, но т.к. для меня приложение не известно и иных доступов нет, то понять принцип работы команд/алгоритмов прогонял на тестовых данных из коробки apps.

Андрей, Егор, как можем начать тестировать промышленные данные? У вас есть dev среда (если пром.данные на неё заходят) или возможно достаточно выложить dump данных (обезличенных конечно если требуется, чтобы подать на вход NLP app's и протестировать на локалке)?

Какой вариант для вас предпочтительный?

Собрания и/или другие варианты общений по этой задаче будут проводиться? Какой предпочтительный формат общения?

@alina.park take this over when you are done with your exams

In GitLab by @alina.park on May 7, 2019, 11:27

E-mailed Alexey about the issue with a meeting suggestion. I am planning on having a meeting with @myrmecophagous and Alexey tomorrow in the same order. Will update the issue afterwards.

Here's the e-mail; I cc-ed the participants:

Добрый день, Алексей,

Пишу вам по поводу вопроса, ранее обсуждаемого с Евгением по тестированию приложения. В дальнейшем я буду курировать проект и помогать вам во взаимодействии с командой, но первым этапом хотелось бы вовлечь вас в Gitlab активность по задаче - https://gitlab.com/IncaOutsourcing/bounty/issues/6

Далее, предлагаю провести онлайн-встречу, где мы обсудим детали. Насколько я понимаю, в будние дни вы свободны после 19 часов по времени Москвы. Я вышлю приглашение на 8 мая, 20 часов по Москве, но время гибкое.

Я, к сожалению, не смогла найти вас в Gitlab по этому email-у, поэтому прошу вас зарегистрироваться и оставить все вопросы, с которыми вы столкнулись, до нашей встречи, чтобы я ответила на них во вторник.

In GitLab by @alt23 on May 8, 2019, 24:49

@alina.park Добрый день, Алина. Спасибо, за ответ, жду приглашения. В профиле сейчас можно увидеть e-mail (иной, чем ранее переписывались). По вопросам, для начала хотелось познакомиться и понять принцип взаимодействаия ) А в частности по задаче, пояснить "NTerminal's NLP module".

In GitLab by @alina.park on May 8, 2019, 11:09

@alt23 приглашение на ваших почтах (отправила на оба адреса), на 20 часов по Москве. Хорошо, обсудим это на встрече!

In GitLab by @alt23 on May 8, 2019, 11:44

@alina.park увидел, спасибо. @anshlykov Андрей, сможете, предоставить доступ до тестовой среды?

In GitLab by @alina.park on May 9, 2019, 14:05

Had a quick introductory meeting with @alt23 , decided on weekly meetings. @myrmecophagous suggested using twitter as the testing basis for the module. The amount of data will be calculated by attempt, and varied to determine deviation @anshlykov Alexey needs access to Nterminal. Will you please help?

In GitLab by @alina.park on May 11, 2019, 22:05

@anshlykov pinging you regarding the Nterminal access; give Alexey access whenever you have time, please! Thank you!

In GitLab by @alina.park on May 20, 2019, 24:08

@alt23 I talked to Andrey and decided that I can help you out to get the materials you need for report comparison, but you need to give the details on that data for me to export.

In GitLab by @alina.park on May 22, 2019, 17:40

Received an email from Alexey asking for a meeting. @myrmecophagous you think you can make it tomorrow afternoon (Paris time)?

In GitLab by @myrmecophagous on May 22, 2019, 19:44

@alina.park Will only be available after 7 pm (Paris)

In GitLab by @evgenydmitriev on May 22, 2019, 20:01

@alina.park, stop swatting this issue. @myrmecophagous is here to help if needed, not to sit on every meeting.

In GitLab by @alina.park on May 23, 2019, 11:10

@myrmecophagous @evgenydmitriev got it, okay. I sent a meeting invite for this evening; will update after the meeting

In GitLab by @alina.park on May 23, 2019, 18:15

Had a meeting with Alexey. Do outs on my side:

[x] Collect raw Twitter data streaming during a day
[x] Collect NTerminal results for that data
[x] Get data on system load of NTerminal
[x] Send these to Alexey asap

In GitLab by @myrmecophagous on May 26, 2019, 21:59

@alina.park Raw Twitter data is available on GD (twitter.splunk_test.json, twitter.splunk_test.jsonl). There are 1298 tweets, collected on 22/05.

In GitLab by @alina.park on May 27, 2019, 11:12

@alt23 sent to your email in case you don't have access to the links. Let me know if you have any questions

In GitLab by @alt23 on May 27, 2019, 23:47

@alina.park data received, took to work.

In GitLab by @alt23 on May 31, 2019, 14:02

@evgenydmitriev current status: data are received The next stage - an analytics and NLP app testing. We need more time for this task.

In GitLab by @alina.park on Jun 6, 2019, 23:41

Received results raw version, set up a call with Alexey for tomorrow, will update the issue after

In GitLab by @alina.park on Jun 7, 2019, 19:22

changed due date to June 14, 2019

In GitLab by @alina.park on Jun 7, 2019, 19:29

Notes on the meeting with @alt23: report raw version

Most of the application's function worked for the twitter data corpus
Clustering function didn't work out for the corpus probably bc of the hardware issues on Alexey's side
@myrmecophagous Alexey needs the following settings for ML algorithms for a better comparison with Nterminal results:
1. N-gram range
2. word weights
3. stop-words list
4. named entities settings: singular or combined

@evgenydmitriev just a quick clarification: should I help Alexey with translation?

In GitLab by @myrmecophagous on Jun 10, 2019, 18:14

@alt23 @alina.park I suggest using different values for these options and see what works better.

n-gram range - up to 3 (or 4 if the hardware allows it), also if there is a possibility to cut the lower boundary you should try this.
word weights - the only thing I could suggest is to attribute a higher weight to our keywords. We have about 5k keywords. I could also filter out those that are not in the data sample. But then, what is the expected format, do you need lower-weight entries too?
stop word list - if I get it right, the app uses the default NLTK stop words for English which are good enough. Cannot think of any alternative or improvement.
named entities - try both options.

In GitLab by @alina.park on Jun 16, 2019, 21:13

@alt23 is away till Thursday, so I'll change the deadline. As we don't have the settings, additional testing will take more time, here's Alexey's email on that:

Я сейчас выехал, ноута нет под рукой, к четвергу вернусь. По данным: по логам NER вроде понятно, показывает по twitt event какие сформированы NER, можно аналог провести как сформировало NER данный apps. А вот по логам sentiment twitt event, тут надо покапаться. Как предварительно понял NT выдает оценки ibm,aws,google, у всех естественно разные. Мысли есть, ещё надо потестировать, как доберусь до компа. Да, кстати в этих логах (9980 event), события и за 2018, 2019 года (не только 22.05.2019). В четверг напишу о состоянии дел.

Gonna set a meeting on friday

In GitLab by @alina.park on Jun 22, 2019, 16:54

The analysis is finished, starting to re-work the report. Will set up a meeting with Alexey afterwarsd

In GitLab by @alina.park on Jun 22, 2019, 16:54

changed due date to June 18, 2019

In GitLab by @alina.park on Jun 24, 2019, 17:11

Set up a call with Alexey to discuss a couple of minor questions, will be able to present the results tomorrow during the Weekly Call

In GitLab by @alina.park on Jun 24, 2019, 17:11

changed due date to June 25, 2019

In GitLab by @alina.park on Jun 26, 2019, 01:31

Here are the links to the results of the tests:

@alt23 thank you for your work!

In GitLab by @alina.park on Jun 26, 2019, 12:15

@evgenydmitriev let @alt23 and me know if anything should be added, thanks!

In GitLab by @evgenydmitriev on Jun 26, 2019, 14:51

@alt23 @alina.park thank you!

@alt23 email me and let me know how you want to get paid

@alina.park, there a lot of information there. Maybe you should set up a report presentation meeting for everyone. Just make sure @myrmecophagous can make it.

In GitLab by @alina.park on Jun 26, 2019, 15:01

@evgenydmitriev sure, I'll talk to him before the presentation and send the invite to everyone on the team. And you also mentioned me instead of Alexey in the 'how to get paid' comment.

In GitLab by @alina.park on Jun 28, 2019, 24:34

Talked to @myrmecophagous, sent invites to everyone for this Monday

In GitLab by @evgenydmitriev on Jul 4, 2019, 11:37

closed

1712n / challenge

Test NLP Text Analytics App on Splunkbase - $500 #7