HK: Elasticsearch: Look for a person that helps us move forward with this project

[jesusitd]

Elastic contacted directly (Johann).

Additionally, look for potential assistants on Upwork.

[jesusitd]

Hiring on Upwork

[jesusitd]

20191120 1500 ELK Meeting summary

Need a quick POC to determine using Elastic Stack as a solution to replace Power BI reporting Data source is MySQL Elasticsearch and Kibana are already installed and ready at client Need to install Logstash either on the existing Elasticsearch server or on a new server of similar type Need to configure the Logstash config file for: a. Input from MySQL existing database b. Demonstrate data transformation (replace a piece of data) using GROK c. Output data to Elasticsearch d. Add a dashboard to Kibana to display the data Need to organize, test and document the process for knowledge transfer to the client Once we have connectivity and a working environment this effort should be completed within 5-10hrs. I am available Friday 20191122 7:00AM EST to 3:00PM EST to work on this effort. If more time is needed to complete the POC I can log in over the weekend if I have connectivity. I look forward to helping. If we move forward, I will bill 1hr for the call, review and summary from today 20191120 3:00PM EST. Otherwise I hope our discussion helped. Thanks

[jesusitd]

Questions:

• Complex calculations
• Visual drillthrough
• Embed Report in CRM.
• Modeling / Relations
• Custom Visual.
• Large data table with export data option.
• Incremental Upsert of data. Real time.

[CesarITD]

Pre-Start Tasks:

• Inform Nick that we will be doing a POC (focused on reports) with an external resource.
• API End Point Testing to POC.
• Define a report and visuals that contain data from different related entities, calculated columns and calculated measures.
• To send Frank new POC scope and work schedule suggested (Thursday Nov-21).

POC Scope:

• Make the necessary setup adjustments to our current ELK Environment to be able to use all the necessary components (Logstash, Elasticsearch and Kibana).
• Grab the data from MySQL directly.
• Understand how ELK handles incremental data synchronization from My SQL.
• Understand how ELK tools can do complex calculations like the ones running in Power BI.
• Get guidance of how we can do Modeling and establish Relationships between data entities. Current model is based on a relational model.
• Being able to visualize live data in the Reports (Kibana) as it refreshes in [almost] real time. Have the dashboard refresh itself every X minutes.
• Being able to drillthrough the data in Kibana Reports.
• Understand how reports can be organized and grouped like PowerBI WorkSpaces and PowerBI Tabs.
• Verify that there is no limit to display high volumes of data detail (Visual like tables) and be able to export the data (data sweep).

[jesusitd]

Hired Frank Foti yesterday, accepted Upwork contract today.

[jesusitd]

Connection endpoints for Elastic Stack tests:

Elastic Endpoint: https://c95def58db0545acb17cd06aa8afcc53.us-central1.gcp.cloud.es.io:9243

Kibana Endpoint: https://f243523d45c945999d8d6b8de7a640d3.us-central1.gcp.cloud.es.io:9243

APM Endpoint: https://b3dd122a523041fea7ad49dad26138e7.apm.us-central1.gcp.cloud.es.io:443

[CesarITD]

2019NOV30 Pending for us:

• Load a large table. For example all Leads.
• Review UPSERT performance of a large table.

Pending with Frank:

• One Index to N Reports. Best practice?
• What is the practical use of * with the Index Template.
• Set up a data update schedule every 5 minutes.
• UTC Data issue.
• How customize Elastic mapping.
• Create a Wall Report with Canvas to get autorefresh every 5 minutes or realtime.
• Drilldown/Drilthrough with Elastic.
• What is the limit of rows in a Table Visual?
• What is the way to group Reports by Projects and Subcategories. (Like Workspace, Reports, Tabs).
• Can be able to embed a Kibana/Canvas report in our CRM

[jesusitd]

Deployed POC of Floor Displays to Canvas.

Email w Results sent to HK.

Observations from NM received (will be handled in separate issue).

Work with Frank Foti finished for this POC.

Further actions required by managements are being awaited and will be handled separately.