search

quoeamaster / bigdata_blogs

projects related to big-data blogs (e.g. elasticsearch)
Apache License 2.0
6 stars 7 forks source link

Using API as source #1

Closed Japhys closed 4 years ago

Japhys commented 4 years ago

I really enjoyed your medium story! I wanted to comment but I am not a Medium member so forgive me for opening an issue, maybe misusing the system here. I am new to Elastic but I was thinking about diving into it, head first. Reading this I wonder if the learning curve isn't a bit too steep for me right now (considering a chronic lack of time I seem to suffer from lately). But what I was wondering: did you try this as a source -> https://github.com/ExpDev07/coronavirus-tracker-api I guess it wasn't available at the time you wrote this piece. I would love to see an auto updated corona Kibana dashboard using this api. A bit like the John Hopkins University dashboard. That would be awesome.

quoeamaster commented 4 years ago

Hi @Japhys,

Thx for the suggestions on using the new api provided; I have taken a look at the json responses from the api.

Typically, will need to re-structure the "history" part of the json response; need to break each historical entry into individual elasticsearch documents; if not the visualizations would not work.

The visualizations under Elasticsearch works best for time-based data - which is why the above re-structuring is a MUST. I will prepare a new blog and code (written in golang) to solve this challenge. Also a new set of visualizations would be created based on this rich api (again some advance techniques are required to accomplish some visualizations)

PS. will add back the link of the new blog and related github links once the blog is ready :)

Que

quoeamaster commented 4 years ago

Hi @Japhys

the blog is ready at here https://medium.com/@quoeamaster/use-case-integrating-a-covid-19-tracker-api-with-the-elastic-stack-plus-story-telling-with-kibana-8805cb67678

One of question was how to make sure the data could be updated regularly, on this, will need a cron job for sure. I have written a golang program to extract and re-structure the data. Hence if we add a cron job to trigger the golang program, theoretically will solve your question

:)

Que

Japhys commented 4 years ago

Oh wow, thank you so very much. This is great. I will certainly give it a try myself now (...I am in lock down anyway and have more time). I appreciate your effort!

quoeamaster commented 4 years ago

hi @Japhys

no worries and do leave me a message if you need any help

:)

Que