search

headrun / SWIFT

2 stars 0 forks source link

Loading HealthKart data into Druid #130

Open jaffrinkirthiga96 opened 4 years ago

jaffrinkirthiga96 commented 4 years ago

To load the 6 xlsx files of HealthKart data into Druid

jaffrinkirthiga96 commented 4 years ago
  1. Have converted the 6 xlsx files into csv as Druid supports only (csv,tsv,json).
  2. Loaded the csv files into Druid's machine.
  3. On loading the csv into Druid, I observed the Date Format in the HealthKart File is (21 Sep 19 --> d MMM yy) which is not allowed as per Druid's Date Format. Only after processing the Date Format we can ingest the data into Druid. Hence I will process the csv into Druid date format and try loading it again.
jaffrinkirthiga96 commented 4 years ago

Have processed the Date Format of csv to dd/mm/yyyy. I will therefore try uploading into Druid and verify. image

jaffrinkirthiga96 commented 4 years ago

I have loaded the HealthKart files into Druid (Datasourse name: HealthKart). It has around 40L + records. http://druid.mie.one/unified-console.html#datasources I have created some Dashboards on top of the HealthKart Data. health-kart-orders-revenue-growth-ananlysis-2020-07-03T09-54-47 238Z http://ccd.mie.one/superset/dashboard/7/ http://ccd.mie.one/superset/dashboard/6/ health-kart-sales-analysis-2020-07-03T09-55-15 076Z

jaffrinkirthiga96 commented 4 years ago

Have created a login for HealthKart with below cred. username : hkart Password : hkart@123 I havent provided the dashboard level based permissions to the users. I checked with the superset guys in slack community and they told like such feature is not available yet and it is In-Progress. https://github.com/apache/incubator-superset/issues/9938

jaffrinkirthiga96 commented 4 years ago

Have created HealthKart Roles and given Permission to Dashboards based on the users.

jaffrinkirthiga96 commented 4 years ago

There is a js file of 5.7MB in superset which takes quite long time(~11secs) to completely load after login page. This load issue occurs only when that js file is not cached(after clearing the cache). I noticed the same slow login in ccd.mie.one and cc.mie.one This happens on the first time login to the Application. I observed that this js file has been added to the latest version of superset and older versions doesnt have this js file. http://demo.hyperbase.io/ (older version) dont have this issue while logging in. I am checking this with the superset community ppl.. If I get any solution will update the status