search

slovensko-digital / ekosystem

Verejný issue tracker
https://ekosystem.slovensko.digital
31 stars 0 forks source link

Performance dashboardy #13

Open jsuchal opened 7 years ago

jsuchal commented 7 years ago

Bolo by fajn rozbehat (je to opensource cele?) nieco ako https://www.gov.uk/performance a napojit na to data z data.gov.sk / https://metais.finance.gov.sk/

Napady:

Dalsie napady co sledovat?

jangaraj commented 7 years ago

Je mozne poziadat o povolenie cross requestov (CORS headers) z https://metais.finance.gov.sk/?

Ak by to bolo, tak by sa to dalo pouzit na vsetky dostupne data z metais - dal by sa na tom urobit kompletne novy frontend (static html, jquery, bootstrap + nejaka lib na grafy):

$.getJSON( "https://metais.finance.gov.sk/monitoring/param-value/list?entityRef=baafb479-407a-43bb-891d-b21571eb1491&parameterTypeId=18&intervalStart=2016-07-31&intervalEnd=2016-09-07&sortAttr=intervalStart&sortAsc=true&perPageSize=100000", function( json ) {
  console.log( "JSON Data: " + json );
});

Nebude to vsak duplicitna funkcionalita, ktore je uz teraz dostupna ako monitoring reporty https://metais.finance.gov.sk/monitoras?

jsuchal commented 7 years ago

Duplikovat funkcionalitu velmi nechceme. Zdroje som len nahadzal kvôli tomu co by sme chceli. Na cors by som sa nespoliehal, ale ani to není kritické.

jangaraj commented 7 years ago

A co verejna Grafana (ako univerzalna a najpopularnejsia platforma na vizualizaciu time series dat) + custom Grafana datasources (pripadne najnovsie aj apps)?

public snaphost: https://snapshot.raintank.io/dashboard/snapshot/cV9J7B3jsaxxbyEbtEkNSfyHop0h6gxR metais POC https://github.com/jangaraj/simple-json-datasource: POC ^ BTW kvalita dat z metais velmi kolise (frekvencia metrik, prazdne obdobia, ziadne hodnoty, ...)

jsuchal commented 7 years ago

@jangaraj toto je slusne, ale grafana aj metais trpia tym istym, je to take trosku necitatelne pre bezneho smrtelnika. S @Xubor sme prave riesili, ze mame v principe 2 veci: ad-hoc analyzy - tam je fajn pouzivat nejake Jypyter/ipython notebooks (chystame, ukazka tu https://github.com/sevo/datahub-sd-jupyter/blob/master/datahub-sample.ipynb) a potom veci co sledujeme pravidelne, kde by sa uzivili nejaky dashboard - ale naozaj pre BFU. Grafana je proste monitorovatko pre technokratov/devops.

a + teda kvalita dat je problem stale.

Zatial sa nam najviac pozdavaju tie Jupyter notebooks, lebo na githube to priamo vedia renderovat, mas cely fork/pullrequest flow, etc etc.

jangaraj commented 7 years ago

Ja suhlasim s Jupyter-om. Ma podporu viacero jazykov, takze aj technokrati si pridu na svoje (napr. R s hotovym ckanr modulom, ktory bude tahat data cez API z data.gov.sk).

jsuchal commented 7 years ago

Mas s tym nejake skusenosti?

jangaraj commented 7 years ago

S Jupyterom/CKAN nie, s R ano.

xar commented 7 years ago

ahoj chcem sa spytat ci sa tym bfu dashboardom mysli nieco taketo => https://dashboardmetais-tsbmyqnjot.now.sh/#/ . Ak ano mozem v tom pokracovat a spravit to rozumne ( dorobit filtre, dalsie sluzby, labele dat podla mesiacov... )

xar commented 7 years ago

... inac http://www.cyclotron.io/

jsuchal commented 7 years ago

@jangaraj pocuj tie data nejako prestali tiect? V tom dashboarde vidim naposledy nieco zo septembra.

@xar na tvojej linke nevidim tiez nic. cyclotron vyzera celkom fajn.

jsuchal commented 7 years ago

@xar aaa, tak uz sa to nacitalo. No grafy su to fajn, ale asi by sme na to mali nejakeho grafika dostat. Oproti gov.uk/performance je to take dost programatorske ui :)

jangaraj commented 7 years ago

https://snapshot.raintank.io/dashboard/snapshot/cV9J7B3jsaxxbyEbtEkNSfyHop0h6gxR je iba snapshot, to zial nie su live data.

A co forknut https://github.com/18F/analytics.usa.gov ?

jsuchal commented 7 years ago

@jangaraj no prave to su len navstevnost statistiky, mne sa paci ovela viac https://www.gov.uk/performance/hmrc-pay-as-you-earn-paye aj by to mohli mat niekde na githube

jangaraj commented 7 years ago

OK, tak forknime UK https://github.com/alphagov:

=> framework, ktoreho prevadzka nestoji nic + takmer ziadny technologicky lock + plne otvorene pre hackerov a technokratov na dalsi rozvoj

Podobny koncept mam viac ako rok nasadeny na https://github.com/monitoringartist/zabbix-searcher - frontend je staticka responzivna stranka s jquery, pre ktoru data (json subor) pripravuje a validuje python skript raz denne na vybranom CI nastroji

jsuchal commented 7 years ago

@jangaraj hladam na alphagov ci nemaju aj template na tie metrics stranky ale zatial marne.

jangaraj commented 7 years ago

Je to pravdepodobne https://github.com/alphagov/spotlight + https://github.com/alphagov/spotlight-config

jsuchal commented 7 years ago

@jangaraj vyzera to tak, len kukam ze pod tym je este nejaky backend. To by sme mozno vedeli nasimulovat.

jsuchal commented 7 years ago

@jangaraj tak som ozivil na metabase na https://bi.ekosystem.slovensko.digital chces pristup?

daju sa tam taketo pekne veci robit. image

Keby sme do datahubu vedeli tlacit data z metais tak vieme celkom lahko "grafovat". A vyzera to aspon trochu dobre. Cakam vsak na metabase 23 release - ten ma uz aj public linkovanie dashboardov