search

electinth / parliament-listening

https://elect.in.th/parliament-listening/
MIT License
19 stars 3 forks source link

Update data #15

Open p16i opened 5 years ago

p16i commented 5 years ago

Related to #2

Current data (./static/data/) that we use in the prototype is prepared with Python. Code for this preparation can be found and hosted on Colab.

It needs to be modified to support our latest design. One of such modifications is to join with Party dataset, such that we can compute statistics of parties.

unnawut commented 5 years ago

I have very little experience with data management so apologies if this idea is nonsensical...

What if we have the raw data in csv format in this repo (exported from google spreadsheet that we use for time marking), and use nodejs to ETL the data... so that it uses the same stack as the web app? Otherwise there's that bottleneck where a contributor needs extra learning curve to participate in the project.

p16i commented 5 years ago

Hi,

I would love to have everything in the same stack. However, i'm not sure how straightforward it is. I used Colab and Python for the preparation because of my familiarity with Pandas.

We could start with Zebras, a Pandas alternative for NodeJS, but I'm not sure whether it'll beneficial for the beta version.