Make a Korea version of Twittermap

[GaeulG]

Overview

• It aims to make Korea version of Twittermap. Currently, Twittermap only provides the version of US. It will help the extension of Twittermap.
• This issue is written all of errors while our team has been making Korean Twitter Map.

Progress

• Current State of Korean Twittermap

Match format

1. Convert polygon coordinate UTM-K to WGS84

Before we develop, we need the Korea administrative district data file. You can download here : gisdeveloper

The polygons of Cloudberry were written by WGS84, that is longitude and latitude. But datas of Korea were written by UTM-K. This is formula that converts UTM-K to WGS84:

from pyproj import Proj
from pyproj import transform

WGS84 = {'proj':'latlong', 'datum':'WGS84', 'ellps':'WGS84'}

GRS80 = {'proj':'tmerc', 'lat_0':'38', 'lon_0':'127.5', 'k':0.9996, 'x_0':1000000,
         'y_0':2000000, 'ellps':'GRS80', 'units':'m'}

def grs80_to_wgs84(x,y):
    return transform(Proj(**GRS80), Proj(**WGS84), x, y)

2. Match properties of json file

Match the properties of Korean json file for working well on original Cloudberry.

• properties of state.json: {"geoID": "11", "stateID": 11, "name": "Seoul", "area": 1.0, "korName": "\uc11c\uc6b8\ud2b9\ubcc4\uc2dc"}
• properties of county.json: {"geoID": "11110", "name": "Jongno-gu", "area": 1.0, "countyID": 110, "stateName": "Seoul", "stateID": 11, "korStateName": "\uc11c\uc6b8\ud2b9\ubcc4\uc2dc", "korName": "\uc885\ub85c\uad6c"}
• properties of city.json: {"geoID": "11110101", "name": "Cheongun-dong", "landArea": 1.0, "waterArea": 1.0, "stateID": 11, "countyID": 110, "cityID": 101, "stateName": "Seoul", "countyName": "Jongno-gu", "korStateName": "\uc11c\uc6b8\ud2b9\ubcc4\uc2dc", "korCountyName": "\uc885\ub85c\uad6c", "korCityName": "\uccad\uc6b4\ub3d9"}

3. Simplify data file

• The size of json files were too big to implement twitter map. Therefore those were needed to simplify using mapshaper.
• When we considered the speed of updating map real time, the simplified percentage of map should be 25 each.
• You can control the percentage at the mapshaper.

4. Solve linestring closed error

• Our datas didn't match front and end of the multipolygon datas. After modified the datas, twittermap worked well.
• ex) [[[128.61022254606834, 35.875018095538714], [128.61071568778024, 35.87486899505937], …, [128.6097357715218, 35.874593196558195], [128.61022254606834, 35.875018095538714]]]

5. Korean crash problem

• When the json file was opened, Korean letters were crashed. Therefore, the files were needed to encode by 'euc-kr'.
• Before the file was imported, you have to encode it.
• You can do it here : mapshaper

6. Bring Korea Twitter Data

• After making the account of Twitter developer, we could get the data real time in the server.
• We changed the Twitter data file into .adm file using this command. gunzip -c Tweets_xxxx.gz | ./geotag.sh 1 2> /dev/null | gzip > somefilename.adm.gz;
• Put the lastest data in the directory : Cloudberry/cloudberry/examples/twittermap/script

What we have to do

• [ ] bring Korean twitter data real time
• [ ] fix the error related to the reset button

[chenlica]

@waans11 @baiqiushi status?

[waans11]

Due to a restriction of AsterixDB regarding the tokenization of languages other than English, we need to fix that issue first.

[baiqiushi]

Low priority.