search

mincloud1501 / Python

Jupyter Notebook을 활용한 Time-series data 분석 및 crawling 기술, D3를 이용한 시각화 기술 구현 및 연구
https://gitter.im/Python_Project/community
MIT License
6 stars 1 forks source link
d3js deck-gl jupyter-notebook pycharm-edu pydeck python webcrawling

readme

The Python Study Project

Gitter chat   

All Codes implemented in Python (with Jupyter Notebook)

These implementations are for learning purposes. They may be less efficient than the implementations in the Python standard library. Use Jupyter Notebook & PyCharm Community Edition.

■ Python Data Analytic Library

Time Series Analytics with Pandas (On Windows)

■ Prerequisites for running with Anaconda3 install

conda create --name mincloud python=3
conda install pandas jupyter seaborn scikit-learn statsmodels
conda info --envs
Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: C:\Users\mincloud\Anaconda3

  added / updated specs:
    - jupyter
    - pandas
    - scikit-learn
    - seaborn
    - statsmodels

The following packages will be downloaded:

            package                |         build
    ---------------------------|--------------------------------
    ca-certificates-2018.03.07 |                0         155 KB
    certifi-2018.4.16          |           py36_0         143 KB
    conda-4.6.14               |           py36_0         2.1 MB
    openssl-1.0.2o             |       h8ea7d77_0         5.4 MB
    ------------------------------------------------------------
                                           Total:         7.8 MB

The following packages will be SUPERSEDED by a higher-priority channel:

  ca-certificates                                anaconda --> pkgs/main
  certifi                                        anaconda --> pkgs/main
  conda                                          anaconda --> pkgs/main
  openssl                                        anaconda --> pkgs/main

■ Setup & Run

> git clone https://github.com/mincloud1501/Python.git

USEFUL SHORTCUTS (Jupyter Notebook)

run_and_pass = 'Shift + Enter'
run_and_add_cell = 'Alt + Enter'
run_and_stay = 'Ctrl + Enter'

recommendations = 'Tab'
docstrings = 'Shift + Tab'

add_hashtag = 'Ctrl + /'
add_cell = 'b'
delete_cell = 'dd'

■ Prerequisites for collecting Facebook data Sources

■ Prerequisites for collecting Naver data Sources

■ D3.js Reference for Data Visualizations

■ deck.gl

Installation

$npm install deck.gl
$conda create -n geo_env
$conda activate geo_env
$conda config --env --add channels conda-forge
$conda config --env --set channel_priority strict
$conda install python=3 geopandas

■ Pydeck

Installation

$pip install pydeck
$jupyter nbextension install --sys-prefix --symlink --overwrite --py pydeck
$jupyter nbextension enable --sys-prefix --py pydeck
pdk.Layer
pdk.Layer
  type = "미리 정의된 레이어 타입",
  id = "이 레이어의 아이디 (optional)",
  data = "pandas.DataFrame 또는 geojson url",
)
pdk.ViewState
pdk.ViewState(
  longitude = "중심 경도 (default 0)",
  latitude = "중심 위도 (default 0)",
  zoom = "줌 레벨 (default 0)",
  pitch = (default 0),
  bearing = (default 0),
  **kwargs,
)
pdk.Deck
pdk.Deck(
  layers=[],
  views=[{"controller": true, "type": "MapView"}],
  map_style='mapbox://styles/mapbox/dark-v9',
  mapbox_key=None,
  initial_view_state={"bearing": 0, "latitude": 0.0, "longitude": 0.0, "maxZoom": 20, "minZoom": 0, "pitch": 0, "zoom": 1},
  width='100%',
  height=500,
  tooltip=True,
)

Mapbox API token

mapbox

Getting started (Demo.ipynb)

import pydeck as pdk

# 2014 locations of car accidents in the UK
UK_ACCIDENTS_DATA = ('https://raw.githubusercontent.com/uber-common/'
                     'deck.gl-data/master/examples/3d-heatmap/heatmap-data.csv')

# Define a layer to display on a map
layer = pdk.Layer(
    'HexagonLayer',
    UK_ACCIDENTS_DATA,
    get_position='[lng, lat]',
    auto_highlight=True,
    elevation_scale=50,
    pickable=True,
    elevation_range=[0, 3000],
    extruded=True,                 
    coverage=1)

# Set the viewport location
view_state = pdk.ViewState(
    longitude=-1.415,
    latitude=52.2323,
    zoom=6,
    min_zoom=5,
    max_zoom=15,
    pitch=40.5,
    bearing=-27.36)

# Render
r = pdk.Deck(layers=[layer], initial_view_state=view_state)
r.to_html('demo.html')

# Jupyter 환경 밖에서 실행 시
r.to_html('demo.html', notebook_display=False)
def multipolygon_to_coordinates(x):
  lon, lat = x[0].exterior.xy
  return [[x, y] for x, y in zip(lon, lat)]

df['coordinates'] = df['geometry'].apply(multipolygon_to_coordinates)
del df['geometry']

PolygonLayer

// Make layer
layer = pdk.Layer(
    'PolygonLayer', // 사용할 Layer 타입
    df,             // 시각화에 쓰일 데이터프레임
    get_polygon='coordinates', // geometry 정보를 담고있는 컬럼 이름
    get_fill_color='[0, 255*정규화인구, 0]', // 각 데이터 별 rgb 또는 rgba 값 (0~255)
    pickable=True, // 지도와 interactive 한 동작 on
    auto_highlight=True // 마우스 오버(hover) 시 박스 출력
)

// Set the viewport location
center = [126.986, 37.565]
view_state = pdk.ViewState(
    longitude=center[0],
    latitude=center[1],
    zoom=10
)

// Render
r = pdk.Deck(layers=[layer], initial_view_state=view_state)
r.show()

HeatmapLayer

layer = pdk.Layer(
    'HeatmapLayer',
    df,
    get_position='[lng, lat]'
)

center = [126.986, 37.565]
view_state = pdk.ViewState(
    longitude=center[0],
    latitude=center[1],
    zoom=10
)

r = pdk.Deck(layers=[layer], initial_view_state=view_state)
r.show()

Python용 GCP Stackdriver Trace 설정 Sources

export PROJECT_ID="zipkin-proxy"
> pip install google-api-python-client

[main.py]

import argparse
import random
import time

from flask import Flask, redirect, url_for

# [START trace_setup_python_configure]
from opencensus.ext.stackdriver import trace_exporter as stackdriver_exporter
import opencensus.trace.tracer

def initialize_tracer(project_id):
    exporter = stackdriver_exporter.StackdriverExporter(
        project_id=project_id
    )
    tracer = opencensus.trace.tracer.Tracer(
        exporter=exporter,
        sampler=opencensus.trace.tracer.samplers.AlwaysOnSampler()
    )

    return tracer
# [END trace_setup_python_configure]

app = Flask(__name__)

@app.route('/', methods=['GET'])
def root():
    return redirect(url_for('index'))

# [START trace_setup_python_quickstart]
@app.route('/index.html', methods=['GET'])
def index():
    tracer = app.config['TRACER']
    tracer.start_span(name='index')

    # Add up to 1 sec delay, weighted toward zero
    time.sleep(random.random() ** 2)
    result = "Tracing requests"

    tracer.end_span()
    return result
# [END trace_setup_python_quickstart]

if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        description=__doc__,
        formatter_class=argparse.RawDescriptionHelpFormatter
    )
    parser.add_argument(
        '--project_id', help='Project ID you want to access.', required=True)
    args = parser.parse_args()

    tracer = initialize_tracer(args.project_id)
    app.config['TRACER'] = tracer

    app.run()

[main_test.py]

import os
import main

def test_index():
    project_id = os.environ['GCLOUD_PROJECT']
    main.app.testing = True
    main.app.config['TRACER'] = main.initialize_tracer(project_id)
    client = main.app.test_client()

    resp = client.get('/index.html')
    assert resp.status_code == 200
    assert 'Tracing requests' in resp.data.decode('utf-8')

def test_redirect():
    project_id = os.environ['GCLOUD_PROJECT']
    main.app.testing = True
    main.app.config['TRACER'] = main.initialize_tracer(project_id)
    client = main.app.test_client()

    resp = client.get('/')
    assert resp.status_code == 302
    assert '/index.html' in resp.headers.get('location', '')

[Trace Result] StackDriver

SK Open API 이용 (Tmap API_무료)

openapi

<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>simpleMap</title>
<script src="https://apis.openapi.sk.com/tmap/jsv2?version=1&appkey=발급받은 Appkey"></script>
<script type="text/javascript">
  // 페이지가 로딩이 된 후 호출하는 함수입니다.
  function initTmap(){
    // map 생성
    // Tmapv2.Map을 이용하여, 지도가 들어갈 div, 넓이, 높이를 설정합니다.
    var map = new Tmapv2.Map("map_div", { // 지도가 생성될 div
      width : "100%", // 지도의 넓이
      height : "400px" // 지도의 높이
    });
  } 
</script>
</head>

<body onload="initTmap()"><!-- 맵 생성 실행 -->
  <div id="map_div"></div>
</body>

</html>

tmap_sample

<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>simpleMap</title>
<script
  src="https://apis.openapi.sk.com/tmap/jsv2?version=1&appkey=발급받은 Appkey"></script>
<script type="text/javascript">                 
  var map;
  // 페이지가 로딩이 된 후 호출하는 함수입니다.
  function initTmap(){

    // map 생성
    // Tmap.map을 이용하여, 지도가 들어갈 div, 넓이, 높이를 설정합니다.
    map = new Tmapv2.Map("map_div", {
      center : new Tmapv2.LatLng(37.566481622437934, 126.98502302169841), // 지도 초기 좌표
      width : "100%", // map의 width 설정
      height : "400px" // map의 height 설정  
    });

    map.setZoom(16);

    var lat = 37.566681;
    var lng = 126.978453;
    var markers1 = [];

    for (var i = 0; i < 5; i++) {
      for (var j = 0; j < 5; j++) {
        var marker = new Tmapv2.Marker({
          position: new Tmapv2.LatLng(lat + (j * 0.001), lng + (i * 0.001)),
          label: 'cluster',
          map: map
            });
            markers1.push(marker);
        }
    }
    // 마커클러스터 등록
    markerCluster = new Tmapv2.extension.MarkerCluster({
      markers: markers1, // 마커들이 담긴 배열
      map: map // 지도 객체
    });

  };
</script>
</head>
<body onload="initTmap()"><!-- 맵 생성 실행 -->
  <div id="map_div"></div>
</body>
</html>

marker_cluster

geocoding

Category

See the Category.