Fixed typo and added new air quality data source in data_sources.md

[arthurlli]

Hi, I added an open data provided by Japanese government, although it is in Japanese. Some keywords are translated below:

大気環境 = atmospheric environment 時間値データ（2009～2018年度）= time series data (from 2009-2018) 都道府県名 = prefectures 全国 = entire country (i.e., all prefectures) 年度 = year

[xoolive]

どうも・多謝 Just wondering about the format of the data we download. It's called txt but it's binary in the end? Any experience using/parsing it?

[arthurlli]

Yes, the data file is in .txt format. The reason why it looks like binary is because it is about SO2 concentration in ppb, which usually be 1 or 0 in Japan. If we open other data, we would see the difference.

However, to import such data, we need to follow the instructions here. Then, find and set the "File Origin" to "932: Japanese (Shift-JIS)" to view column names (as screenshot below): The corresponding columns are translated:	Year	Measuring station code	District code	Type of pollutant	Unit	Month	Day	1st ~ 24th hour

Unfortunately there is no official english version.

[xoolive]

I couldn't reproduce what you recommend but got it the old fashioned way with iconv. I'll keep my opinions about encoding to myself... 🙄

$ iconv -f SHIFT_JIS -t UTF-8 TD20180126.txt | head -n5
測定年度,項目種類コード,項目コード_数字,項目コード_英数字,測定方法コード,都道府県コード,都道府県名,都道府県名_ローマ字,市区町村コード,市区町村名,市区町村名_ローマ字,測定局コード,測定局名,測定局名_ローマ字,測定局区分コード,測定局種別コード,用途地域コード,用途地域名,令別表第３の区分,有効測定日数(日),測定時間(時間),年平均値(ppm),１時間値が0.1ppmを超えた時間数(時間),１時間値が0.1ppmを超えた時間数の測定時間数に対する割合(%),日平均値が0.04ppmを超えた日数(日),日平均値が0.04ppmを超えた日数の有効測定日数に対する割合(%),１時間値の最高値(ppm),日平均値の２%除外値(ppm),日平均値が0.04ppmを超えた日が２日以上連続したことの有無(有:X・無:O),環境基準の長期的評価による日平均値が0.04ppmを超えた日数(日),測定方法,年間集計項目13,年間集計項目14,年間集計項目15,年間集計項目16,有効測定日数(日)_４月,有効測定日数(日)_５月,有効測定日数(日)_６月,有効測定日数(日)_７月,有効測定日数(日)_８月,有効測定日数(日)_９月,有効測定日数(日)_10月,有効測定日数(日)_11月,有効測定日数(日)_12月,有効測定日数(日)_１月,有効測定日数(日)_２月,有効測定日数(日)_３月,測定時間(時間)_４月,測定時間(時間)_５月,測定時間(時間)_６月,測定時間(時間)_７月,測定時間(時間)_８月,測定時間(時間)_９月,測定時間(時間)_10月,測定時間(時間)_11月,測定時間(時間)_12月,測定時間(時間)_１月,測定時間(時間)_２月,測定時間(時間)_３月,月平均値(ppm)_４月,月平均値(ppm)_５月,月平均値(ppm)_６月,月平均値(ppm)_７月,月平均値(ppm)_８月,月平均値(ppm)_９月,月平均値(ppm)_10月,月平均値(ppm)_11月,月平均値(ppm)_12月,月平均値(ppm)_１月,月平均値(ppm)_２月,月平均値(ppm)_３月,１時間値が0.1ppmを超えた時間数(時間)_４月,１時間値が0.1ppmを超えた時間数(時間)_５月,１時間値が0.1ppmを超えた時間数(時間)_６月,１時間値が0.1ppmを超えた時間数(時間)_７月,１時間値が0.1ppmを超えた時間数(時間)_８月,１時間値が0.1ppmを超えた時間数(時間)_９月,１時間値が0.1ppmを超えた時間数(時間)_10月,１時間値が0.1ppmを超えた時間数(時間)_11月,１時間値が0.1ppmを超えた時間数(時間)_12月,１時間値が0.1ppmを超えた時間数(時間)_１月,１時間値が0.1ppmを超えた時間数(時間)_２月,１時間値が0.1ppmを超えた時間数(時間)_３月,日平均値が0.04ppmを超えた日数(日)_４月,日平均値が0.04ppmを超えた日数(日)_５月,日平均値が0.04ppmを超えた日数(日)_６月,日平均値が0.04ppmを超えた日数(日)_７月,日平均値が0.04ppmを超えた日数(日)_８月,日平均値が0.04ppmを超えた日数(日)_９月,日平均値が0.04ppmを超えた日数(日)_10月,日平均値が0.04ppmを超えた日数(日)_11月,日平均値が0.04ppmを超えた日数(日)_12月,日平均値が0.04ppmを超えた日数(日)_１月,日平均値が0.04ppmを超えた日数(日)_２月,日平均値が0.04ppmを超えた日数(日)_３月,１時間値の最高値(ppm)_４月,１時間値の最高値(ppm)_５月,１時間値の最高値(ppm)_６月,１時間値の最高値(ppm)_７月,１時間値の最高値(ppm)_８月,１時間値の最高値(ppm)_９月,１時間値の最高値(ppm)_10月,１時間値の最高値(ppm)_11月,１時間値の最高値(ppm)_12月,１時間値の最高値(ppm)_１月,１時間値の最高値(ppm)_２月,１時間値の最高値(ppm)_３月,日平均値の最高値(ppm)_４月,日平均値の最高値(ppm)_５月,日平均値の最高値(ppm)_６月,日平均値の最高値(ppm)_７月,日平均値の最高値(ppm)_８月,日平均値の最高値(ppm)_９月,日平均値の最高値(ppm)_10月,日平均値の最高値(ppm)_11月,日平均値の最高値(ppm)_12月,日平均値の最高値(ppm)_１月,日平均値の最高値(ppm)_２月,日平均値の最高値(ppm)_３月,月間集計項目8_４月,月間集計項目8_５月,月間集計項目8_６月,月間集計項目8_７月,月間集計項目8_８月,月間集計項目8_９月,月間集計項目8_10月,月間集計項目8_11月,月間集計項目8_12月,月間集計項目8_１月,月間集計項目8_２月,月間集計項目8_３月,月間集計項目9_４月,月間集計項目9_５月,月間集計項目9_６月,月間集計項目9_７月,月間集計項目9_８月,月間集計項目9_９月,月間集計項目9_10月,月間集計項目9_11月,月間集計項目9_12月,月間集計項目9_１月,月間集計項目9_２月,月間集計項目9_３月
2018,1,01,SO2,2,26,京都府,Kyoto-fu,26104,京都市中京区,Kyouto-shi Nakagyou-ku,26104060,壬生,Mibu,1,0,3,準工,560,364,8703,0.004,0,0,0,0,0.013,0.008,O,0,2,,,,,30,31,30,31,31,30,31,30,31,31,28,30,716,740,715,738,740,718,741,715,737,740,670,733,0.004,0.005,0.005,0.006,0.006,0.005,0.003,0.003,0.003,0.003,0.003,0.003,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.013,0.011,0.01,0.012,0.013,0.009,0.006,0.006,0.008,0.007,0.007,0.008,0.007,0.008,0.007,0.009,0.009,0.007,0.005,0.004,0.004,0.004,0.005,0.005,,,,,,,,,,,,,,,,,,,,,,,,
2018,1,01,SO2,3,26,京都府,Kyoto-fu,26109,京都市伏見区,Kyouto-shi Fushimi-ku,26109010,伏見,Fushimi,1,0,3,準工,560,362,8672,0.001,0,0,0,0,0.014,0.003,O,0,3,,,,,30,31,30,31,31,30,31,30,30,29,28,31,712,735,715,736,739,715,738,715,733,729,667,738,0.001,0.002,0.001,0.001,0.001,0.001,0.001,0.001,0.001,0.001,0.001,0.001,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.014,0.006,0.004,0.004,0.004,0.003,0.002,0.004,0.004,0.006,0.004,0.004,0.004,0.004,0.002,0.002,0.002,0.002,0.002,0.002,0.003,0.003,0.002,0.002,,,,,,,,,,,,,,,,,,,,,,,,
2018,1,01,SO2,3,26,京都府,Kyoto-fu,26110,京都市山科区,Kyouto-shi Yamashina-ku,26110010,山科,Yamashina,1,0,1,住,560,362,8666,0.001,0,0,0,0,0.012,0.002,O,0,3,,,,,30,31,30,31,31,30,31,30,30,29,28,31,714,735,713,738,738,713,736,715,733,727,666,738,0.001,0.001,0.001,0.001,0.001,0.001,0.001,0.001,0.001,0.001,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.012,0.006,0.004,0.003,0.004,0.003,0.002,0.003,0.003,0.004,0.004,0.003,0.003,0.003,0.002,0.002,0.002,0.001,0.001,0.002,0.002,0.002,0.002,0.001,,,,,,,,,,,,,,,,,,,,,,,,
2018,1,01,SO2,3,26,京都府,Kyoto-fu,26111,京都市西京区,Kyouto-shi Nishikyou-ku,26111010,西京,Nishikyou,1,0,1,住,560,361,8661,0.001,0,0,0,0,0.011,0.002,O,0,3,,,,,30,31,30,31,31,30,31,29,30,29,28,31,714,736,715,738,739,713,736,705,733,728,667,737,0.001,0.001,0.001,0.001,0.001,0.001,0.001,0.001,0.001,0.001,0,0.001,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.011,0.006,0.004,0.004,0.003,0.002,0.005,0.008,0.003,0.004,0.004,0.006,0.003,0.003,0.002,0.001,0.002,0.001,0.001,0.002,0.001,0.001,0.001,0.002,,,,,,,,,,,,,,,,,,,,,,,,

[arthurlli]

I used to handle it with excel but haven't tried with iconv, which seems interesting. Another way to import the data is using python pandas library with argument "encoding='cp932": import pandas as pd dtf = pd.read_csv('j012018_01.txt', sep=',', encoding='cp932') print(dtf) it'll show the result as same as previous one.