Open qinxuye opened 4 years ago
Definition of haversine.
import numpy as np
def haversine(lat1, lon1, lat2, lon2):
dlon = np.radians(lon2 - lon1)
dlat = np.radians(lat2 - lat1)
a = np.sin(dlat / 2) ** 2 + np.cos(np.radians(lat1)) * np.cos(np.radians(lat2)) * np.sin(dlon / 2) ** 2
c = 2 * np.arcsin(np.sqrt(a))
r = 6371
return c * r
Data is available at: https://www.kaggle.com/max-mind/world-cities-database/data
For the local run, I executed read_csv
first, then I can accomplish apply
stuff, but there is no any speed boost.
In [40]: df = pd.read_csv('Downloads/world-cities-database/worldcitiespop.csv')
/Users/xuyeqin/miniconda3/lib/python3.7/site-packages/IPython/core/interactiveshell.py:3058: DtypeWarning: Columns (3) have mixed types.Specify dtype option on import or set low_memory=False.
interactivity=interactivity, compiler=compiler, result=result)
In [41]: df.head()
Out[41]:
Country City AccentCity Region Population Latitude Longitude
0 ad aixas Aixàs 6 NaN 42.483333 1.466667
1 ad aixirivali Aixirivali 6 NaN 42.466667 1.500000
2 ad aixirivall Aixirivall 6 NaN 42.466667 1.500000
3 ad aixirvall Aixirvall 6 NaN 42.466667 1.500000
4 ad aixovall Aixovall 6 NaN 42.466667 1.483333
In [42]: %%time
...: df.apply(lambda r: haversine(r['Latitude'], r['Longitude'], 30.25, 120.17), result_type='reduce', axis=1)
...:
...:
CPU times: user 1min 49s, sys: 410 ms, total: 1min 50s
Wall time: 1min 50s
Out[42]:
0 9789.135208
1 9788.270528
2 9788.270528
3 9788.270528
4 9789.307210
...
3173953 10899.720735
3173954 11220.703197
3173955 10912.645753
3173956 11318.038981
3173957 11141.080171
Length: 3173958, dtype: float64
In [55]: df = md.read_csv('Downloads/world-cities-database/worldcitiespop.csv', chunk_bytes='16M', dtype={'Region': object
...: })
In [56]: df.execute(fetch=False)
In [57]: %%time
...: df.apply(lambda r: haversine(r['Latitude'], r['Longitude'], 30.25, 120.17), result_type='reduce', axis=1).execute
...: ()
...:
...:
CPU times: user 1min 50s, sys: 724 ms, total: 1min 51s
Wall time: 1min 48s
Out[57]:
0 9789.135208
1 9788.270528
2 9788.270528
3 9788.270528
4 9789.307210
...
248061 10899.720735
248062 11220.703197
248063 10912.645753
248064 11318.038981
248065 11141.080171
Length: 3173958, dtype: float64
Thus, I try to use local cluster, still I executed read_csv
first, then run the apply
, still cannot see any promotion.
In [4]: cluster = new_cluster(web=True)
/opt/concourse/worker/volumes/live/b066e71c-c221-49d1-533f-e4cc9fc71f42/volume/arrow-cpp_1540956703648/work/cpp/src/plasma/store.cc:971: Allowing the Plasma store to use up to 6.87195GB of memory.
/opt/concourse/worker/volumes/live/b066e71c-c221-49d1-533f-e4cc9fc71f42/volume/arrow-cpp_1540956703648/work/cpp/src/plasma/store.cc:1001: Starting object store with directory /tmp and huge page support disabled
WARNING:bokeh.server.util:Host wildcard '*' will allow connections originating from multiple (or possibly all) hostnames or IPs. Use non-wildcard values to restrict access explicitly
Web endpoint started at http://0.0.0.0:61352
In [5]: WARNING:tornado.access:404 GET /favicon.ico (127.0.0.1) 0.35ms
In [7]: import mars.dataframe as md
In [8]: df = md.read_csv('Downloads/world-cities-database/worldcitiespop.csv', chunk_bytes='16M', dtype={'Region': object}
...: )
In [9]: df.execute(fetch=False)
/Users/xuyeqin/miniconda3/lib/python3.7/site-packages/pyarrow/serialization.py:104: FutureWarning: The SparseSeries class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
if isinstance(obj, pd.SparseSeries):
/Users/xuyeqin/miniconda3/lib/python3.7/site-packages/pyarrow/serialization.py:93: FutureWarning: The SparseDataFrame class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
if isinstance(obj, pd.SparseDataFrame):
/Users/xuyeqin/miniconda3/lib/python3.7/site-packages/pyarrow/serialization.py:104: FutureWarning: The SparseSeries class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
if isinstance(obj, pd.SparseSeries):
/Users/xuyeqin/miniconda3/lib/python3.7/site-packages/pyarrow/serialization.py:93: FutureWarning: The SparseDataFrame class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
if isinstance(obj, pd.SparseDataFrame):
/Users/xuyeqin/miniconda3/lib/python3.7/site-packages/pyarrow/serialization.py:93: FutureWarning: The SparseDataFrame class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
if isinstance(obj, pd.SparseDataFrame):
/Users/xuyeqin/miniconda3/lib/python3.7/site-packages/pyarrow/serialization.py:93: FutureWarning: The SparseDataFrame class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
if isinstance(obj, pd.SparseDataFrame):
/Users/xuyeqin/miniconda3/lib/python3.7/site-packages/pyarrow/serialization.py:93: FutureWarning: The SparseDataFrame class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
if isinstance(obj, pd.SparseDataFrame):
/Users/xuyeqin/miniconda3/lib/python3.7/site-packages/pyarrow/serialization.py:93: FutureWarning: The SparseDataFrame class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
if isinstance(obj, pd.SparseDataFrame):
/Users/xuyeqin/miniconda3/lib/python3.7/site-packages/pyarrow/serialization.py:93: FutureWarning: The SparseDataFrame class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
if isinstance(obj, pd.SparseDataFrame):
/Users/xuyeqin/miniconda3/lib/python3.7/site-packages/pyarrow/serialization.py:93: FutureWarning: The SparseDataFrame class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
if isinstance(obj, pd.SparseDataFrame):
/Users/xuyeqin/miniconda3/lib/python3.7/site-packages/pyarrow/serialization.py:93: FutureWarning: The SparseDataFrame class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
if isinstance(obj, pd.SparseDataFrame):
/Users/xuyeqin/miniconda3/lib/python3.7/site-packages/pyarrow/serialization.py:93: FutureWarning: The SparseDataFrame class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
if isinstance(obj, pd.SparseDataFrame):
/Users/xuyeqin/miniconda3/lib/python3.7/site-packages/pyarrow/serialization.py:93: FutureWarning: The SparseDataFrame class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
if isinstance(obj, pd.SparseDataFrame):
/Users/xuyeqin/miniconda3/lib/python3.7/site-packages/pyarrow/serialization.py:93: FutureWarning: The SparseDataFrame class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
if isinstance(obj, pd.SparseDataFrame):
In [10]: %%time
...: df.apply(lambda r: haversine(r['Latitude'], r['Longitude'], 30.25, 120.17), result_type='reduce', axis=1).execute
...: ()
...:
...:
/Users/xuyeqin/miniconda3/lib/python3.7/site-packages/pyarrow/serialization.py:104: FutureWarning: The SparseSeries class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
if isinstance(obj, pd.SparseSeries):
/Users/xuyeqin/miniconda3/lib/python3.7/site-packages/pyarrow/serialization.py:93: FutureWarning: The SparseDataFrame class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
if isinstance(obj, pd.SparseDataFrame):
/Users/xuyeqin/miniconda3/lib/python3.7/site-packages/pyarrow/serialization.py:104: FutureWarning: The SparseSeries class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
if isinstance(obj, pd.SparseSeries):
CPU times: user 23.6 s, sys: 226 ms, total: 23.9 s
Wall time: 2min 20s
Out[10]:
0 9789.135208
1 9788.270528
2 9788.270528
3 9788.270528
4 9789.307210
...
248061 10899.720735
248062 11220.703197
248063 10912.645753
248064 11318.038981
248065 11141.080171
Length: 3173958, dtype: float64
Describe the bug
read_csv().apply(func, axis=1) failed.
To Reproduce To help us reproducing this bug, please provide information below: