search

ianozsvald / dtype_pandas_numpy_speed_test

Speed test on Pandas or NumPy
MIT License
2 stars 2 forks source link

Timings #1

Open gwvr opened 4 years ago

gwvr commented 4 years ago

i7z is Intel only, hence inxi -C.

$ inxi -C CPU: Topology: 6-Core model: AMD Ryzen 5 3600 bits: 64 type: MT MCP L2 cache: 3072 KiB Speed: 2199 MHz min/max: 2200/3600 MHz Core speeds (MHz): 1: 2200 2: 2200 3: 2199 4: 2203 5: 2201 6: 2200 7: 2200 8: 2196 9: 2197 10: 2196 11: 2200 12: 2200

timings

abayirli commented 4 years ago

Here are the results for my case:

CPU: Topology: 6-Core model: Intel Core i7-9750H bits: 64 type: MT MCP L2 cache: 12.0 MiB Speed: 800 MHz min/max: 800/4500 MHz Core speeds (MHz): 1: 800 2: 800 3: 800 4: 800 5: 800 6: 800 7: 801 8: 800 9: 800 10: 800 11: 800 12: 800 

Pandas: 1.0.3
NumPy: 1.18.1
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000000 entries, 0 to 9999999
Data columns (total 9 columns):
 #   Column      Dtype   
---  ------      -----   
 0   int64       int64   
 1   int32       int32   
 2   int16       int16   
 3   int8        int8    
 4   uint8       uint8   
 5   longdouble  float128
 6   float64     float64 
 7   float32     float32 
 8   float16     float16 
dtypes: float128(1), float16(1), float32(1), float64(1), int16(1), int32(1), int64(1), int8(1), uint8(1)
memory usage: 438.7 MB
Working on int64
Working on int32
Working on int16
Working on int8
Working on uint8
Working on longdouble
Working on float64
Working on float32
Working on float16
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1800 entries, 0 to 1799
Data columns (total 3 columns):
 #   Column   Non-Null Count  Dtype  
---  ------   --------------  -----  
 0   fn_name  1800 non-null   object 
 1   col      1800 non-null   object 
 2   t        1800 non-null   float64
dtypes: float64(1), object(2)
memory usage: 42.3+ KB

timings

ThomasLecocq commented 4 years ago

Windows 10 laptop, i7, 16GB ram

Pandas: 1.0.1
NumPy: 1.18.1
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000000 entries, 0 to 9999999
Data columns (total 9 columns):
 #   Column      Dtype
---  ------      -----
 0   int64       int64
 1   int32       int32
 2   int16       int16
 3   int8        int8
 4   uint8       uint8
 5   longdouble  float64
 6   float64     float64
 7   float32     float32
 8   float16     float16
dtypes: float16(1), float32(1), float64(2), int16(1), int32(1), int64(1), int8(1), uint8(1)
memory usage: 362.4 MB
Working on int64
Working on int32
Working on int16
Working on int8
Working on uint8
Working on longdouble
Working on float64
Working on float32
Working on float16
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1800 entries, 0 to 1799
Data columns (total 3 columns):
 #   Column   Non-Null Count  Dtype
---  ------   --------------  -----
 0   fn_name  1800 non-null   object
 1   col      1800 non-null   object
 2   t        1800 non-null   float64
dtypes: float64(1), object(2)
memory usage: 42.3+ KB

timings pickle can't be uploaded here: https://asktom.be/python/tmp/

Linux (Ubuntu) computing blade, 16 core, 192GB ram:

Pandas: 0.25.1
NumPy: 1.17.2
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000000 entries, 0 to 9999999
Data columns (total 9 columns):
int64         int64
int32         int32
int16         int16
int8          int8
uint8         uint8
longdouble    float128
float64       float64
float32       float32
float16       float16
dtypes: float128(1), float16(1), float32(1), float64(1), int16(1), int32(1), int64(1), int8(1), uint8(1)
memory usage: 438.7 MB
Working on int64
Working on int32
Working on int16
Working on int8
Working on uint8
Working on longdouble
Working on float64
Working on float32
Working on float16
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1800 entries, 0 to 1799
Data columns (total 3 columns):
fn_name    1800 non-null object
col        1800 non-null object
t          1800 non-null float64
dtypes: float64(1), object(2)
memory usage: 42.3+ KB

linux-timings

pickle can't be uploaded here: https://asktom.be/python/tmp/

ThomasLecocq commented 4 years ago

one more on a fresh env with only pandas, numpy and matplotlib and their dependencies:

Pandas: 1.0.4
NumPy: 1.18.5
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000000 entries, 0 to 9999999
Data columns (total 9 columns):
 #   Column      Dtype
---  ------      -----
 0   int64       int64
 1   int32       int32
 2   int16       int16
 3   int8        int8
 4   uint8       uint8
 5   longdouble  float128
 6   float64     float64
 7   float32     float32
 8   float16     float16
dtypes: float128(1), float16(1), float32(1), float64(1), int16(1), int32(1), int64(1), int8(1), uint8(1)
memory usage: 438.7 MB
Working on int64
Working on int32
Working on int16
Working on int8
Working on uint8
Working on longdouble
Working on float64
Working on float32
Working on float16
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1800 entries, 0 to 1799
Data columns (total 3 columns):
 #   Column   Non-Null Count  Dtype
---  ------   --------------  -----
 0   fn_name  1800 non-null   object
 1   col      1800 non-null   object
 2   t        1800 non-null   float64
dtypes: float64(1), object(2)
memory usage: 42.3+ KB

linux2-timings

ThomasLecocq commented 4 years ago

with the last pre-1.0 version:

Pandas: 0.25.3
NumPy: 1.18.5
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000000 entries, 0 to 9999999
Data columns (total 9 columns):
int64         int64
int32         int32
int16         int16
int8          int8
uint8         uint8
longdouble    float128
float64       float64
float32       float32
float16       float16
dtypes: float128(1), float16(1), float32(1), float64(1), int16(1), int32(1), int64(1), int8(1), uint8(1)
memory usage: 438.7 MB
Working on int64
Working on int32
Working on int16
Working on int8
Working on uint8
Working on longdouble
Working on float64
Working on float32
Working on float16
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1800 entries, 0 to 1799
Data columns (total 3 columns):
fn_name    1800 non-null object
col        1800 non-null object
t          1800 non-null float64
dtypes: float64(1), object(2)
memory usage: 42.3+ KB

linux3-timings

ThomasLecocq commented 4 years ago

just for fun, ran the previous code (removing fstrings etc) on py27:

('Pandas:', u'0.24.2')
('NumPy:', '1.16.5')
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000000 entries, 0 to 9999999
Data columns (total 9 columns):
float16       float16
float32       float32
float64       float64
int16         int16
int32         int32
int64         int64
int8          int8
longdouble    float128
uint8         uint8
dtypes: float128(1), float16(1), float32(1), float64(1), int16(1), int32(1), int64(1), int8(1), uint8(1)
memory usage: 438.7 MB
('Working on', 'float16')
('Working on', 'float32')
('Working on', 'float64')
('Working on', 'int16')
('Working on', 'int32')
('Working on', 'int64')
('Working on', 'int8')
('Working on', 'longdouble')
('Working on', 'uint8')
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1800 entries, 0 to 1799
Data columns (total 3 columns):
col        1800 non-null object
fn_name    1800 non-null object
t          1800 non-null float64
dtypes: float64(1), object(2)
memory usage: 42.3+ KB

py27timings