search

abhishekkrthakur / csv_test

25 stars 24 forks source link

Pandas solution #4

Open abhishekkrthakur opened 3 years ago

abhishekkrthakur commented 3 years ago

Runs in ~192 seconds on my machine

import pandas as pd
import glob

import time

start = time.time()

path = "../data/"
all_files = glob.glob(path + "/*.csv")
li = []

for filename in all_files:
    df = pd.read_csv(filename, index_col=None, header=0)
    li.append(df)

frame = pd.concat(li, axis=0, ignore_index=True)
frame.to_csv("out_pandas.csv", index=False)

end = time.time()
print(end - start)