[Enhancement] Automate Data Download

[yehyunsuh]

What

Downloading Data

Why

Automate downloading data process

How

1. Download .xls file OpenData_PotOpenTabletIdntfc20220412.xls
2. conda install openpyxl
3. Create /data directory
4. python data_preprocessing.py

data_preprocessing.py

import os, time
import pandas as pd
import urllib.request as req
from tqdm import tqdm

filename = 'OpenData_PotOpenTabletIdntfc20220412.xls'

df = pd.read_excel(filename, engine='openpyxl')
data_dir = "/opt/ml/final_project/data"  ## data download directory
start = time.time()

for idx in tqdm(range(len(df))):
    image_key = list(df['품목일련번호'])[idx]
    image_url = list(df['큰제품이미지'])[idx]
    downloaded_file = req.urlretrieve(image_url, f"{data_dir}/{image_key}.jpg")
print(time.time()-start)

ETA: over 100 mins

[seoulsky-field]

성공적으로 진행되고 있습니다! 감사합니다.

[yehyunsuh]

Issue closed. Pull Request will be done after mentoring.