Review data pipeline

[dao-duc-tung]

1. Bài Feature Store

Thêm dev_requirements.txt vào data_pipeline dir để cài đặt các libs trong quá trình học, thay vì hướng dẫn cài từng lib

Giống file training_pipeline/dev_requirements.txt trong bài hướng dẫn training pipeline

2. Bài Xây dựng pipeline

Fix bullets

3. Bài Xây dựng pipeline

Cần thêm 1 bước nói ng dùng click bật pipeline để chạy

Cần thêm 1 bước nữa để chụp ảnh pipeline đã hoàn thiện

Fix format chỗ "dagid: _db_to_offline_store"

4. Bài Xây dựng pipeline

Đã làm theo hướng dẫn nhưng pipeline đầu tiên db_to_offline_store nhưng không chạy, bị lỗi Permission denied như dưới đây.

*** Reading local file: /opt/airflow/logs/dag_id=db_to_offline_store/run_id=manual__2022-10-16T14:01:47.432949+00:00/task_id=ingest_task/attempt=1.log
[2022-10-16, 14:01:48 UTC] {taskinstance.py:1171} INFO - Dependencies all met for <TaskInstance: db_to_offline_store.ingest_task manual__2022-10-16T14:01:47.432949+00:00 [queued]>
[2022-10-16, 14:01:48 UTC] {taskinstance.py:1171} INFO - Dependencies all met for <TaskInstance: db_to_offline_store.ingest_task manual__2022-10-16T14:01:47.432949+00:00 [queued]>
[2022-10-16, 14:01:48 UTC] {taskinstance.py:1368} INFO - 
--------------------------------------------------------------------------------
[2022-10-16, 14:01:48 UTC] {taskinstance.py:1369} INFO - Starting attempt 1 of 1
[2022-10-16, 14:01:48 UTC] {taskinstance.py:1370} INFO - 
--------------------------------------------------------------------------------
[2022-10-16, 14:01:48 UTC] {taskinstance.py:1389} INFO - Executing <Task(DockerOperator): ingest_task> on 2022-10-16 14:01:47.432949+00:00
[2022-10-16, 14:01:48 UTC] {standard_task_runner.py:52} INFO - Started process 1308 to run task
[2022-10-16, 14:01:48 UTC] {standard_task_runner.py:79} INFO - Running: ['***', 'tasks', 'run', 'db_to_offline_store', 'ingest_task', 'manual__2022-10-16T14:01:47.432949+00:00', '--job-id', '29', '--raw', '--subdir', 'DAGS_FOLDER/data_pipeline/db_to_offline_store.py', '--cfg-path', '/tmp/tmpe5oiegu4', '--error-file', '/tmp/tmp7fwjhxg6']
[2022-10-16, 14:01:48 UTC] {standard_task_runner.py:80} INFO - Job 29: Subtask ingest_task
[2022-10-16, 14:01:48 UTC] {task_command.py:371} INFO - Running <TaskInstance: db_to_offline_store.ingest_task manual__2022-10-16T14:01:47.432949+00:00 [running]> on host 085aa7617ef8
[2022-10-16, 14:01:48 UTC] {taskinstance.py:1583} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=mlopsvn
AIRFLOW_CTX_DAG_ID=db_to_offline_store
AIRFLOW_CTX_TASK_ID=ingest_task
AIRFLOW_CTX_EXECUTION_DATE=2022-10-16T14:01:47.432949+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=manual__2022-10-16T14:01:47.432949+00:00
[2022-10-16, 14:01:48 UTC] {taskinstance.py:1902} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
    chunked=chunked,
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.7/http/client.py", line 976, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/transport/unixconn.py", line 30, in connect
    sock.connect(self.unix_socket)
PermissionError: [Errno 13] Permission denied

Lỗi này được fix bằng càng chạy

sudo chmod 666 /var/run/docker.sock

[dao-duc-tung]

@quan-dang a mới update log cho lỗi 4