search

airscholar / e2e-data-engineering

An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.
https://www.youtube.com/watch?v=GqAcTrqKcrY
180 stars 81 forks source link

Container start error #3

Closed vraj-apto closed 9 months ago

vraj-apto commented 9 months ago

Hi @airscholar, Attaching to broker, cassandra, control-center, postgres-1, scheduler-1, spark-master-1, spark-worker-1, webserver-1, schema-registry, zookeeper Gracefully stopping... (press Ctrl+C again to force) Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/run/desktop/mnt/host/c/Users/vrajp/OneDrive/Desktop/Data Engineering/requirements.txt" to rootfs at "/opt/airflow/requirements.txt": mount /run/desktop/mnt/host/c/Users/vrajp/OneDrive/Desktop/Data Engineering/requirements.txt:/opt/airflow/requirements.txt (via /proc/self/fd/9), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

I have my 'requirements.txt' in a specified location. Could you/anyone please help me with this error?

airscholar commented 9 months ago

@vraj-apto This error is usually because you did not create the requirements.txt file before running the docker compose up -d now when you ran the command, it created a folder with the name on your local system.

I will suggest that before you run the docker compose command, you should delete the requirements.txt folder created when you initially ran the command. Delete the folder with the name, run the pip freeze > requirements.txt to create a requirements.txt then docker compose up -d again.