search

airscholar / e2e-data-engineering

An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.
https://www.youtube.com/watch?v=GqAcTrqKcrY
131 stars 62 forks source link

ython dags/kafka_stream.py not working #7

Open oavioz opened 1 month ago

oavioz commented 1 month ago

Hello,

I'm running docker desktop on windows and all containers are running.

when i trying to run python dags/kafka_stream.py from my VSCODE this is the message i'm getting. topic not created!

WARNING:root:OSError while attempting to symlink the latest log directory C:\Users\USER\Documents\Oshik\Python_Projects\data-engineering\dags\kafka_stream.py:60 RemovedInAirflow3Warning: Param schedule_interval is deprecated and will be removed in a future release. Please use schedule instead.

Any help?

Thanks!

oavioz commented 1 month ago

This is what i got on ChatGPT:

This warning message indicates that there was an error related to creating a symbolic link to the latest log directory. This can occur due to various reasons such as:

Permission Issues: The process does not have the necessary permissions to create a symbolic link. File System Restrictions: The file system where the link is being created does not support symbolic links. Path Issues: The specified path for the symbolic link is incorrect or invalid. Existing Symlink: A symlink with the same name already exists and cannot be overwritten. File System Quotas or Limits: Exceeding quotas or limits on the number of files or links.

Appriciate your help here

emrhnksck commented 8 hours ago

Using schedule instead of schedule_interval working for me