airscholar / SparkingFlow

This project demonstrates how to use Apache Airflow to submit jobs to Apache spark cluster in different programming laguages using Python, Scala and Java as an example.
https://www.datamasterylab.com/home/course/apache-airflow-on-steriods-for-data-engineers/9
15 stars 15 forks source link

Aiflow Scheduler & Webserver keeps exiting #4

Open RhythmBear opened 4 months ago

RhythmBear commented 4 months ago

`ERROR! Maximum number of retries (20) reached. 2024-02-20 22:47:30 2024-02-20 22:47:30 Last check result: 2024-02-20 22:47:30 $ airflow db check 2024-02-20 22:47:30 Unable to load the config, contains a configuration error. 2024-02-20 22:47:30 Traceback (most recent call last): 2024-02-20 22:47:30 File "/usr/local/lib/python3.11/pathlib.py", line 1116, in mkdir 2024-02-20 22:47:30 os.mkdir(self, mode) 2024-02-20 22:47:30 FileNotFoundError: [Errno 2] No such file or directory: '/opt/airflow/logs/scheduler/2024-02-20' 2024-02-20 22:47:30 2024-02-20 22:47:30 During handling of the above exception, another exception occurred: 2024-02-20 22:47:30 2024-02-20 22:47:30 Traceback (most recent call last): 2024-02-20 22:47:30 File "/usr/local/lib/python3.11/logging/config.py", line 573, in configure 2024-02-20 22:47:30 handler = self.configure_handler(handlers[name]) 2024-02-20 22:47:30 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-02-20 22:47:30 File "/usr/local/lib/python3.11/logging/config.py", line 758, in configure_handler 2024-02-20 22:47:30 result = factory(**kwargs) 2024-02-20 22:47:30 ^^^^^^^^^^^^^^^^^ 2024-02-20 22:47:30 File "/home/airflow/.local/lib/python3.11/site-packages/airflow/utils/log/file_processor_handler.py", line 50, in init 2024-02-20 22:47:30 Path(self._get_log_directory()).mkdir(parents=True, exist_ok=True) 2024-02-20 22:47:30 File "/usr/local/lib/python3.11/pathlib.py", line 1120, in mkdir 2024-02-20 22:47:30 self.parent.mkdir(parents=True, exist_ok=True) 2024-02-20 22:47:30 File "/usr/local/lib/python3.11/pathlib.py", line 1116, in mkdir 2024-02-20 22:47:30 os.mkdir(self, mode) 2024-02-20 22:47:30 PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler' 2024-02-20 22:47:30 2024-02-20 22:47:30 The above exception was the direct cause of the following exception: 2024-02-20 22:47:30 2024-02-20 22:47:30 Traceback (most recent call last): 2024-02-20 22:47:30 File "/home/airflow/.local/bin/airflow", line 5, in 2024-02-20 22:47:30 from airflow.main import main 2024-02-20 22:47:30 File "/home/airflow/.local/lib/python3.11/site-packages/airflow/init.py", line 68, in 2024-02-20 22:47:30 settings.initialize() 2024-02-20 22:47:30 File "/home/airflow/.local/lib/python3.11/site-packages/airflow/settings.py", line 521, in initialize 2024-02-20 22:47:30 LOGGING_CLASS_PATH = configure_logging() 2024-02-20 22:47:30 ^^^^^^^^^^^^^^^^^^^ 2024-02-20 22:47:30 File "/home/airflow/.local/lib/python3.11/site-packages/airflow/logging_config.py", line 74, in configure_logging 2024-02-20 22:47:30 raise e 2024-02-20 22:47:30 File "/home/airflow/.local/lib/python3.11/site-packages/airflow/logging_config.py", line 69, in configure_logging 2024-02-20 22:47:30 dictConfig(logging_config) 2024-02-20 22:47:30 File "/usr/local/lib/python3.11/logging/config.py", line 823, in dictConfig 2024-02-20 22:47:30 dictConfigClass(config).configure() 2024-02-20 22:47:30 File "/usr/local/lib/python3.11/logging/config.py", line 580, in configure 2024-02-20 22:47:30 raise ValueError('Unable to configure handler ' 2024-02-20 22:47:30 ValueError: Unable to configure handler 'processor'```

RhythmBear commented 4 months ago

This is the error message i keep seeing when I open the scheduler and Webserver's logs