Open KYUNGJU-SHIM opened 2 years ago
정보
현재 사용되는 스케쥴러는 BackgroundScheduler 인스턴스임
BackgroundScheduler 는 별도 thread 를 생성하여 작업을 실행 (A scheduler that runs in the background using a separate thread
)
장고에서는 스레드마다 새로운 DB 커넥션 을 생성함
그리고 각 스레드에서 만들어딪 커넥션은 스레드가 실행시킨 프로세스 종료될 때 함께 닫힘
pymysql.err.InterfaceError: (0, '')
은 커넥션이 제대로 종료되지 않았을 때 주로 발생
가설
<1> 스케쥴러가 멀티 스레드를 사용하여 커넥션이 다량으로 생성 & 닫히지 않고 대기하는 상황이 발생하였다 - ThreadPool 이 아닌 Thread 이고, API 호출은 여러번이지만 실행 작업 자체는 하나이기 때문에 다량 커넥션이 생성된다고 보기 어렵지 않을까 - 에러 발생 위치가 작업이 시작되는 부분이어서 커넥션이 누적된 결과로 보기 어려움 => 작업의 첫 쿼리에서 발생하였다는 것이 힌트아닌 힌트 <2> 실행 동작에서의 exception 발생 등으로 커넥션이 제대로 종료되지 않았다 - 역시나 작업 첫 쿼리 발생이라 원인으로 보기 어려움 ㅠ <3> ...그리고 각 스레드에서 만들어딪 커넥션은 스레드가 실행시킨 프로세스 종료될 때 함께 닫힘
이 제대로 동작하지 않는다는 레퍼런스도 있음, 맞다면, 가능성은 어플리케이션 구동 이후 계속 누적된 커넥션들로 위 이슈가 발생
https://zkkmin.com/2018/05/11/database-connection-leak-while-using-threads-in-django/
=> 커넥션을 수동으로 닫는 방법으로 보완 예정
022-07-07T15:00:00.236416841Z Job "execute_update_currencies_task (trigger: cron[hour='0', minute='0', second='0'], next run at: 2022-07-09 00:00:00 KST)" raised an exception
2022-07-07T15:00:00.243210904Z Traceback (most recent call last):
2022-07-07T15:00:00.243221317Z File "/usr/local/lib/python3.8/dist-packages/django/db/backends/mysql/base.py", line 272, in _set_autocommit
2022-07-07T15:00:00.243226610Z self.connection.autocommit(autocommit)
2022-07-07T15:00:00.243231210Z File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 437, in autocommit
2022-07-07T15:00:00.243237372Z self._send_autocommit_mode()
2022-07-07T15:00:00.243241997Z File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 452, in _send_autocommit_mode
2022-07-07T15:00:00.243246839Z self._execute_command(
2022-07-07T15:00:00.243251090Z File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 793, in _execute_command
2022-07-07T15:00:00.243255619Z raise err.InterfaceError(0, "")
2022-07-07T15:00:00.243260088Z pymysql.err.InterfaceError: (0, '')
2022-07-07T15:00:00.243264313Z
2022-07-07T15:00:00.243268413Z The above exception was the direct cause of the following exception:
2022-07-07T15:00:00.243274444Z
2022-07-07T15:00:00.243278560Z Traceback (most recent call last):
2022-07-07T15:00:00.243282784Z File "/usr/local/lib/python3.8/dist-packages/apscheduler/executors/base.py", line 125, in run_job
2022-07-07T15:00:00.243287303Z retval = job.func(*job.args, **job.kwargs)
2022-07-07T15:00:00.243291552Z File "/home/ubuntu/travel_calculator/./currencies/services.py", line 33, in execute
2022-07-07T15:00:00.243295968Z Currency.objects.update_or_create(
2022-07-07T15:00:00.243300176Z File "/usr/local/lib/python3.8/dist-packages/django/db/models/manager.py", line 85, in manager_method
2022-07-07T15:00:00.243304594Z return getattr(self.get_queryset(), name)(*args, **kwargs)
2022-07-07T15:00:00.243310105Z File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 605, in update_or_create
2022-07-07T15:00:00.243323012Z with transaction.atomic(using=self.db):
2022-07-07T15:00:00.243327169Z File "/usr/local/lib/python3.8/dist-packages/django/db/transaction.py", line 207, in __enter__
2022-07-07T15:00:00.243331196Z connection.set_autocommit(False, force_begin_transaction_with_broken_autocommit=True)
2022-07-07T15:00:00.243335114Z File "/usr/local/lib/python3.8/dist-packages/django/db/backends/base/base.py", line 415, in set_autocommit
2022-07-07T15:00:00.243339161Z self._set_autocommit(autocommit)
2022-07-07T15:00:00.243343037Z File "/usr/local/lib/python3.8/dist-packages/django/db/backends/mysql/base.py", line 272, in _set_autocommit
2022-07-07T15:00:00.243347313Z self.connection.autocommit(autocommit)
2022-07-07T15:00:00.243352558Z File "/usr/local/lib/python3.8/dist-packages/django/db/utils.py", line 90, in __exit__
2022-07-07T15:00:00.243408312Z raise dj_exc_value.with_traceback(traceback) from exc_value
2022-07-07T15:00:00.243412397Z File "/usr/local/lib/python3.8/dist-packages/django/db/backends/mysql/base.py", line 272, in _set_autocommit
2022-07-07T15:00:00.243416513Z self.connection.autocommit(autocommit)
2022-07-07T15:00:00.243420366Z File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 437, in autocommit
2022-07-07T15:00:00.243424506Z self._send_autocommit_mode()
2022-07-07T15:00:00.243428349Z File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 452, in _send_autocommit_mode
2022-07-07T15:00:00.243433652Z self._execute_command(
2022-07-07T15:00:00.243437525Z File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 793, in _execute_command
2022-07-07T15:00:00.243441524Z raise err.InterfaceError(0, "")
2022-07-07T15:00:00.243445527Z django.db.utils.InterfaceError: (0, '')
2022-07-07T15:03:06.377930923Z Not Found: /robots.txt
2022-07-07T15:03:06.378791425Z [pid: 9|app: 0|req: 1109/1109] 192.168.96.1 () {44 vars in 629 bytes} [Thu Jul 7 15:03:06 2022] GET /robots.txt => generated 2505 bytes in 6 msecs (HTTP/1.0 404) 5 headers in 159 bytes (1 switches on core 0)
2022-07-07T15:03:14.414752262Z Not Found: /
2022-07-07T15:03:14.415693573Z [pid: 9|app: 0|req: 1110/1110] 192.168.96.1 () {48 vars in 698 bytes} [Thu Jul 7 15:03:14 2022] GET / => generated 2457 bytes in 5 msecs (HTTP/1.0 404) 5 headers in 159 bytes (1 switches on core 0)
2022-07-07T15:24:25.421512389Z Not Found: /robots.txt
2022-07-07T15:24:25.422423752Z [pid: 9|app: 0|req: 1111/1111] 192.168.96.1 () {38 vars in 697 bytes} [Thu Jul 7 15:24:25 2022] GET /robots.txt => generated 2505 bytes in 5 msecs (HTTP/1.0 404) 5 headers in 159 bytes (1 switches on core 0)
2022-07-07T15:24:25.792410439Z Not Found: /
2022-07-07T15:24:25.793284895Z [pid: 9|app: 0|req: 1112/1112] 192.168.96.1 () {38 vars in 677 bytes} [Thu Jul 7 15:24:25 2022] GET / => generated 2457 bytes in 5 msecs (HTTP/1.0 404) 5 headers in 159 bytes (1 switches on core 0)
2022-07-07T16:00:00.023787489Z Job "execute_update_hotel_prices_task (trigger: cron[hour='1', minute='0', second='0'], next run at: 2022-07-09 01:00:00 KST)" raised an exception
2022-07-07T16:00:00.023868592Z Traceback (most recent call last):
2022-07-07T16:00:00.023875277Z File "/usr/local/lib/python3.8/dist-packages/django/db/backends/utils.py", line 84, in _execute
2022-07-07T16:00:00.023880937Z return self.cursor.execute(sql, params)
2022-07-07T16:00:00.023884993Z File "/usr/local/lib/python3.8/dist-packages/django/db/backends/mysql/base.py", line 73, in execute
2022-07-07T16:00:00.023889099Z return self.cursor.execute(query, args)
2022-07-07T16:00:00.023893015Z File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute
2022-07-07T16:00:00.023897086Z result = self._query(query)
2022-07-07T16:00:00.023900953Z File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query
2022-07-07T16:00:00.023905036Z conn.query(q)
2022-07-07T16:00:00.023908833Z File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 547, in query
2022-07-07T16:00:00.023914128Z self._execute_command(COMMAND.COM_QUERY, sql)
2022-07-07T16:00:00.023918088Z File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 793, in _execute_command
2022-07-07T16:00:00.023922094Z raise err.InterfaceError(0, "")
2022-07-07T16:00:00.023926021Z pymysql.err.InterfaceError: (0, '')
2022-07-07T16:00:00.023929861Z
2022-07-07T16:00:00.023933688Z The above exception was the direct cause of the following exception:
2022-07-07T16:00:00.023937613Z
2022-07-07T16:00:00.023941340Z Traceback (most recent call last):
2022-07-07T16:00:00.023968075Z File "/usr/local/lib/python3.8/dist-packages/apscheduler/executors/base.py", line 125, in run_job
2022-07-07T16:00:00.023973251Z retval = job.func(*job.args, **job.kwargs)
2022-07-07T16:00:00.023977116Z File "/home/ubuntu/travel_calculator/./hotels/services/schedule_manager.py", line 42, in execute
2022-07-07T16:00:00.023981204Z for city in cities:
2022-07-07T16:00:00.023984966Z File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 280, in __iter__
2022-07-07T16:00:00.023989070Z self._fetch_all()
2022-07-07T16:00:00.023992856Z File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 1324, in _fetch_all
2022-07-07T16:00:00.023996996Z self._result_cache = list(self._iterable_class(self))
2022-07-07T16:00:00.024007154Z File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 51, in __iter__
2022-07-07T16:00:00.024011146Z results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
2022-07-07T16:00:00.024015202Z File "/usr/local/lib/python3.8/dist-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
2022-07-07T16:00:00.024019233Z cursor.execute(sql, params)
2022-07-07T16:00:00.024028788Z File "/usr/local/lib/python3.8/dist-packages/django/db/backends/utils.py", line 98, in execute
2022-07-07T16:00:00.024033027Z return super().execute(sql, params)
2022-07-07T16:00:00.024036865Z File "/usr/local/lib/python3.8/dist-packages/django/db/backends/utils.py", line 66, in execute
2022-07-07T16:00:00.024041048Z return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
2022-07-07T16:00:00.024046356Z File "/usr/local/lib/python3.8/dist-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
2022-07-07T16:00:00.024050386Z return executor(sql, params, many, context)
2022-07-07T16:00:00.024054245Z File "/usr/local/lib/python3.8/dist-packages/django/db/backends/utils.py", line 84, in _execute
2022-07-07T16:00:00.024058284Z return self.cursor.execute(sql, params)
2022-07-07T16:00:00.024080199Z File "/usr/local/lib/python3.8/dist-packages/django/db/utils.py", line 90, in __exit__
2022-07-07T16:00:00.024084375Z raise dj_exc_value.with_traceback(traceback) from exc_value
2022-07-07T16:00:00.024088246Z File "/usr/local/lib/python3.8/dist-packages/django/db/backends/utils.py", line 84, in _execute
2022-07-07T16:00:00.024092215Z return self.cursor.execute(sql, params)
2022-07-07T16:00:00.024097371Z File "/usr/local/lib/python3.8/dist-packages/django/db/backends/mysql/base.py", line 73, in execute
2022-07-07T16:00:00.024101362Z return self.cursor.execute(query, args)
2022-07-07T16:00:00.024105295Z File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute
2022-07-07T16:00:00.024109247Z result = self._query(query)
2022-07-07T16:00:00.024113069Z File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query
2022-07-07T16:00:00.024117110Z conn.query(q)
2022-07-07T16:00:00.024120917Z File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 547, in query
2022-07-07T16:00:00.024124883Z self._execute_command(COMMAND.COM_QUERY, sql)
2022-07-07T16:00:00.024130038Z File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 793, in _execute_command
2022-07-07T16:00:00.024134037Z raise err.InterfaceError(0, "")
2022-07-07T16:00:00.024137945Z django.db.utils.InterfaceError: (0, '')
2022-07-07T17:00:00.017455633Z Job "execute_update_flight_summary_task (trigger: cron[hour='2', minute='0', second='0'], next run at: 2022-07-09 02:00:00 KST)" raised an exception
2022-07-07T17:00:00.017496199Z Traceback (most recent call last):
2022-07-07T17:00:00.017502383Z File "/usr/local/lib/python3.8/dist-packages/django/db/backends/utils.py", line 84, in _execute
2022-07-07T17:00:00.017507254Z return self.cursor.execute(sql, params)
2022-07-07T17:00:00.017513298Z File "/usr/local/lib/python3.8/dist-packages/django/db/backends/mysql/base.py", line 73, in execute
2022-07-07T17:00:00.017527076Z return self.cursor.execute(query, args)
2022-07-07T17:00:00.017531359Z File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute
2022-07-07T17:00:00.017535454Z result = self._query(query)
2022-07-07T17:00:00.017564007Z File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query
2022-07-07T17:00:00.017569327Z conn.query(q)
2022-07-07T17:00:00.017573275Z File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 547, in query
2022-07-07T17:00:00.017577351Z self._execute_command(COMMAND.COM_QUERY, sql)
2022-07-07T17:00:00.017584915Z File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 793, in _execute_command
2022-07-07T17:00:00.017589310Z raise err.InterfaceError(0, "")
2022-07-07T17:00:00.017593407Z pymysql.err.InterfaceError: (0, '')
2022-07-07T17:00:00.017597332Z
2022-07-07T17:00:00.017601234Z The above exception was the direct cause of the following exception:
2022-07-07T17:00:00.017605256Z
2022-07-07T17:00:00.017609205Z Traceback (most recent call last):
2022-07-07T17:00:00.017613278Z File "/usr/local/lib/python3.8/dist-packages/apscheduler/executors/base.py", line 125, in run_job
2022-07-07T17:00:00.017618615Z retval = job.func(*job.args, **job.kwargs)
2022-07-07T17:00:00.017622458Z File "/home/ubuntu/travel_calculator/./flight/services/flight.py", line 234, in execute
2022-07-07T17:00:00.017626536Z target_dates = self.get_target_dates()
2022-07-07T17:00:00.017630389Z File "/home/ubuntu/travel_calculator/./flight/services/flight.py", line 227, in get_target_dates
2022-07-07T17:00:00.017634388Z start_date = self.get_target_start_date()
2022-07-07T17:00:00.017638230Z File "/home/ubuntu/travel_calculator/./flight/services/flight.py", line 217, in get_target_start_date
2022-07-07T17:00:00.017642265Z latest_summarized_summary = self.get_latest_summarized_summary()
2022-07-07T17:00:00.017646145Z File "/home/ubuntu/travel_calculator/./flight/services/flight.py", line 209, in get_latest_summarized_summary
2022-07-07T17:00:00.017651417Z return FlightSummary.objects.all().order_by("-created_at").first()
2022-07-07T17:00:00.017655379Z File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 674, in first
2022-07-07T17:00:00.017679281Z for obj in (self if self.ordered else self.order_by('pk'))[:1]:
2022-07-07T17:00:00.017683314Z File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 280, in __iter__
2022-07-07T17:00:00.017687298Z self._fetch_all()
2022-07-07T17:00:00.017691085Z File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 1324, in _fetch_all
2022-07-07T17:00:00.017695348Z self._result_cache = list(self._iterable_class(self))
2022-07-07T17:00:00.017699289Z File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 51, in __iter__
2022-07-07T17:00:00.017709237Z results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
2022-07-07T17:00:00.017713326Z File "/usr/local/lib/python3.8/dist-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
2022-07-07T17:00:00.017717436Z cursor.execute(sql, params)
2022-07-07T17:00:00.017721245Z File "/usr/local/lib/python3.8/dist-packages/django/db/backends/utils.py", line 98, in execute
2022-07-07T17:00:00.017725344Z return super().execute(sql, params)
2022-07-07T17:00:00.017729157Z File "/usr/local/lib/python3.8/dist-packages/django/db/backends/utils.py", line 66, in execute
2022-07-07T17:00:00.017733165Z return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
2022-07-07T17:00:00.017737156Z File "/usr/local/lib/python3.8/dist-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
2022-07-07T17:00:00.017742427Z return executor(sql, params, many, context)
2022-07-07T17:00:00.017746348Z File "/usr/local/lib/python3.8/dist-packages/django/db/backends/utils.py", line 84, in _execute
2022-07-07T17:00:00.017750301Z return self.cursor.execute(sql, params)
2022-07-07T17:00:00.017754149Z File "/usr/local/lib/python3.8/dist-packages/django/db/utils.py", line 90, in __exit__
2022-07-07T17:00:00.017758224Z raise dj_exc_value.with_traceback(traceback) from exc_value
2022-07-07T17:00:00.017762068Z File "/usr/local/lib/python3.8/dist-packages/django/db/backends/utils.py", line 84, in _execute
2022-07-07T17:00:00.017766013Z return self.cursor.execute(sql, params)
2022-07-07T17:00:00.017787111Z File "/usr/local/lib/python3.8/dist-packages/django/db/backends/mysql/base.py", line 73, in execute
2022-07-07T17:00:00.017793265Z return self.cursor.execute(query, args)
2022-07-07T17:00:00.017797111Z File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute
2022-07-07T17:00:00.017801088Z result = self._query(query)
2022-07-07T17:00:00.017804926Z File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query
2022-07-07T17:00:00.017808866Z conn.query(q)
2022-07-07T17:00:00.017812746Z File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 547, in query
2022-07-07T17:00:00.017858791Z self._execute_command(COMMAND.COM_QUERY, sql)
2022-07-07T17:00:00.017866490Z File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 793, in _execute_command
2022-07-07T17:00:00.017870723Z raise err.InterfaceError(0, "")
2022-07-07T17:00:00.017874682Z django.db.utils.InterfaceError: (0, '')
2022-07-07T17:01:26.624493856Z Not Found: /
7/9 : 위의 커넥션 leak 관련 에러는 발생하지 않음 (단, 커넥션이 더 누적 ...? 되는지 확인을 위해 좀 더 지켜봐야함)