Open vpogrebi opened 2 years ago
open-collective-bot[bot]
commented
2 years ago Hey @vpogrebi :wave:, Thank you for opening an issue. We will get back to you as soon as we can. Also, check out our Open Collective and consider backing us - every little helps!
We also offer priority support for our sponsors. If you require immediate assistance please consider sponsoring us.
vpogrebi
commented
2 years ago [2022-06-07 17:04:11,297: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2022-06-07 17:04:11,602: INFO/MainProcess] Scheduler: Sending due task AZMSABG01R1-enable-2022-06-07-17:04:09.182051_b70fd6d8-4f3a-4d8d-8069-0b0d09ddc4f9 (nautobot.extras.jobs.scheduled_job_handler)
[2022-06-07 17:04:11,785: INFO/MainProcess] Scheduler: Sending due task AZMSABG01R1-console-2022-06-07-17:04:07.342526_d2e2aa52-db47-475c-b934-c12d43d91c57 (nautobot.extras.jobs.scheduled_job_handler)
[2022-06-07 17:04:17,094: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2022-06-07 17:04:17,490: INFO/MainProcess] Scheduler: Sending due task AZMSABG01R1-console-2022-06-07-17:04:07.342526_d2e2aa52-db47-475c-b934-c12d43d91c57 (nautobot.extras.jobs.scheduled_job_handler)In this log - observe following: 1. Two tasks were started at the same time ("_Scheduler: Sending due task_" logs at 2022-06-07 17:04:11) 2. Few seconds later, one task completed - indicated by the "_DatabaseScheduler: Schedule changed_" log (2022-06-07 17:04:17,094) 3. This schedule change event triggered scheduler sync - which in turn executed _tick()_ call with event argument corresponding to the second (still running) task. 4. In _tick()_, _self.is_due(entry)_ returned True (task is due) - which resulted in scheduler send another message to start this second task: "_Scheduler: Sending due task_" log at 2022-06-07 17:04:17,490 End result - one of the tasks (_AZMSABG01R1-console-2022-06-07-17:04:07.342526_d2e2aa52-db47-475c-b934-c12d43d91c57_) executed twice. # Additional thoughts I noticed that every time this happens - two (or more) tasks are being kicked off by the same (single) sync event: - Single "_DatabaseScheduler: Schedule changed_" log followed by ... - Multiple (two or more) "_Scheduler: Sending due task_" logs I am not sure if this fact has anything to do with this issue (bug) - but somehow I have strong feeling that it does. I suspect that these multiple tasks get "associated" with the same (single) event, and when one of the tasks completes - sync (_tick()_) evaluates each of the other tasks (schedule entries), discovers that they are "due" (_is_due()_ returns True) and sends message to (re)start them again. I may be wrong, but... this is one suspicion that I've got after giving this issue some more thought.
auvipy
commented
2 years ago you could just modify the previous issue. btw can you please try & check new release https://github.com/celery/django-celery-beat/releases/tag/v2.3.0 ?
vpogrebi
commented
2 years ago @auvipy
you could just modify the previous issue. btw can you please try & check new release https://github.com/celery/django-celery-beat/releases/tag/v2.3.0 ?
Turns out someone else from the team has upgraded our dev lane to django_celery_beat v2.3.0 - and that did not help.
vpogrebi
commented
2 years ago One other thought... _django_celery_beat.schedulers.ModelEntry.isdue() - has this logic:
# ONE OFF TASK: Disable one off tasks after they've ran once
if self.model.one_off and self.model.enabled \
and self.model.total_run_count > 0:
self.model.enabled = False
self.model.total_run_count = 0 # Reset
self.model.no_changes = False # Mark the model entry as changed
self.model.save()
# Don't recheck
return schedules.schedstate(False, NEVER_CHECK_TIMEOUT)When testing, I noticed that self.total_run_count does not match self.model.total_run_count when task (that is already running) is evaluated again - which results in starting same task again. I was able to test my suspicion by providing a custom override of _celery.beat.Scheduler.applyasync() that offsets (corrects) this situation:
def apply_async(self, entry, producer=None, advance=True, **kwargs):
"""Send event to the worker to start task execution.
This is an override of the celery.beat.Scheduler.apply_async() method.
After executing original apply_async() call, it synchronizes 'total_run_count'
and saves model. This prevents same task from being started again while it is running.
"""
resp = super().apply_async(entry, producer=None, advance=True, **kwargs)
if entry.total_run_count != entry.model.total_run_count:
entry.total_run_count = entry.model.total_run_count
entry.model.save()
return respUsing this custom override - resolves this issue!
I was able to prove that by ensuring that self.total_run_count and self.model.total_run_count are same at a time apply_async() is executed - somehow prevents multiple executions of the same task. Therefore, I suspect that ensuring that these two versions of total_run_count match at a time apply_async() is executed - somehow ensures that self.model.total_run_count > 0 within _django_celery_beat.schedulers.ModelEntry.isdue() which prevents duplicate task execution.
auvipy
commented
2 years ago I think we can move this issue to django-celery-beat now :)
vpogrebi
commented
2 years ago There is one other issue that is being resolved by the above "override" (listed in a comment above)... Before (the override), total_run_count stored in the database after job's execution - was unreliable: for the job that ran and completed - total_run_count could be 0, 1, 2, etc. For the job that executed once - it was either 0 or 1, and I could not figure out the "pattern", or predict in advance (while the job was running) whether it would end up with 0 or 1.
Somehow, my override also fixes that issue, too - all total_run_count are 1 for the one-off jobs.
ja-gooding
commented
2 years ago Are there any plans to fix these issues in any reasonable timeframe? Months or less? As far as I can tell, celery, especially celery beat, is fundamentally broken. It is not even able to know if a task is already running or not, and this causes issue, upon issue, upon issue, down the road.
There have been comments about this going back half a decade, with no root solution in sight. These issues and core fundamentals should have been thoughtfully and robustly tested with units-testing from day one with sound software engineering.
And please don't say if you have an issue; fork it or put in a pull request. At a certain point, the problems are too numerous and deep to fix. That's why I am asking if there is a plan to resolve these issues soon, or should we cut our losses and move on?
pythobot
commented
1 year ago I just upgraded to django-celery-beat==2.4.0 with celery==5.2.3, and that didn't help. now tasks duplicate even more.
Does anybody know how to solve this issue?
mishra-ankit-dev
commented
1 year ago Hi Team,
Is it still unresolved ? I am still facing the above stated issue.
I think the hotfix has been applied to apply_async, delay still has the same issue. apply_async seems to work fine with this fix added.
Maryam077
commented
1 year ago Hello Team,
Is this still not resolved?
BaiJiangJie
commented
1 year ago auvipy
commented
1 year ago Try it.
660
i would like your feedback on this
guillaumeldc
commented
1 year ago @auvipy is there a sure way to reproduce this issue. The way it's described in the original post doesn't always lead to multiple trigger of the same task. I do have the same issue in production but can't identify what is causing the issue. It appears randomly. I would be happy to take the new release for a spin but need a baseline for testing.
guillaumeldc
commented
1 year ago Ok, I finally found a way to consistently reproduce the error. I will do some more testing but wanted to provide the code I'm using to reproduce it. What @vpogrebi is mentioning about having the duplicates around the "schedule changed" is correct and this is what is causing the issue. Here are the logs of the code executions, as you can see task_3_1_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 is sent twice at the beginning and at the end.
[2023-08-15 23:17:00,071: INFO/MainProcess] Scheduler: Sending due task task_3_1_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,080: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,085: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,121: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,142: INFO/MainProcess] Scheduler: Sending due task task_3_2_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,159: INFO/MainProcess] Scheduler: Sending due task task_3_3_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,163: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,192: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,214: INFO/MainProcess] Scheduler: Sending due task task_3_4_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,220: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,301: INFO/MainProcess] Scheduler: Sending due task task_3_5_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,308: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,340: INFO/MainProcess] Scheduler: Sending due task task_3_6_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,346: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,379: INFO/MainProcess] Scheduler: Sending due task task_3_7_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,384: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,416: INFO/MainProcess] Scheduler: Sending due task task_3_8_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,422: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,450: INFO/MainProcess] Scheduler: Sending due task task_3_9_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,456: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,481: INFO/MainProcess] Scheduler: Sending due task task_3_10_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,486: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,511: INFO/MainProcess] Scheduler: Sending due task task_3_11_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,516: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,538: INFO/MainProcess] Scheduler: Sending due task task_3_12_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,542: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,568: INFO/MainProcess] Scheduler: Sending due task task_3_13_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,571: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,594: INFO/MainProcess] Scheduler: Sending due task task_3_14_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,598: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,623: INFO/MainProcess] Scheduler: Sending due task task_3_15_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,628: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,649: INFO/MainProcess] Scheduler: Sending due task task_3_16_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,655: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,677: INFO/MainProcess] Scheduler: Sending due task task_3_17_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,681: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,701: INFO/MainProcess] Scheduler: Sending due task task_3_18_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,705: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,729: INFO/MainProcess] Scheduler: Sending due task task_3_19_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,733: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,758: INFO/MainProcess] Scheduler: Sending due task task_3_20_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,763: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,784: INFO/MainProcess] Scheduler: Sending due task task_3_1_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)You can use this code to reproduce:
def sleep_until(target_datetime):
"""
Pauses the script until the specified datetime is reached.
:param target_datetime: datetime.datetime object specifying the time to wait until
"""
now = timezone.now()
duration = (target_datetime - now).total_seconds()
if duration > 0: # Only sleep if the target datetime is in the future
time.sleep(duration)
def django_celery_beat_test(nb_tasks=5, nb_batches=5):
now = timezone.now().replace(second=0, microsecond=0)
iter_id = uuid.uuid4()
for i in range(1, nb_batches + 1):
ps = []
clocked_time = now + timedelta(minutes=i)
clocked_schedule = ClockedSchedule.objects.get_or_create(
clocked_time=clocked_time,
)[0]
for x in range(1, nb_tasks + 1):
task_name = f"task_{i}_{x}_{iter_id}"
ps.append(
PeriodicTask(
name=task_name,
enabled=True,
clocked=clocked_schedule,
one_off=True,
task="send_task",
expires=clocked_time + timedelta(minutes=5),
kwargs=json.dumps(
{
"source_model": "stream",
}
),
)
)
PeriodicTask.objects.bulk_create(ps)
for task in ps:
PeriodicTasks.changed(task)
sleep_until(clocked_time)send_task is as follow:
@app.task(name="send_task", bind=True, base=AbortableTask, track_started=True)
def send_task(self, source_mode=None):
import time
import random
print(f"id is: {self.request.id}")
time.sleep(random.randint(10, 90))
pass@auvipy I tested the main branch briefly using the method I'm highlighting above.
It's just 3 iterations, but I can run this all night long and analyze the logs if needed. To parse the logs, you can use this.
def find_duplicate_sequences(text, sequence_length, starting_pattern=""):
"""
Find duplicated sequences in a long text that begin with a certain pattern.
:param text: Text to search for duplicated sequences
:param sequence_length: Length of sequences to search for (including starting pattern)
:param starting_pattern: Initial pattern to start the sequence with
:return: Set of duplicated sequences
"""
if len(starting_pattern) >= sequence_length:
raise ValueError(
"The starting pattern is equal or longer than the sequence length!"
)
seen_sequences = set()
duplicated_sequences = set()
# Adjust sequence length based on the starting pattern's length
sequence_length -= len(starting_pattern)
for i in range(0, len(text) - sequence_length + 1):
if text[i : i + len(starting_pattern)] == starting_pattern:
sequence = text[i : i + sequence_length + len(starting_pattern)]
if sequence in seen_sequences:
duplicated_sequences.add(sequence)
seen_sequences.add(sequence)
return duplicated_sequencesso is it apparent that a new release could some how mitigate this issue?
guillaumeldc
commented
1 year ago Well based on my initial findings, yes. Now keep in mind that we need to see results on a long term analysis which I would be happy to report on when deployed at scale.
auvipy
commented
1 year ago thank you for your kind feedback
sar5430
commented
11 months ago Hi! Do you plan to release a new version anytime soon ?
AlessandroSalvetti
commented
6 months ago Hi any news on this? I have the same problem and sometimes a task that imports stuff, runs twice, importing the double of the data!
alexgmin
commented
5 months ago According to the release notes of the 2.6.0 version on March 3, the PR https://github.com/celery/django-celery-beat/pull/660 that solves https://github.com/celery/django-celery-beat/issues/388 and according to the comments here, also this one. @guillaumeldc Do you still have this issue?
partizaans
commented
2 months ago According to the release notes of the 2.6.0 version on March 3, the PR #660 that solves #388 and according to the comments here, also this one. @guillaumeldc Do you still have this issue?
@alexgmin For our production instance still happend on 2.6.0. Single instance of beat and multiple workers.
thedrow
commented
1 month ago @cclauss Can you please take a look and see if you can fix this issue?
Checklist
masterbranch of Celery.Mandatory Debugging Information
celery -A proj reportin the issue. (if you are not able to do this, then at least specify the Celery version affected).masterbranch of Celery.pip freezein the issue.Optional Debugging Information
Related Issues and Possible Duplicates
Related Issues
Possible Duplicates
Environment & Settings
Celery version: 5.2.7
celery reportOutput:``` software -> celery:5.2.7 (dawn-chorus) kombu:5.2.4 py:3.8.13 billiard:3.6.4.0 redis:4.3.1 platform -> system:Linux arch:64bit, ELF kernel version:3.10.0-1160.62.1.el7.x86_64 imp:CPython loader -> celery.loaders.default.Loader settings -> transport:redis results:disabled deprecated_settings: None ```
Steps to Reproduce
Required Dependencies
Python Packages
pip freezeOutput:``` amqp @ file:///data/zkci8im/.cache/pypoetry/artifacts/22/94/d8/7787846037071cf203833b4891393d7fee750673537eccf83374f4e68a/amqp-5.1.1-py3-none-any.whl aniso8601 @ file:///data/zkci8im/.cache/pypoetry/artifacts/cb/bb/0b/f3e74131c6390dbb7622617f9c8a590661cd4910e2e342c07ab40fe8d5/aniso8601-7.0.0-py2.py3-none-any.whl ansible==5.1.0 ansible-core @ file:///data/zkci8im/.cache/pypoetry/artifacts/6d/b4/59/459ce5c2e21433c1443caaecc97cbaa260085761ef58ad0794f9ef1894/ansible-core-2.12.6.tar.gz asgiref @ file:///data/zkci8im/.cache/pypoetry/artifacts/60/d3/77/8e5b9c433ba4707af49c8a8879a4792689501bc21468d866dcba293387/asgiref-3.5.2-py3-none-any.whl async-timeout @ file:///data/zkci8im/.cache/pypoetry/artifacts/9e/20/3e/134e90bed6cb6814ec11555be6900e30ab163641792ec5003bd6aa59d7/async_timeout-4.0.2-py3-none-any.whl attrs @ file:///data/zkci8im/.cache/pypoetry/artifacts/4e/b8/3f/509c2f0fdcb7f29b8a253972760d0c7830bb0b0403328ae0eab3cea799/attrs-21.4.0-py2.py3-none-any.whl billiard @ file:///data/zkci8im/.cache/pypoetry/artifacts/d7/9f/c2/e3fc46d8ea60110a6d6c6b23cabe49bb06db28d7ef3fcef9f560c37eb1/billiard-3.6.4.0-py3-none-any.whl boto3 @ file:///data/zkci8im/.cache/pypoetry/artifacts/1d/2a/5a/b2c1f3b722c8e02330dba16637ed5b9f12d195bef6226e2d72e6112a7b/boto3-1.23.10-py3-none-any.whl botocore @ file:///data/zkci8im/.cache/pypoetry/artifacts/10/6b/56/b501c514fffa9d9a45ecf0d334e7717c7d963cae46d6cdfd1c90d0a192/botocore-1.26.10-py3-none-any.whl cached-property @ file:///data/zkci8im/.cache/pypoetry/artifacts/c4/6c/91/92f6a180669978ba5b5cfeea34239598e4a62f798452e667bab474eca3/cached_property-1.5.2-py2.py3-none-any.whl celery @ file:///data/zkci8im/.cache/pypoetry/artifacts/ad/72/4d/52e0f4d27000064a2266f9d939f1a9e6e8a385310ea28ef12102c76768/celery-5.2.7-py3-none-any.whl certifi @ file:///data/zkci8im/.cache/pypoetry/artifacts/9a/24/91/91ca3aa0f55638b8b2e2e0751f2e1cba1f342c36d4eea2028b6fc96431/certifi-2022.5.18.1-py3-none-any.whl cffi @ file:///data/zkci8im/.cache/pypoetry/artifacts/e8/1b/a4/670c457d4f006c93740689d474309dfa24a89159e27073f4b3fc44bb98/cffi-1.15.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl charset-normalizer @ file:///data/zkci8im/.cache/pypoetry/artifacts/f7/e6/c2/e485049fad66be34cb361166725f59e4e9ef24804c75713cbb3bd6699d/charset_normalizer-2.0.12-py3-none-any.whl click @ file:///data/zkci8im/.cache/pypoetry/artifacts/e2/03/35/a5b93ff313b614d10ef8c1dfecb531470309e3799cbf6fbfeed0a1ce0d/click-8.1.3-py3-none-any.whl click-didyoumean @ file:///data/zkci8im/.cache/pypoetry/artifacts/39/30/2c/da96c86e55be9b67c908b36e64e815e3c9db43db453e65b2f57ef7f52d/click_didyoumean-0.3.0-py3-none-any.whl click-plugins @ file:///data/zkci8im/.cache/pypoetry/artifacts/d0/16/38/b7c58246dc79c9091376c792a8981f86944dd0ebe9581a58ba9a5ce0ea/click_plugins-1.1.1-py2.py3-none-any.whl click-repl @ file:///data/zkci8im/.cache/pypoetry/artifacts/f2/c0/d5/b58b59f548e7a632f7780eae3f73f0258eb1e926d52f0170c21304ad4b/click_repl-0.2.0-py3-none-any.whl colorama @ file:///data/zkci8im/.cache/pypoetry/artifacts/da/e1/ed/183662e38ad81210d1ff1a86d8ce34409a7d6a1f7d0ef629715dab9344/colorama-0.4.4-py2.py3-none-any.whl coreapi @ file:///data/zkci8im/.cache/pypoetry/artifacts/e0/15/0a/3368989abeaa634a32dd92a149db85845315e1528a202d163bd8def96b/coreapi-2.3.3-py2.py3-none-any.whl coreschema @ file:///data/zkci8im/.cache/pypoetry/artifacts/ac/20/6f/08c281ffb0c8eff16526a9b74d5af61dbcac6172ac2605f4db6c4c480a/coreschema-0.0.4.tar.gz cryptography @ file:///data/zkci8im/.cache/pypoetry/artifacts/dd/ef/07/3dd5838cbc8f2b3a22dc4db8399410fe13a6a81f32ce2b40d2c946c788/cryptography-37.0.2-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl cx-Oracle @ file:///data/zkci8im/.cache/pypoetry/artifacts/73/a1/48/01f42f211f534f73aed19b07156e2f58bc51f3e931b7ff24e916bc6183/cx_Oracle-8.3.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl cycler @ file:///data/zkci8im/.cache/pypoetry/artifacts/32/5f/b5/69e6bd574bcff38cf319bbc2937a6f71f47b5532fff18dfd7dc221591a/cycler-0.11.0-py3-none-any.whl defer @ file:///data/zkci8im/.cache/pypoetry/artifacts/7f/d4/64/12aa60c68128ded2c3b83733ede1376668571bee98f2234f998d6fccfd/defer-1.0.4.tar.gz defusedxml @ file:///data/zkci8im/.cache/pypoetry/artifacts/52/8e/a8/6b6077e99f00435c289f1e6d72413a518d187c42f4f16018e9748996aa/defusedxml-0.7.1-py2.py3-none-any.whl Deprecated @ file:///data/zkci8im/.cache/pypoetry/artifacts/1b/df/01/0a69c6fa606522ad7535a9433d6f353e57e674a41f23b0c75851455ade/Deprecated-1.2.13-py2.py3-none-any.whl diffsync @ file:///data/zkci8im/.cache/pypoetry/artifacts/25/f2/03/71d9fcbe3efd41c93400434afd2eecc791427d325371195d79b0ecba7e/diffsync-1.4.3-py3-none-any.whl Django @ file:///data/zkci8im/.cache/pypoetry/artifacts/84/b8/62/2618b9e6130ebb2661834718637fd186dcaf9d4bbdbe5d8e40b5044e90/Django-3.1.14-py3-none-any.whl django-ajax-tables @ file:///data/zkci8im/.cache/pypoetry/artifacts/41/4b/c2/0b226c181d25ebe682f1c331ff835fc913da1840e63970f43b2d4d5025/django_ajax_tables-1.1.1-py3-none-any.whl django-appconf @ file:///data/zkci8im/.cache/pypoetry/artifacts/15/b0/42/820bd462dcd763df6043e873d416d949595b144ac76bda950a7059322e/django_appconf-1.0.5-py3-none-any.whl django-cacheops @ file:///data/zkci8im/.cache/pypoetry/artifacts/77/ab/1a/858382fe25b8025eddd02dc56b2ed986f86d9343e51186984d4c87de0b/django_cacheops-5.1-py2.py3-none-any.whl django-celery-beat @ file:///data/zkci8im/.cache/pypoetry/artifacts/3f/71/03/feac735288ca43683afd8f5e889f6c1670345f879acfb60f5391f4f559/django_celery_beat-2.2.1-py2.py3-none-any.whl django-constance @ file:///data/zkci8im/.cache/pypoetry/artifacts/92/73/f6/b27afffb80673c8e7cff1f06405d06d8078cce3ccf8156b94349e7417e/django_constance-2.8.0-py3-none-any.whl django-cors-headers @ file:///data/zkci8im/.cache/pypoetry/artifacts/ad/95/72/c2b1d92874644571f8e4b3b90951706ac3d96db0ae5f8ed50a196fc1b1/django_cors_headers-3.7.0-py3-none-any.whl django-cryptography @ file:///data/zkci8im/.cache/pypoetry/artifacts/be/36/af/6730dd6bebebfddbcac7ffe452b9b6db03977659dad14b31f5417f1f85/django_cryptography-1.0-py3-none-any.whl django-db-file-storage @ file:///data/zkci8im/.cache/pypoetry/artifacts/73/ed/f1/84bb26eaf28ee7772f62bf319f95bc47b58964c9c45758b73b7df111ee/django-db-file-storage-0.5.5.tar.gz django-downloadview @ file:///data/zkci8im/.cache/pypoetry/artifacts/b1/2e/86/bbffe2f5c2568e7f427eb695e6b21f312104be6bbd968b66012ff5804a/django_downloadview-2.3.0-py3-none-any.whl django-entangled @ file:///data/zkci8im/.cache/pypoetry/artifacts/2e/37/48/145379ef54ff01cb65af0eb2397a87e83fdf209dcc1aaebb1e4a913bbd/django_entangled-0.5.3-py3-none-any.whl django-extensions @ file:///data/zkci8im/.cache/pypoetry/artifacts/e4/44/36/92d6e2883b730fa9f5870d82725511e42c87732a524094cef75e9080b9/django_extensions-3.1.5-py3-none-any.whl django-filter @ file:///data/zkci8im/.cache/pypoetry/artifacts/0c/56/fc/d46f2e843498b83624f789ec451754d9caa1da8e04dafcbbb61d273a1e/django_filter-2.4.0-py3-none-any.whl django-health-check @ file:///data/zkci8im/.cache/pypoetry/artifacts/68/fa/24/e2b9ae1477deafa6b5dcbf3878000c6eca08b90da6cbb2ef09b31cfe3a/django_health_check-3.16.5-py2.py3-none-any.whl django-jinja @ file:///data/zkci8im/.cache/pypoetry/artifacts/7d/7d/e5/ebaf16f467f945ac40af64aa5470aa6c16b6d15d1bba0d941c128ddf3d/django_jinja-2.7.1-py3-none-any.whl django-js-asset @ file:///data/zkci8im/.cache/pypoetry/artifacts/8d/c3/d7/d7ec51ee2330df3de9033568464f80b75a61399948db022bf6a2faf604/django_js_asset-2.0.0-py3-none-any.whl django-mptt @ file:///data/zkci8im/.cache/pypoetry/artifacts/26/12/ef/a313335a7225e0b4cd0f2b6d6cfe0057523ce7b5fb803295fd2dc7b25a/django_mptt-0.11.0-py2.py3-none-any.whl django-ordered-model @ file:///data/zkci8im/.cache/pypoetry/artifacts/ac/a1/1c/7deedf732708cd7a331a55900332d66b6377a30637f28de520f7da79d0/django_ordered_model-3.6-py3-none-any.whl django-picklefield @ file:///data/zkci8im/.cache/pypoetry/artifacts/a9/e2/04/f4ab1ce6c3ba035cb9f3492562ff2772f6bcd522690498e141ea225844/django_picklefield-3.0.1-py3-none-any.whl django-prometheus @ file:///data/zkci8im/.cache/pypoetry/artifacts/a8/08/89/a058c01936e8c1ff21a3c3696258faf6dd5c236ad1ca7ce74ec7a2744a/django_prometheus-2.1.0-py2.py3-none-any.whl django-redis @ file:///data/zkci8im/.cache/pypoetry/artifacts/e6/f4/01/66c9ff3f089f1e3cd1a4363cf0931618d904c54676ace18cfdf94a3f83/django_redis-5.2.0-py3-none-any.whl django-rq @ file:///data/zkci8im/.cache/pypoetry/artifacts/ce/18/a5/8ca3fd96dee56aa6a520a9fbd8305feb9fd8cb1eab77d0fff267e2d872/django_rq-2.4.1-py2.py3-none-any.whl django-storages @ file:///data/zkci8im/.cache/pypoetry/artifacts/84/52/bb/5ac21f52e4e07dcddd223ea08d7ec643a9ae872226ad79b1e4daed3769/django_storages-1.12.3-py3-none-any.whl django-tables2 @ file:///data/zkci8im/.cache/pypoetry/artifacts/78/46/84/69794e310d82527ae5e95fefc3fdde4d223a76d7e9dcbf58446b61e597/django_tables2-2.3.4-py2.py3-none-any.whl django-taggit @ file:///data/zkci8im/.cache/pypoetry/artifacts/0a/ba/45/8fd1c16264accb6c7b76eb13fd536bdbc36bce8e5e4f37f9d976a834c0/django_taggit-1.3.0-py3-none-any.whl django-timezone-field @ file:///data/zkci8im/.cache/pypoetry/artifacts/bc/1a/83/6e3f3fd42c1cb4bc4744298de8f3eee498573bd3a5d45e87b0dd58eb4b/django_timezone_field-4.1.2-py3-none-any.whl django-webserver @ file:///data/zkci8im/.cache/pypoetry/artifacts/39/e2/f5/581bc4bba43156a12191374c64b3de7e357126ab5a32b51e8c832ad52b/django_webserver-1.2.0-py2.py3-none-any.whl djangorestframework @ file:///data/zkci8im/.cache/pypoetry/artifacts/a0/f6/64/3c17dd26b71862f114e3edeb3fbbe5a113b97fbabfc02ddc01ac73d325/djangorestframework-3.12.4-py3-none-any.whl drf-yasg @ file:///data/zkci8im/.cache/pypoetry/artifacts/aa/b1/f2/b6d572c8bf1b12ea52c42f019d95c72855926fbf2e4a21158eb68c9425/drf_yasg-1.20.0-py2.py3-none-any.whl ecdsa @ file:///data/zkci8im/.cache/pypoetry/artifacts/88/fc/5e/0169779073adb3d1920c38c24f1b47601e447622ec221514fdd9cdb1cb/ecdsa-0.17.0-py2.py3-none-any.whl et-xmlfile @ file:///data/zkci8im/.cache/pypoetry/artifacts/fd/6d/4f/3e758a754df0ad5b0be20ae9e4c3a4d3b07e778b38a0ff76c980cb6f9d/et_xmlfile-1.1.0-py3-none-any.whl fonttools @ file:///data/zkci8im/.cache/pypoetry/artifacts/57/40/9d/ad9a026cdfc88d8ab195b1228e254f69a91a3a7084d69b4ff79de5ba6d/fonttools-4.33.3-py3-none-any.whl funcy @ file:///data/zkci8im/.cache/pypoetry/artifacts/ff/29/3c/7612d8bae9b3af86333d22fe6156fa13ae2d26c70803da765b7f87f8ed/funcy-1.17-py2.py3-none-any.whl GES.Entitlements @ file:///data/zkci8im/.cache/pypoetry/artifacts/76/1d/60/7e657bbeee6cc1becde51c9d21ed5618547b8d71a5e42fe2e02c0c2275/GES.Entitlements-7.4.0.0-py3-none-any.whl gitdb @ file:///data/zkci8im/.cache/pypoetry/artifacts/20/48/e1/e92559540498e6fc81acc2b79725aed4831869ac4930e4875f042a3bc6/gitdb-4.0.9-py3-none-any.whl GitPython @ file:///data/zkci8im/.cache/pypoetry/artifacts/45/99/a3/8dc50fe3361368d219d1dc12474ce03ee847dede7e83853f36238f7525/GitPython-3.1.18-py3-none-any.whl graphene @ file:///data/zkci8im/.cache/pypoetry/artifacts/e4/b4/e5/ebf568635788b1888e94de08f61bf6df7bc035c66fdb8130caccde0534/graphene-2.1.9-py2.py3-none-any.whl graphene-django @ file:///data/zkci8im/.cache/pypoetry/artifacts/ed/bb/91/d0a9103f49ff61786d14c64d62b62d3bf3ac06aee2f44eed7f19c7d285/graphene_django-2.15.0-py2.py3-none-any.whl graphene-django-optimizer @ file:///data/zkci8im/.cache/pypoetry/artifacts/e7/41/e6/a86f233609fe6e007737bc3960a4d869720e2d7f6fbbac49fd76a7b90f/graphene-django-optimizer-0.8.0.tar.gz graphql-core @ file:///data/zkci8im/.cache/pypoetry/artifacts/42/0a/5d/522bcaafc471de5533ac9aeef689edc9402c048bc5566cc746f7e17928/graphql_core-2.3.2-py2.py3-none-any.whl graphql-relay @ file:///data/zkci8im/.cache/pypoetry/artifacts/2d/bb/de/db5baddb597438d931a9cb986596b9382ab4153c628523bd52e4446fd6/graphql_relay-2.0.1-py3-none-any.whl ibm-db @ file:///data/zkci8im/.cache/pypoetry/artifacts/40/f3/30/bf498333533e3962d2dd380fa2c15dd56280b298d4d6d29e57373c2c4e/ibm_db-3.1.1.tar.gz idna @ file:///data/zkci8im/.cache/pypoetry/artifacts/6f/d1/a9/d0625f343d7d25e3e0effced7539d67c94c0426cd32bdf4fbb3a7528f0/idna-3.3-py3-none-any.whl importlib-metadata @ file:///data/zkci8im/.cache/pypoetry/artifacts/b6/26/b1/de9225713e00b48ca565477825c09a513cf6e597d5b32a6f11a74f1eb8/importlib_metadata-4.4.0-py3-none-any.whl inflection @ file:///data/zkci8im/.cache/pypoetry/artifacts/98/bc/19/dd3ca5a6cc7c1d2ebc3fdeb01ae68dc6ec59c3a27f27d6f8ba2ed7560e/inflection-0.5.1-py2.py3-none-any.whl isodate @ file:///data/zkci8im/.cache/pypoetry/artifacts/7f/4c/a9/6ee9a5be06dbc032cddbdd42142dc7828a67c6d959f9cc7eda6f6cec3a/isodate-0.6.1-py2.py3-none-any.whl itypes @ file:///data/zkci8im/.cache/pypoetry/artifacts/27/5d/31/a7c551c3439cd7711a903bf67c5313d27a79bb908adcb734413f6ff3b8/itypes-1.2.0-py2.py3-none-any.whl Jinja2 @ file:///data/zkci8im/.cache/pypoetry/artifacts/50/1b/94/55cae878ce19751af0ec5f40765ef51e5af01c903cad84e0cdeaa3d7f0/Jinja2-2.11.3-py2.py3-none-any.whl jmespath @ file:///data/zkci8im/.cache/pypoetry/artifacts/65/c5/84/a16f584a42b02584726b43531ed19ca400f538f299a302ba4ad88b4cb3/jmespath-1.0.0-py3-none-any.whl jsonschema @ file:///data/zkci8im/.cache/pypoetry/artifacts/d7/ea/c1/138bfd39b9e843940605bd7170380281f7e0df8fb2a47fa5d70974fe90/jsonschema-3.2.0-py2.py3-none-any.whl kiwisolver @ file:///data/zkci8im/.cache/pypoetry/artifacts/90/ec/b4/941f119eada2e8221280a7b23e75a5d3e24ca089327c4cdca627412e85/kiwisolver-1.4.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl kombu @ file:///data/zkci8im/.cache/pypoetry/artifacts/b0/16/df/66aefa2d180ceace98dd7c2b0b1014c4380669e7382214e953aeed23ab/kombu-5.2.4-py3-none-any.whl lxml @ file:///data/zkci8im/.cache/pypoetry/artifacts/35/b0/67/93757e946c5baba990e7c2c1e16f618c9073f09d6ae033d506db95ce5a/lxml-4.8.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl Markdown @ file:///data/zkci8im/.cache/pypoetry/artifacts/34/32/83/707e90466826044af311cf500aca2bc915a5bbecd661201fbce6a7cf77/Markdown-3.3.7-py3-none-any.whl MarkupSafe @ file:///data/zkci8im/.cache/pypoetry/artifacts/18/2e/dc/4e3a84074e8f7e62f0badbe8fd7e1c027f256893e82d756e7153c758d1/MarkupSafe-2.0.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl matplotlib @ file:///data/zkci8im/.cache/pypoetry/artifacts/0f/f5/b5/8d4616bd0e92bad21257d238a1cb30c3a643277a1eb7887e274b140ccf/matplotlib-3.5.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl mysqlclient @ file:///data/zkci8im/.cache/pypoetry/artifacts/3e/49/38/62c5c2ca4417ce1aab974d62c8ed023b4731edb2ec0cf30e8700a817d2/mysqlclient-2.1.0.tar.gz nautobot @ file:///data/zkci8im/.cache/pypoetry/artifacts/a6/02/d4/096e1c7aade77e4b2ec96f51bcafd5d384ee2a705847dde9423a24bf8e/nautobot-1.2.11-py3-none-any.whl nautobot-anap-circuits @ file:///data/zkci8im/.cache/pypoetry/artifacts/63/b5/0d/e8252d47044b6425c000fe541a92a2562ecd22d9a0f24a790fb8b934a4/nautobot_anap_circuits-0.3.15-py3-none-any.whl nautobot-anap-circuits-sync @ file:///data/zkci8im/.cache/pypoetry/artifacts/4c/c5/58/a610a0576329c19eedcad609c2c4ffd9461819b1e73b80a982eb0ff53c/nautobot_anap_circuits_sync-0.3.6-py3-none-any.whl nautobot-anap-symphony @ file:///data/zkci8im/.cache/pypoetry/artifacts/aa/39/86/f78b31665c81bdddf246d3272a5907d4996a6c861c80f92e9f1dc3455a/nautobot_anap_symphony-0.6.44-py3-none-any.whl nautobot-bofa-ipam @ file:///data/zkci8im/.cache/pypoetry/artifacts/81/a8/a2/96f2ac7d2200b988b7a731f8e45efef6c3263fb7b823175e9d0e578e29/nautobot_bofa_ipam-1.4.3-py3-none-any.whl nautobot-bpa @ file:///data/zkci8im/.cache/pypoetry/artifacts/22/d0/80/7de73e8e798b5c9f6a99223f59db1ec60dbaf6da104c56769a014d3e20/nautobot_bpa-2.1.0-py3-none-any.whl nautobot-cert-letters @ file:///data/zkci8im/.cache/pypoetry/artifacts/ff/a5/0a/2e65e3eebb1f67727efffa57e9616d043e2dd7501d2e432b10fa8ae471/nautobot_cert_letters-0.1.17-py3-none-any.whl nautobot-clearpass @ file:///data/zkci8im/.cache/pypoetry/artifacts/02/d7/8b/84e2b6ff52cf8a23bee6f935d8cdc881668f43c25afa704d90735894e1/nautobot_clearpass-1.0.9-py3-none-any.whl nautobot-cmdb @ file:///data/zkci8im/.cache/pypoetry/artifacts/d4/90/8a/4e5bbf6bf6b32de2d33e3ba2330ea72cf19e34ee6abaf2fc30a8e4a81d/nautobot_cmdb-1.13.5-py3-none-any.whl nautobot-common @ file:///data/zkci8im/.cache/pypoetry/artifacts/b5/de/92/7de8b566158ff1df03b33c41f8c87e84eabb8c27fe09d0a94e47e392f3/nautobot_common-0.3.5-py3-none-any.whl nautobot-cyberark @ file:///data/zkci8im/.cache/pypoetry/artifacts/48/e4/68/148f4623324c647d1733b23fa41b93a906ca17ad4f681d177528773617/nautobot_cyberark-2.2.0-py3-none-any.whl nautobot-data-classification @ file:///data/zkci8im/.cache/pypoetry/artifacts/25/90/36/a34aeabb27539900ee5341cd73aae55f85ee6ac6cb4e43355645965ec2/nautobot_data_classification-1.0.1-py3-none-any.whl nautobot-data-ownership @ file:///data/zkci8im/.cache/pypoetry/artifacts/52/37/8f/72de92d33a4ff9ac987f52427afa3d7ae7ed222ac34b06885d0c2361c0/nautobot_data_ownership-0.2.0-py3-none-any.whl nautobot-device-lifecycle-mgmt @ file:///data/zkci8im/.cache/pypoetry/artifacts/d2/c8/5f/cda93c10d2f60155cc1a5fb3e359d291ef2ebf69903032b3524d7cbbbc/nautobot_device_lifecycle_mgmt-0.4.1-py3-none-any.whl nautobot-eps @ file:///data/zkci8im/.cache/pypoetry/artifacts/8d/86/03/b34e76b9b4c08aca090def035abbb5ec5d56fc30a710d3934c6eb1c2b5/nautobot_eps-0.1.0-py3-none-any.whl nautobot-network-classification @ file:///data/zkci8im/.cache/pypoetry/artifacts/c8/a6/2a/a00e72efb95bac27c3903d1f04ac888b19383e16c312f447952e67c134/nautobot_network_classification-1.1.0-py3-none-any.whl nautobot-nidm-core @ file:///data/zkci8im/.cache/pypoetry/artifacts/70/f8/e1/e03db47852cad4fd24b84c92d6dda32a5d3e68448a66cd23c0eadc5c7d/nautobot_nidm_core-1.2.1-py3-none-any.whl nautobot-rdr @ file:///data/zkci8im/.cache/pypoetry/artifacts/b9/d0/bb/db1717b5380cc3bb98c70bde3190dd278194e350fc7e1b3ebcbb68492b/nautobot_rdr-1.0.1-py3-none-any.whl nautobot-sevone-onboarding @ file:///data/zkci8im/.cache/pypoetry/artifacts/5a/b0/ee/48d5c3db983b7c208d26b32de54965f275a0408f507dd87294408b4b97/nautobot_sevone_onboarding-0.0.1-py3-none-any.whl nautobot-ssot @ file:///data/zkci8im/.cache/pypoetry/artifacts/15/19/8a/6aa9f5ad8232fa1c9e196fed6bd113b0d4f7cab6fd67d4a4c50f6a0c94/nautobot_ssot-1.1.2-py3-none-any.whl nautobot-standards @ file:///data/zkci8im/.cache/pypoetry/artifacts/e3/23/18/61dcab40bb5e3561a0cb412be3917595c317131a9211f19c575cf4ed24/nautobot_standards-1.8.5-py3-none-any.whl netaddr @ file:///data/zkci8im/.cache/pypoetry/artifacts/f9/16/e8/d9224c3975d478de3009d1d47712cf111b0a4b5d854dc9e9fabc65dbde/netaddr-0.8.0-py2.py3-none-any.whl netutils @ file:///data/zkci8im/.cache/pypoetry/artifacts/57/db/16/4111ea796f3afb67d33e86f63f35aba5a21019cda3f95e903ba63e8762/netutils-1.0.0-py3-none-any.whl numpy @ file:///data/zkci8im/.cache/pypoetry/artifacts/de/d5/28/b0cfd2d4c3e9a1c8ae4a78490049d8a3412d697a7f0a5b34c44f4d1d2b/numpy-1.22.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl oauthlib @ file:///data/zkci8im/.cache/pypoetry/artifacts/07/0e/03/4277e4ccf4acbe4361be783f58dd746cb95b78649a558edf1307a5a1f6/oauthlib-3.2.0-py3-none-any.whl openpyxl @ file:///data/zkci8im/.cache/pypoetry/artifacts/e8/b7/32/e56fe150a5abcd4739833b3a0a1548d230679b29074ef42402272629c9/openpyxl-3.0.10-py2.py3-none-any.whl packaging @ file:///data/zkci8im/.cache/pypoetry/artifacts/3b/2b/d0/229eacac699b7990b07c4a20d0f449f10465f21747f61e1b16ca0788b6/packaging-21.3-py3-none-any.whl Pillow @ file:///data/zkci8im/.cache/pypoetry/artifacts/47/f1/78/5bbff537c3d88fb8237793032585a247723b14e00240af6cb68de37365/Pillow-9.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl platformdirs @ file:///data/zkci8im/.cache/pypoetry/artifacts/45/0a/73/b60f9509a6bc1223523fb621fa706e105b8b75953ba5f6b20671b297ce/platformdirs-2.5.2-py3-none-any.whl prometheus-client @ file:///data/zkci8im/.cache/pypoetry/artifacts/29/46/d1/ae1047fbfd3702e1bcdc496b5ac6058f6522256abda4f7da8893f32891/prometheus_client-0.14.1-py3-none-any.whl promise @ file:///data/zkci8im/.cache/pypoetry/artifacts/62/2a/40/f263aa397064cb03a099f84abc98a7f75bdc1a34962f4eb0012ea78a53/promise-2.3.tar.gz prompt-toolkit @ file:///data/zkci8im/.cache/pypoetry/artifacts/a6/f6/75/fb1c4bae58f6c4622619dc9dbb1733b76f7ff3c0578d91c15ae91feae7/prompt_toolkit-3.0.29-py3-none-any.whl psycopg2-binary @ file:///data/zkci8im/.cache/pypoetry/artifacts/96/8d/16/3d21fd6c74323042ba23e37cbdb4d0d2a2bcfaa8e50d5e394378ca4a38/psycopg2_binary-2.8.6-cp38-cp38-manylinux1_x86_64.whl pyasn1 @ file:///data/zkci8im/.cache/pypoetry/artifacts/44/6a/a9/c21cf90c1e0756f209e70aa7f38486615fed1dd0d0aea32e06f20ddb4c/pyasn1-0.4.8-py2.py3-none-any.whl pycountry @ file:///data/zkci8im/.cache/pypoetry/artifacts/42/ef/f9/4f6319d77b9ae31577947642412faf82f7bfe8bc6a9bd52ea490088cba/pycountry-20.7.3.tar.gz pycparser @ file:///data/zkci8im/.cache/pypoetry/artifacts/33/0c/a0/326b0a2cf7e136e67ea4dbdbcfe7147a7db9c8eac3f45eaf5120b95303/pycparser-2.21-py2.py3-none-any.whl pycryptodome @ file:///data/zkci8im/.cache/pypoetry/artifacts/5a/a7/db/1f6483e9468f56bf8129100ace0c108433c24ebb8302620d1db5caa52c/pycryptodome-3.10.4-cp35-abi3-manylinux2010_x86_64.whl pydantic @ file:///data/zkci8im/.cache/pypoetry/artifacts/6b/fd/fb/a8b33721ecebb58ba4b2fde8b2e799fb03ef090283041c8ff8136cb96c/pydantic-1.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl PyJWT @ file:///data/zkci8im/.cache/pypoetry/artifacts/4e/58/c4/35a0ce0b263912f8cb70102a00ea6ad3e4df070fc1999acd502193b950/PyJWT-2.4.0-py3-none-any.whl pymssql @ file:///data/zkci8im/.cache/pypoetry/artifacts/0c/f4/f0/5a68c58d9ad2ca40ecbedd7675d11f7ecc10d2efebb0f7ad38b2d52c4f/pymssql-2.2.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl pyparsing @ file:///data/zkci8im/.cache/pypoetry/artifacts/fc/60/07/1b440c7539c1662fbae8276892b463f7ff5daa3dbde20a91709721264d/pyparsing-3.0.9-py3-none-any.whl pyrsistent @ file:///data/zkci8im/.cache/pypoetry/artifacts/b3/07/67/b2f8ed0cbef09958ed5340a50cfb1649662be07ba14110e999b6cfda0c/pyrsistent-0.18.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl python-crontab @ file:///data/zkci8im/.cache/pypoetry/artifacts/80/4f/88/386611efe39c69c86592f1eb8d3c44f6dc730192df536ca8351a5e7721/python-crontab-2.6.0.tar.gz python-dateutil @ file:///data/zkci8im/.cache/pypoetry/artifacts/a8/e2/44/bdcaa1ff9495f3ad9e18dd6e63bb9e887647aba953d9ad9fc05a86fae7/python_dateutil-2.8.2-py2.py3-none-any.whl python-jose @ file:///data/zkci8im/.cache/pypoetry/artifacts/f6/2a/b8/b6494aa866f7cc75ab568808374a5b083ffdc16206e8f3b289ee18c2b0/python_jose-3.3.0-py2.py3-none-any.whl python3-openid @ file:///data/zkci8im/.cache/pypoetry/artifacts/0c/12/9f/ea7f74cf5238a45aadf81b3b7eadcfefa8951666c12b69fec11aa464bd/python3_openid-3.2.0-py3-none-any.whl python3-saml @ file:///data/zkci8im/.cache/pypoetry/artifacts/ad/dc/e4/ebc0a921e5a1728077b6deb835157f90121645d5052b614e3982f3d670/python3_saml-1.12.0-py3-none-any.whl pytz @ file:///data/zkci8im/.cache/pypoetry/artifacts/70/34/d6/c45c9e208f555808c5cd019789091f10dd40a2f3022649a3f20800d62a/pytz-2022.1-py2.py3-none-any.whl pyuwsgi @ file:///data/zkci8im/.cache/pypoetry/artifacts/94/5a/2a/3ebdbc0013fd6046b29a1b273279cdbe5092fbf951c00a490d69f83c42/pyuwsgi-2.0.20-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl PyYAML @ file:///data/zkci8im/.cache/pypoetry/artifacts/f7/3b/da/8e567f27818a41bf1549e53b5cd23647b1d7e2e647e0822dabb01a63bb/PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl redis @ file:///data/zkci8im/.cache/pypoetry/artifacts/af/f2/d3/bd5e4c2a389a010eee4acc4f08add48552e1be122bc7c768db5b8375ab/redis-4.3.1-py3-none-any.whl requests @ file:///data/zkci8im/.cache/pypoetry/artifacts/02/17/db/7a356bc4171d71bd271a00d6df46c824f4370f2ef39d88b74e6083bc03/requests-2.27.1-py2.py3-none-any.whl requests-file @ file:///data/zkci8im/.cache/pypoetry/artifacts/82/c0/ba/0889eaaa9e09451feea27723a96f52056ef402da06ae9e2b0b785d1879/requests_file-1.5.1-py2.py3-none-any.whl requests-oauthlib @ file:///data/zkci8im/.cache/pypoetry/artifacts/b7/88/21/a294759fa4bda84604868791a9fb7ae94d995a0a5a3e446867437b62ef/requests_oauthlib-1.3.1-py2.py3-none-any.whl requests-toolbelt @ file:///data/zkci8im/.cache/pypoetry/artifacts/70/d0/4e/12d5b1fabbdf8cc813b65a85b4a44885ddccf80ddbe42cdab87df3c9c4/requests_toolbelt-0.9.1-py2.py3-none-any.whl resolvelib @ file:///data/zkci8im/.cache/pypoetry/artifacts/95/f0/cd/33d25064e3e20dd4c1ad2fa315e04cff25fd0c6d6ad79b9068a3c33af2/resolvelib-0.5.5-py2.py3-none-any.whl robot-bpa-sdk @ file:///data/zkci8im/.cache/pypoetry/artifacts/99/87/39/0a25e697d3f52ae3c0b2520d8249d5abb5573d30222ec78f51518d79e4/robot_bpa_sdk-2021.7.15-py3-none-any.whl rq @ file:///data/zkci8im/.cache/pypoetry/artifacts/d6/8c/a8/a5173f44dfe1a74a0593828ff56e0245df984f348daadd3560c5c1024c/rq-1.10.1-py2.py3-none-any.whl rsa @ file:///data/zkci8im/.cache/pypoetry/artifacts/70/83/7d/515ad00d6559117fd24ee2dd69dbfdaa556c8986e6e35879cabad091d2/rsa-4.8-py3-none-any.whl ruamel.yaml @ file:///data/zkci8im/.cache/pypoetry/artifacts/d6/a3/d5/50b355606c7e8cbf1d3e88a8da1b29c1957319ceb41a587c45aca1ee5c/ruamel.yaml-0.17.21-py3-none-any.whl ruamel.yaml.clib @ file:///data/zkci8im/.cache/pypoetry/artifacts/da/9f/c3/0b970a7488666902b22784c2305a321e4fc5176d96eee119d8ea9c053f/ruamel.yaml.clib-0.2.6-cp38-cp38-manylinux1_x86_64.whl Rx @ file:///data/zkci8im/.cache/pypoetry/artifacts/34/ec/6f/bc1186029e7ad679cf82dde5131f5790bc62d83b3ba2aa0500222de304/Rx-1.6.1-py2.py3-none-any.whl s3transfer @ file:///data/zkci8im/.cache/pypoetry/artifacts/20/db/c0/28a2baf88f972584bd05f1495d1a7a4c4e0a318b9831c56534a13df0b5/s3transfer-0.5.2-py3-none-any.whl setuptools-scm @ file:///data/zkci8im/.cache/pypoetry/artifacts/c1/0e/d6/bf340b99a2ade5d92443240719f8a443e7bd88b591bcef74956a16899a/setuptools_scm-6.4.2-py3-none-any.whl singledispatch @ file:///data/zkci8im/.cache/pypoetry/artifacts/49/64/ca/e60267abe7b859719ae4e08a0422e8f56ad4a98b8767fb76a8272deb99/singledispatch-3.7.0-py2.py3-none-any.whl six @ file:///data/zkci8im/.cache/pypoetry/artifacts/f7/55/4d/44b471ed2a50fcce09e4bda1a06c8a7ec642c53b5166ce5c0097275802/six-1.16.0-py2.py3-none-any.whl smmap @ file:///data/zkci8im/.cache/pypoetry/artifacts/57/bf/ea/1042000312501eb13606330aaa12de6e8c3a57ed7d33f1d659eef2e720/smmap-5.0.0-py3-none-any.whl social-auth-app-django @ file:///data/zkci8im/.cache/pypoetry/artifacts/66/15/43/2212ab82e7ada52755f3131966a17d91a60bdd662aab49f5b1beb5b559/social_auth_app_django-4.0.0-py3-none-any.whl social-auth-core @ file:///data/zkci8im/.cache/pypoetry/artifacts/c7/fe/5b/dd39fab35925f9c3c27852572b608f78b47335bd33238cd16a47a12110/social_auth_core-4.1.0-py3-none-any.whl sqlparse @ file:///data/zkci8im/.cache/pypoetry/artifacts/35/2a/c7/b94bd2500225f7c258ebddb92714290de3566d4a03a5f849b4c2b47c7c/sqlparse-0.4.2-py3-none-any.whl structlog @ file:///data/zkci8im/.cache/pypoetry/artifacts/1d/1c/fd/7df0160dbd1830fdb46c740b0a81ea252fb6a73b0677353ebaa3f0d674/structlog-20.2.0-py2.py3-none-any.whl svgwrite @ file:///data/zkci8im/.cache/pypoetry/artifacts/2e/b0/bb/551a83c1df6d04ead4a3e7370f68d0b57e02fad656042cad5df2a310ed/svgwrite-1.4.2-py3-none-any.whl swagger-spec-validator @ file:///data/zkci8im/.cache/pypoetry/artifacts/71/71/63/7a52697831713b439971a032fb83e31dd1df44aa638e90be82a5ac3d5d/swagger_spec_validator-2.7.4-py2.py3-none-any.whl text-unidecode @ file:///data/zkci8im/.cache/pypoetry/artifacts/38/cc/09/814f7784e36c8140d07513c0e1709fd9a3807cdd27e6be6efbbdb628fa/text_unidecode-1.3-py2.py3-none-any.whl tomli @ file:///data/zkci8im/.cache/pypoetry/artifacts/aa/1c/82/1fc4d637d7cad4a6d39e21ab22bd7957ec3228f36c499c4b2d6ff33b4e/tomli-2.0.1-py3-none-any.whl types-PyYAML @ file:///data/zkci8im/.cache/pypoetry/artifacts/85/94/c2/81254c1bdbfd106a966adca8c00f5cb3a20cdcee1cd50975f3fa345019/types_PyYAML-6.0.7-py3-none-any.whl typing_extensions @ file:///data/zkci8im/.cache/pypoetry/artifacts/0d/6f/04/b8bd0c34c092d479e1f9ee7ac241d6f264cc01db2832e08fd4a8c1fb3e/typing_extensions-4.2.0-py3-none-any.whl uritemplate @ file:///data/zkci8im/.cache/pypoetry/artifacts/76/08/ec/048fc3fa6c2480f858b637ddfe52729642b79c01e8f47ce663d74df8d7/uritemplate-4.1.1-py2.py3-none-any.whl urllib3 @ file:///data/zkci8im/.cache/pypoetry/artifacts/69/80/ba/5408bb5fec831682384684c11eab54da6b8dd997fb85871eba696966b2/urllib3-1.26.9-py2.py3-none-any.whl vine @ file:///data/zkci8im/.cache/pypoetry/artifacts/d0/af/cd/40d4b255297e753565b27ec2cecb352311be0f65fb821c71a02d9ed109/vine-5.0.0-py2.py3-none-any.whl wcwidth @ file:///data/zkci8im/.cache/pypoetry/artifacts/16/12/ba/6211db7a12b3c00978b51a85563bdb0d710e9397027fa8f9c8025daa76/wcwidth-0.2.5-py2.py3-none-any.whl wrapt @ file:///data/zkci8im/.cache/pypoetry/artifacts/9b/83/b3/ee7bc92d3d7376c36e018277e1cd90f052e70ce5d76bf5c672ec271d34/wrapt-1.14.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl xmlsec @ file:///data/zkci8im/.cache/pypoetry/artifacts/d4/76/2c/871220815f373e94acaa810926b0e857a351d3b24e87a258dc72a2e0c1/xmlsec-1.3.12.tar.gz zeep @ file:///data/zkci8im/.cache/pypoetry/artifacts/3a/55/89/f82f1d16b4e53c6a318a21628bb26e6921baf442bd1cf65eeeb3fe6920/zeep-4.1.0-py2.py3-none-any.whl zipp @ file:///data/zkci8im/.cache/pypoetry/artifacts/6d/cc/4e/46283fca4ce6eff04a6e6bfe21e67d2c54e2e6bc27ba1d60c7be3bc0da/zipp-3.8.0-py3-none-any.whl ```
Other Dependencies
Python: 3.8 celery: 5.2.7 django_celery_beat: 2.2.1
Minimally Reproducible Test Case
```python ```
Expected Behavior
A single task (both one-off, and periodic) - should execute ONCE AND ONLY ONCE .
Actual Behavior
When multiple tasks are started at roughly the same time, and one task completes while other task(s) still running - running task(s) get started again resulting in MULTIPLE TASK EXECUTIONS.
Package Versions
celery: 5.2.7 django_celery_beat: 2.2.1
Issue Description
Celery beat Scheduler (re)starts same task while task is running, resulting in multiple executions of the same task.
Details
Consider celery.beat.Scheduler.tick() implementation (code below is from v5.2.7):
In this implementation, _self.isdue() is executed and then (if _isdue == True is returned) - _self.applyentry() sends event to start task execution to the celery worker (assuming 'verify' is event):
is_due, next_time_to_run = self.is_due(entry)Now, consider case where given task is already executing... Since task is executing, it is already "due" - therefore, _self.isdue() returns True (_isdue == True). Scheduler.tick() happily proceeds and executes _self.applyentry() - which essentially starts same task's execution (by the worker) again. This continues on every Scheduler.tick() execution - until _self.isdue() returns False - resulting in same task (single scheduled instance of the task) executed multiple times.
Problem summary
Celery beat Scheduler does not "recognize" that given task is already executing (running) - thus, sends event to start same task over and over again.
Additional details
I work on a project where multiple one-off tasks can be executed at any given moment. We have noticed (and proved that to be true) that same task is often executed multiple times (while some - only once). Having same task executed more than once - "kills" this project; it causes many negative side effects that cannot be handled or predicted (at least, not all of them).
We spent a lot of time troubleshooting this issue - which included debugging (use of breakpoints and stepping through) underlying Celery implementation. This troubleshooting led us to understanding the root cause of the issue: Celery does not recognize that task is "running" and solely relies on is_due() response to send event to start task execution.
We have also proved that same case applies not only to one-off tasks, but also to periodic scheduled tasks: periodic task that should run let's say once a day, or once an hour - can get executed multiple times during same day (or same hour) while initial task is running.
Conclusion
This is a HUGE issue (at least for us). Single task (single "instance" of a task) - must be executed ONCE AND ONLY ONCE. Celery scheduler's inability to recognize that task is already executing and prevent subsequent executions - is a VERY SUBSTANTIAL BUG, causing lots of unpredictable side effects.
To resolve this issue, some sort of "task is running" recognition must be implemented (either within Scheduler.tick() or (more likely) within _isdue() method - that recognizes that task is already running and prevents subsequent task executions (while it is actually "is due").
NOTICE: Django celery beat is used, so particular is_due() method that gets executed - is: _django_celery_beat.schedulers.ModelEntry.isdue().
Steps Taken
Following steps have been taken while trying to eliminate this issue:
Steps to reproduce