generateorders: "order.models.MultipleObjectsReturned: get() returned more than one Order -- it returned 2!"

[kousu]

EDIT: sometimes it returns 3 as well. It could be any number. 1 is the most likely, 2 is the next most likely.

On our production server, at on Fri, Mar 2, 2018 at 9:00 PM, running the cronjob

/usr/bin/docker exec souschef_web_1 python src/manage.py generateorders $(date +%Y-%m-%d) --days 100

gave

Traceback (most recent call last):
  File "src/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/code/src/order/management/commands/generateorders.py", line 40, in handle
    delivery_date, clients
  File "/code/src/order/models.py", line 170, in auto_create_orders
    delivery_date=delivery_date)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 384, in get
    (self.model._meta.object_name, num)
order.models.MultipleObjectsReturned: get() returned more than one Order -- it returned 2!

and the next night, Sat, Mar 3, 2018 at 9:00 PM

Traceback (most recent call last):
  File "src/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/code/src/order/management/commands/generateorders.py", line 40, in handle
    delivery_date, clients
  File "/code/src/order/models.py", line 170, in auto_create_orders
    delivery_date=delivery_date)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 384, in get
    (self.model._meta.object_name, num)
order.models.MultipleObjectsReturned: get() returned more than one Order -- it returned 2!

We only just thought to make the cronjobs log what they're doing to email, so I don't know how long this has been going on.

The output was back to normal tonight (Sun, Mar 4, 2018 at 9:00 PM):

0 orders created on 2018-03-04: to be delivered on 2018-03-04.
81 orders created on 2018-03-04: to be delivered on 2018-03-05.
76 orders created on 2018-03-04: to be delivered on 2018-03-06.
77 orders created on 2018-03-04: to be delivered on 2018-03-07.
0 orders created on 2018-03-04: to be delivered on 2018-03-08.
81 orders created on 2018-03-04: to be delivered on 2018-03-09.
63 orders created on 2018-03-04: to be delivered on 2018-03-10.
0 orders created on 2018-03-04: to be delivered on 2018-03-11.
81 orders created on 2018-03-04: to be delivered on 2018-03-12.
76 orders created on 2018-03-04: to be delivered on 2018-03-13.
77 orders created on 2018-03-04: to be delivered on 2018-03-14.
0 orders created on 2018-03-04: to be delivered on 2018-03-15.
81 orders created on 2018-03-04: to be delivered on 2018-03-16.
63 orders created on 2018-03-04: to be delivered on 2018-03-17.
0 orders created on 2018-03-04: to be delivered on 2018-03-18.
81 orders created on 2018-03-04: to be delivered on 2018-03-19.
76 orders created on 2018-03-04: to be delivered on 2018-03-20.
77 orders created on 2018-03-04: to be delivered on 2018-03-21.
0 orders created on 2018-03-04: to be delivered on 2018-03-22.
81 orders created on 2018-03-04: to be delivered on 2018-03-23.
63 orders created on 2018-03-04: to be delivered on 2018-03-24.
0 orders created on 2018-03-04: to be delivered on 2018-03-25.
81 orders created on 2018-03-04: to be delivered on 2018-03-26.
76 orders created on 2018-03-04: to be delivered on 2018-03-27.
77 orders created on 2018-03-04: to be delivered on 2018-03-28.
0 orders created on 2018-03-04: to be delivered on 2018-03-29.
81 orders created on 2018-03-04: to be delivered on 2018-03-30.
63 orders created on 2018-03-04: to be delivered on 2018-03-31.
0 orders created on 2018-03-04: to be delivered on 2018-04-01.
81 orders created on 2018-03-04: to be delivered on 2018-04-02.
76 orders created on 2018-03-04: to be delivered on 2018-04-03.
77 orders created on 2018-03-04: to be delivered on 2018-04-04.
0 orders created on 2018-03-04: to be delivered on 2018-04-05.
81 orders created on 2018-03-04: to be delivered on 2018-04-06.
63 orders created on 2018-03-04: to be delivered on 2018-04-07.
0 orders created on 2018-03-04: to be delivered on 2018-04-08.
81 orders created on 2018-03-04: to be delivered on 2018-04-09.
76 orders created on 2018-03-04: to be delivered on 2018-04-10.
77 orders created on 2018-03-04: to be delivered on 2018-04-11.
0 orders created on 2018-03-04: to be delivered on 2018-04-12.
81 orders created on 2018-03-04: to be delivered on 2018-04-13.
63 orders created on 2018-03-04: to be delivered on 2018-04-14.
0 orders created on 2018-03-04: to be delivered on 2018-04-15.
81 orders created on 2018-03-04: to be delivered on 2018-04-16.
76 orders created on 2018-03-04: to be delivered on 2018-04-17.
77 orders created on 2018-03-04: to be delivered on 2018-04-18.
0 orders created on 2018-03-04: to be delivered on 2018-04-19.
81 orders created on 2018-03-04: to be delivered on 2018-04-20.
63 orders created on 2018-03-04: to be delivered on 2018-04-21.
0 orders created on 2018-03-04: to be delivered on 2018-04-22.
81 orders created on 2018-03-04: to be delivered on 2018-04-23.
76 orders created on 2018-03-04: to be delivered on 2018-04-24.
77 orders created on 2018-03-04: to be delivered on 2018-04-25.
0 orders created on 2018-03-04: to be delivered on 2018-04-26.
81 orders created on 2018-03-04: to be delivered on 2018-04-27.
63 orders created on 2018-03-04: to be delivered on 2018-04-28.
0 orders created on 2018-03-04: to be delivered on 2018-04-29.
81 orders created on 2018-03-04: to be delivered on 2018-04-30.
76 orders created on 2018-03-04: to be delivered on 2018-05-01.
77 orders created on 2018-03-04: to be delivered on 2018-05-02.
0 orders created on 2018-03-04: to be delivered on 2018-05-03.
81 orders created on 2018-03-04: to be delivered on 2018-05-04.
63 orders created on 2018-03-04: to be delivered on 2018-05-05.
0 orders created on 2018-03-04: to be delivered on 2018-05-06.
81 orders created on 2018-03-04: to be delivered on 2018-05-07.
76 orders created on 2018-03-04: to be delivered on 2018-05-08.
76 orders created on 2018-03-04: to be delivered on 2018-05-09.
0 orders created on 2018-03-04: to be delivered on 2018-05-10.
81 orders created on 2018-03-04: to be delivered on 2018-05-11.
62 orders created on 2018-03-04: to be delivered on 2018-05-12.
0 orders created on 2018-03-04: to be delivered on 2018-05-13.
81 orders created on 2018-03-04: to be delivered on 2018-05-14.
75 orders created on 2018-03-04: to be delivered on 2018-05-15.
76 orders created on 2018-03-04: to be delivered on 2018-05-16.
0 orders created on 2018-03-04: to be delivered on 2018-05-17.
81 orders created on 2018-03-04: to be delivered on 2018-05-18.
62 orders created on 2018-03-04: to be delivered on 2018-05-19.
0 orders created on 2018-03-04: to be delivered on 2018-05-20.
81 orders created on 2018-03-04: to be delivered on 2018-05-21.
75 orders created on 2018-03-04: to be delivered on 2018-05-22.
76 orders created on 2018-03-04: to be delivered on 2018-05-23.
0 orders created on 2018-03-04: to be delivered on 2018-05-24.
81 orders created on 2018-03-04: to be delivered on 2018-05-25.
62 orders created on 2018-03-04: to be delivered on 2018-05-26.
0 orders created on 2018-03-04: to be delivered on 2018-05-27.
81 orders created on 2018-03-04: to be delivered on 2018-05-28.
75 orders created on 2018-03-04: to be delivered on 2018-05-29.
76 orders created on 2018-03-04: to be delivered on 2018-05-30.
0 orders created on 2018-03-04: to be delivered on 2018-05-31.
81 orders created on 2018-03-04: to be delivered on 2018-06-01.
62 orders created on 2018-03-04: to be delivered on 2018-06-02.
0 orders created on 2018-03-04: to be delivered on 2018-06-03.
81 orders created on 2018-03-04: to be delivered on 2018-06-04.
75 orders created on 2018-03-04: to be delivered on 2018-06-05.
76 orders created on 2018-03-04: to be delivered on 2018-06-06.
0 orders created on 2018-03-04: to be delivered on 2018-06-07.
81 orders created on 2018-03-04: to be delivered on 2018-06-08.
62 orders created on 2018-03-04: to be delivered on 2018-06-09.
0 orders created on 2018-03-04: to be delivered on 2018-06-10.
81 orders created on 2018-03-04: to be delivered on 2018-06-11.

[kousu]

I'm able to reproduce it with manually running

$ /usr/bin/docker exec souschef_web_1 python src/manage.py generateorders 2018-03-03 --days 100

Traceback (most recent call last):
  File "src/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/code/src/order/management/commands/generateorders.py", line 40, in handle
    delivery_date, clients
  File "/code/src/order/models.py", line 170, in auto_create_orders
    delivery_date=delivery_date)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 384, in get
    (self.model._meta.object_name, num)
order.models.MultipleObjectsReturned: get() returned more than one Order -- it returned 2!

[kousu]

Going further back in time, sometimes one day will get generated and then the crash will happen:

$ /usr/bin/docker exec souschef_web_1 python src/manage.py generateorders 2018-03-02 --days 100

Traceback (most recent call last):
  File "src/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/code/src/order/management/commands/generateorders.py", line 40, in handle
    delivery_date, clients
  File "/code/src/order/models.py", line 170, in auto_create_orders
    delivery_date=delivery_date)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 384, in get
    (self.model._meta.object_name, num)
order.models.MultipleObjectsReturned: get() returned more than one Order -- it returned 2!

$ /usr/bin/docker exec souschef_web_1 python src/manage.py generateorders 2018-03-01 --days 100

0 orders created on 2018-03-01: to be delivered on 2018-03-01.
Traceback (most recent call last):
  File "src/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/code/src/order/management/commands/generateorders.py", line 40, in handle
    delivery_date, clients
  File "/code/src/order/models.py", line 170, in auto_create_orders
    delivery_date=delivery_date)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 384, in get
    (self.model._meta.object_name, num)
order.models.MultipleObjectsReturned: get() returned more than one Order -- it returned 2!

$ /usr/bin/docker exec souschef_web_1 python src/manage.py generateorders 2018-02-28 --days 100

Traceback (most recent call last):
  File "src/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/code/src/order/management/commands/generateorders.py", line 40, in handle
    delivery_date, clients
  File "/code/src/order/models.py", line 170, in auto_create_orders
    delivery_date=delivery_date)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 384, in get
    (self.model._meta.object_name, num)
order.models.MultipleObjectsReturned: get() returned more than one Order -- it returned 2!

$ /usr/bin/docker exec souschef_web_1 python src/manage.py generateorders 2018-02-27 --days 100

75 orders created on 2018-02-27: to be delivered on 2018-02-27.
Traceback (most recent call last):
  File "src/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/code/src/order/management/commands/generateorders.py", line 40, in handle
    delivery_date, clients
  File "/code/src/order/models.py", line 170, in auto_create_orders
    delivery_date=delivery_date)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 384, in get
    (self.model._meta.object_name, num)
order.models.MultipleObjectsReturned: get() returned more than one Order -- it returned 2!

[kousu]

I've traced the code; what's happening is that in auto_create_orders there's

            try:
                order = Order.objects.get(client=client,
                                          delivery_date=delivery_date)
                created_orders.append(order)
                continue
            except Order.DoesNotExist:
                # If no order for this client/date, create it and attach items
                ...

like the exception says, that get() is choking on multiple results where it's only expecting one. I found a client ("xxxx") that triggers this and poked around the database. I found, for example:

MariaDB [feast]> select * from order_order where delivery_date = "2017-12-06" and client_id=xxxx;                  
+-------+---------------+---------------+--------+-----------+
| id    | creation_date | delivery_date | status | client_id |
+-------+---------------+---------------+--------+-----------+
| yyyyy | 2017-12-06    | 2017-12-06    | C      |      xxxx |
| yyyyy | 2017-12-06    | 2017-12-06    | C      |      xxxx |
| yyyyy | 2017-12-06    | 2017-12-06    | D      |      xxxx |
+-------+---------------+---------------+--------+-----------+

There's quite a few of these duplicate orders:

select count(*) from (select *, count(*) as c from order_order group by client_id, delivery_date) 
as x where x.c > 1;                                                                                                
+----------+
| count(*) |
+----------+
|      217 |
+----------+

Here they all our (censored for privacy):

root@juniorroulant:/var/www/sous-chef# cat a
id  creation_date   delivery_date   status  cliend_id   c
yyyyy   2017-12-25  2017-12-25  C   xxx 2
yyyyy   2017-12-30  2017-12-30  C   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-12-06  2017-12-06  C   xxx 2
yyyyy   2017-12-30  2017-12-30  C   xxx 2
yyyyy   2017-03-30  2017-05-10  C   xxx 2
yyyyy   2017-08-16  2017-10-31  C   xxx 2
yyyyy   2017-11-03  2017-11-03  C   xxx 2
yyyyy   2017-11-04  2017-11-04  C   xxx 2
yyyyy   2017-11-06  2017-11-06  C   xxx 2
yyyyy   2017-11-08  2017-11-08  C   xxx 2
yyyyy   2017-11-10  2017-11-10  C   xxx 2
yyyyy   2017-11-11  2017-11-11  C   xxx 2
yyyyy   2017-11-13  2017-11-13  C   xxx 2
yyyyy   2017-11-15  2017-11-15  C   xxx 3
yyyyy   2017-11-17  2017-11-17  C   xxx 2
yyyyy   2017-11-18  2017-11-18  C   xxx 3
yyyyy   2017-11-20  2017-11-20  C   xxx 2
yyyyy   2017-11-21  2017-11-21  C   xxx 2
yyyyy   2017-11-22  2017-11-22  C   xxx 2
yyyyy   2017-11-24  2017-11-24  C   xxx 2
yyyyy   2017-11-28  2017-11-28  C   xxx 2
yyyyy   2017-11-29  2017-11-29  C   xxx 2
yyyyy   2017-12-02  2017-12-02  C   xxx 2
yyyyy   2017-12-04  2017-12-04  C   xxx 2
yyyyy   2017-12-05  2017-12-05  C   xxx 3
yyyyy   2017-12-06  2017-12-06  C   xxx 2
yyyyy   2017-12-08  2017-12-08  C   xxx 2
yyyyy   2017-12-09  2017-12-09  C   xxx 2
yyyyy   2017-12-11  2017-12-11  C   xxx 2
yyyyy   2017-12-12  2017-12-12  C   xxx 2
yyyyy   2017-12-15  2017-12-15  C   xxx 2
yyyyy   2017-12-16  2017-12-16  C   xxx 2
yyyyy   2017-12-18  2017-12-18  C   xxx 2
yyyyy   2017-12-19  2017-12-19  C   xxx 2
yyyyy   2017-12-20  2017-12-20  C   xxx 2
yyyyy   2017-12-23  2017-12-23  C   xxx 2
yyyyy   2017-12-25  2017-12-25  C   xxx 2
yyyyy   2017-12-27  2017-12-27  C   xxx 2
yyyyy   2017-12-30  2017-12-30  C   xxx 2
yyyyy   2018-01-01  2018-01-01  C   xxx 3
yyyyy   2018-01-02  2018-01-02  C   xxx 2
yyyyy   2018-01-03  2018-01-03  C   xxx 2
yyyyy   2018-01-05  2018-01-05  C   xxx 2
yyyyy   2018-01-06  2018-01-06  C   xxx 2
yyyyy   2018-01-08  2018-01-08  C   xxx 2
yyyyy   2018-01-09  2018-01-09  C   xxx 2
yyyyy   2018-01-10  2018-01-10  C   xxx 2
yyyyy   2018-01-12  2018-01-12  C   xxx 2
yyyyy   2018-01-13  2018-01-13  C   xxx 2
yyyyy   2018-01-16  2018-01-16  C   xxx 2
yyyyy   2018-01-19  2018-01-19  C   xxx 2
yyyyy   2018-01-20  2018-01-20  C   xxx 2
yyyyy   2018-01-22  2018-01-22  C   xxx 2
yyyyy   2018-01-23  2018-01-23  C   xxx 2
yyyyy   2018-01-24  2018-01-24  C   xxx 2
yyyyy   2018-01-26  2018-01-26  C   xxx 2
yyyyy   2018-01-27  2018-01-27  C   xxx 2
yyyyy   2018-01-29  2018-01-29  C   xxx 2
yyyyy   2018-01-31  2018-01-31  C   xxx 2
yyyyy   2018-02-02  2018-02-02  C   xxx 2
yyyyy   2018-02-03  2018-02-03  C   xxx 2
yyyyy   2018-02-05  2018-02-05  C   xxx 2
yyyyy   2018-02-06  2018-02-06  C   xxx 2
yyyyy   2018-02-07  2018-02-07  C   xxx 2
yyyyy   2018-02-10  2018-02-10  C   xxx 2
yyyyy   2018-02-12  2018-02-12  C   xxx 2
yyyyy   2018-02-16  2018-02-16  C   xxx 2
yyyyy   2018-02-19  2018-02-19  C   xxx 2
yyyyy   2018-02-20  2018-02-20  C   xxx 2
yyyyy   2018-02-21  2018-02-21  C   xxx 2
yyyyy   2018-02-23  2018-02-23  C   xxx 2
yyyyy   2018-02-24  2018-02-24  C   xxx 2
yyyyy   2018-02-26  2018-02-26  C   xxx 2
yyyyy   2018-02-28  2018-02-28  C   xxx 3
yyyyy   2018-03-02  2018-03-02  C   xxx 2
yyyyy   2017-12-30  2017-12-30  C   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-11-24  2017-11-24  C   xxx 2
yyyyy   2017-11-27  2017-11-27  C   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-08-16  2017-10-31  C   xxx 2
yyyyy   2017-11-04  2017-11-04  C   xxx 2
yyyyy   2017-11-24  2017-11-24  C   xxx 2
yyyyy   2017-11-28  2017-11-28  C   xxx 2
yyyyy   2017-12-02  2017-12-02  C   xxx 2
yyyyy   2017-12-08  2017-12-08  C   xxx 2
yyyyy   2017-12-13  2017-12-13  C   xxx 2
yyyyy   2017-12-19  2017-12-19  C   xxx 2
yyyyy   2017-12-26  2017-12-26  C   xxx 2
yyyyy   2017-12-30  2017-12-30  C   xxx 2
yyyyy   2018-01-03  2018-01-03  C   xxx 2
yyyyy   2018-01-06  2018-01-06  C   xxx 2
yyyyy   2018-01-12  2018-01-12  D   xxx 2
yyyyy   2018-01-16  2018-01-16  C   xxx 2
yyyyy   2018-01-19  2018-01-19  C   xxx 2
yyyyy   2018-01-27  2018-01-27  C   xxx 2
yyyyy   2018-01-31  2018-01-31  C   xxx 2
yyyyy   2018-02-13  2018-02-13  C   xxx 2
yyyyy   2018-02-26  2018-02-26  C   xxx 2
yyyyy   2018-03-03  2018-03-03  C   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-12-05  2017-12-05  C   xxx 2
yyyyy   2018-01-29  2018-01-29  C   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-12-30  2017-12-30  C   xxx 2
yyyyy   2018-01-29  2018-01-29  C   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2018-01-23  2018-01-23  C   xxx 2
yyyyy   2018-01-29  2018-01-29  C   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-12-30  2017-12-30  C   xxx 2
yyyyy   2018-01-29  2018-01-29  C   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-11-28  2017-11-28  C   xxx 2
yyyyy   2017-12-15  2017-12-15  C   xxx 2
yyyyy   2018-01-29  2018-01-29  C   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-11-08  2017-11-08  C   xxx 2
yyyyy   2017-12-08  2017-12-08  C   xxx 2
yyyyy   2017-12-18  2017-12-18  C   xxx 2
yyyyy   2017-12-22  2017-12-22  C   xxx 2
yyyyy   2017-12-29  2017-12-29  C   xxx 2
yyyyy   2017-12-30  2017-12-30  C   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2018-01-29  2018-01-29  C   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2018-02-03  2018-02-03  C   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-11-14  2017-11-14  C   xxx 2
yyyyy   2017-11-15  2017-11-15  C   xxx 2
yyyyy   2017-11-17  2017-11-17  C   xxx 2
yyyyy   2018-02-26  2018-02-26  C   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2018-01-29  2018-01-29  C   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-12-22  2017-12-22  C   xxx 2
yyyyy   2018-01-02  2018-01-02  C   xxx 2
yyyyy   2018-01-13  2018-01-13  C   xxx 2
yyyyy   2018-01-19  2018-01-19  C   xxx 2
yyyyy   2018-01-20  2018-01-20  C   xxx 2
yyyyy   2018-01-26  2018-01-26  C   xxx 2
yyyyy   2017-01-24  2016-03-29  D   xxx 2
yyyyy   2017-12-30  2017-12-30  C   xxx 2
yyyyy   2018-03-03  2018-03-03  C   xxx 2
yyyyy   2017-12-11  2017-12-11  C   xxx 2
yyyyy   2017-12-12  2017-12-12  C   xxx 2
yyyyy   2017-12-20  2017-12-20  C   xxx 2
yyyyy   2017-11-06  2017-11-06  C   xxx 2
yyyyy   2018-01-29  2018-01-29  C   xxx 2
yyyyy   2017-11-04  2017-11-04  C   xxx 2
yyyyy   2018-01-29  2018-01-29  C   xxx 2
yyyyy   2017-12-08  2017-12-08  C   xxx 2
yyyyy   2018-01-29  2018-01-29  C   xxx 2
yyyyy   2017-12-16  2017-12-16  C   xxx 2
yyyyy   2017-12-06  2017-12-06  C   xxx 2
yyyyy   2017-12-30  2017-12-30  C   xxx 2
yyyyy   2018-02-03  2018-02-03  C   xxx 2
yyyyy   2018-02-05  2018-02-05  C   xxx 2
yyyyy   2018-02-09  2018-02-09  C   xxx 2
yyyyy   2018-02-16  2018-02-16  C   xxx 3
yyyyy   2018-02-19  2018-02-19  C   xxx 2
yyyyy   2018-02-20  2018-02-20  C   xxx 2
yyyyy   2017-11-04  2017-11-04  C   xxx 2
yyyyy   2017-11-06  2017-11-06  C   xxx 2
yyyyy   2017-11-08  2017-11-08  C   xxx 2
yyyyy   2017-11-15  2017-11-15  C   xxx 2
yyyyy   2017-12-06  2017-12-06  C   xxx 3
yyyyy   2017-12-18  2017-12-18  C   xxx 2
yyyyy   2018-02-02  2018-02-02  C   xxx 2
yyyyy   2018-02-03  2018-02-03  C   xxx 2
yyyyy   2018-02-10  2018-02-10  C   xxx 2
yyyyy   2018-02-26  2018-02-26  C   xxx 2
yyyyy   2018-02-28  2018-02-28  C   xxx 2
yyyyy   2017-12-30  2017-12-30  C   xxx 2
yyyyy   2018-01-29  2018-01-29  C   xxx 2

[kousu]

I've spot checked a few, and most cases contain a cancellation (state=C, but some contain only two or three Delivered (state=D). My theory is that orders are getting call-in cancellations, but then on the day of the delivery or soon before a re-order came in, the coordinator @JeanFrancoisFournier or whoever went in to manually add one, and bam now we've got a consistency problem. The UI probably doesn't highlight when, and there's no UNIQUE of (client_id, delivery_date) on order_order that would at least crash an attempt to do such a thing.

We never want to have more than one order per client per day, that would break our business logic, so I think adding a UNIQUE constraint is the proper first step, and then we have to work backwards from that to every create on this table, turning them all into upserts that flips existing order into state=O if it exists. Or maybe add a bunch of error UI that reports the problem to the user and directs them to manually edit the order status.

Also generateorders should be more tolerant of exceptions. If the orders fail to generate for a single day that shouldn't break all the other days. Probably.

[erozqba]

@kousu Great analysis! Could you provide a patch to fix this?

[JeanFrancoisFournier]

Before hacking away at this issue, I'd love to check in with you Nick about that. Whenever you're in the office

[kousu]

To be clear, this causes generateorders to fail to run (usually) completely. This means that orders in the future may be missing. The current configuration generates 100 days in the future, so failing on one day may not be a problem because orders will have already been generated by a previous run -- for example, I saw this fail Friday and Saturday, but work again on Sunday -- but it might bite us by surprise later.

[erozqba]

@kousu @JeanFrancoisFournier any information in this issue? It's not clear to me what should be the solution.

[kousu]

@erozqba this got down-prioritized on our end because the daily workflow involves regenerated all the orders for the day, fresh, so we never run into an order made by generateorders except when looking ahead in the calendar for prediction.

The immediate solution here would be, I think, adding some other try-except clauses in judicious places. I haven't sat down to do this but I would like to, unless someone beats me to it. I kind of doubt that will happen because reproducing this is tricky without access to our production database; it's really not clear to me what the root cause here is; some combination of cancelling some portion of repeated orders and then regenerating them, maybe? Without working that out the best I can think to do is to try to patch this in production, which I'm not suuuuper keen on, and then extract the patch and push it up here.