Migration error

[darshan-talati]

Following Docker installation guide, and hit with an error when creating django-migrations container for the first time. Here is the error -

Traceback (most recent call last): File "/home/jovyan/py-trading-bot/manage.py", line 22, in main() File "/home/jovyan/py-trading-bot/manage.py", line 18, in main execute_from_command_line(sys.argv) File "/opt/conda/lib/python3.9/site-packages/django/core/management/init.py", line 442, in execute_from_command_line utility.execute() File "/opt/conda/lib/python3.9/site-packages/django/core/management/init.py", line 416, in execute django.setup() File "/opt/conda/lib/python3.9/site-packages/django/init.py", line 24, in setup apps.populate(settings.INSTALLED_APPS) File "/opt/conda/lib/python3.9/site-packages/django/apps/registry.py", line 124, in populate app_config.ready() File "/home/jovyan/py-trading-bot/orders/apps.py", line 13, in ready if len(StockStatus.objects.all())==0: File "/opt/conda/lib/python3.9/site-packages/django/db/models/query.py", line 380, in len self._fetch_all() File "/opt/conda/lib/python3.9/site-packages/django/db/models/query.py", line 1881, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/opt/conda/lib/python3.9/site-packages/django/db/models/query.py", line 91, in iter results = compiler.execute_sql( File "/opt/conda/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql cursor.execute(sql, params) File "/opt/conda/lib/python3.9/site-packages/django/db/backends/utils.py", line 102, in execute return super().execute(sql, params) File "/opt/conda/lib/python3.9/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers( File "/opt/conda/lib/python3.9/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers return executor(sql, params, many, context) File "/opt/conda/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute return self.cursor.execute(sql, params) File "/opt/conda/lib/python3.9/site-packages/django/db/utils.py", line 91, in exit raise dj_exc_value.with_traceback(traceback) from exc_value File "/opt/conda/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "orders_stockstatus" does not exist LINE 1: ...tegy_id", "orders_stockstatus"."order_in_ib" FROM "orders_st...

[haroldmei]

I hit the same problem running with WSL2:

Traceback (most recent call last):
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/utils/autoreload.py", line 64, in wrapper
    fn(*args, **kwargs)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 125, in inner_run
    autoreload.raise_last_exception()
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/utils/autoreload.py", line 87, in raise_last_exception
    raise _exception[1]
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/core/management/__init__.py", line 394, in execute
    autoreload.check_errors(django.setup)()
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/utils/autoreload.py", line 64, in wrapper
    fn(*args, **kwargs)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/apps/registry.py", line 124, in populate
    app_config.ready()
  File "/home/hmei/py-trading-bot/py-trading-bot/orders/apps.py", line 13, in ready
    if len(StockStatus.objects.all())==0:
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/db/models/query.py", line 382, in __len__
    self._fetch_all()
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/db/models/query.py", line 1928, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/db/models/query.py", line 91, in __iter__
    results = compiler.execute_sql(
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
    cursor.execute(sql, params)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/db/backends/utils.py", line 122, in execute
    return super().execute(sql, params)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/db/backends/utils.py", line 100, in _execute
    with self.db.wrap_database_errors:
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "orders_stockstatus" does not exist
LINE 1: ...tegy_id", "orders_stockstatus"."order_in_ib" FROM "orders_st...

[psemdel]

Ok, I will try it on my side today evening. Sorry for the inconvenience.

[psemdel]

I confirm that there are some issues there. I will work on it.

[psemdel]

So I added a

    try:
        if len(StockStatus.objects.all())==0:
            create_ss_sub()
    except:
        pass

To handle the problem. In addition, I added a try/except to handle the fact that keras may not be installed. I hope it will work on your side.

[haroldmei]

Hi @psemdel, Thanks a lot for spending time looking into the issue. I tried running again with my wsl2 environment, however there is still a problem with the missing relation.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
    cursor.execute(sql, params)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/db/backends/utils.py", line 122, in execute
    return super().execute(sql, params)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/db/backends/utils.py", line 100, in _execute
    with self.db.wrap_database_errors:
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "orders_stockex" does not exist
LINE 1: ...in_index_id", "orders_stockex"."calc_report" FROM "orders_st...
                                                             ^

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/hmei/py-trading-bot/py-trading-bot/reporting/views.py", line 14, in reportsView
    return render(request, 'reporting/reports.html', context)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/shortcuts.py", line 25, in render
    content = loader.render_to_string(template_name, context, request, using=using)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/template/loader.py", line 62, in render_to_string
    return template.render(context, request)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/template/backends/django.py", line 61, in render
    return self.template.render(context)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/template/base.py", line 171, in render
    return self._render(context)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/template/base.py", line 163, in _render
    return self.nodelist.render(context)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/template/base.py", line 1000, in render
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/template/base.py", line 1000, in <listcomp>
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/template/base.py", line 961, in render_annotated
    return self.render(context)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/template/base.py", line 1065, in render
    return render_value_in_context(output, context)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/template/base.py", line 1042, in render_value_in_context
    value = str(value)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/forms/utils.py", line 79, in __str__
    return self.as_widget()
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/forms/boundfield.py", line 108, in as_widget
    return widget.render(
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/forms/widgets.py", line 278, in render
    context = self.get_context(name, value, attrs)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/forms/widgets.py", line 764, in get_context
    context = super().get_context(name, value, attrs)
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/forms/widgets.py", line 715, in get_context
    context["widget"]["optgroups"] = self.optgroups(
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/forms/widgets.py", line 655, in optgroups
    for index, (option_value, option_label) in enumerate(self.choices):
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django_filters/fields.py", line 256, in __iter__
    yield from iterable
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/forms/models.py", line 1422, in __iter__
    for obj in queryset:
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/db/models/query.py", line 518, in _iterator
    yield from iterable
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/db/models/query.py", line 91, in __iter__
    results = compiler.execute_sql(
  File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1565, in execute_sql
    cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139757766047296_sync_1" does not exist

[24/Mar/2024 11:22:35] "GET / HTTP/1.1" 500 211282

PS, I tried running the optional 'import dump.json' as well, and got the following error:

python3 manage.py loaddata dump.json

Traceback (most recent call last):
File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/core/serializers/json.py", line 69, in Deserializer
objects = json.loads(stream_or_string)
File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 6 (char 5)

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/hmei/py-trading-bot/py-trading-bot/manage.py", line 22, in main() File "/home/hmei/py-trading-bot/py-trading-bot/manage.py", line 18, in main execute_from_command_line(sys.argv) File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/core/management/init.py", line 442, in execute_from_command_line utility.execute() File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/core/management/init.py", line 436, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/core/management/base.py", line 413, in run_from_argv self.execute(*args, *cmd_options) File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/core/management/base.py", line 459, in execute output = self.handle(args, **options) File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/core/management/commands/loaddata.py", line 102, in handle self.loaddata(fixture_labels) File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/core/management/commands/loaddata.py", line 163, in loaddata self.load_label(fixture_label) File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/core/management/commands/loaddata.py", line 251, in load_label for obj in objects: File "/home/hmei/miniconda3/envs/vectorbtpro/lib/python3.10/site-packages/django/core/serializers/json.py", line 74, in Deserializer raise DeserializationError() from exc django.core.serializers.base.DeserializationError: Problem installing fixture '/home/hmei/py-trading-bot/py-trading-bot/dump.json':

Cheers.

[psemdel]

No problem, I like also when the code is working.

The errors you see, django_curs_139757766047296..., are because the migration did not take place. Check the migration job.

For the dump, there was that at the beginning (and yes, no idea, why I did not see that yesterday): [2023-12-08 16:47:11] INFO 159 NumExpr defaulting to 4 threads.

I deleted it, it seems fine now.

I use Python 3.9. I am building an environment with Python 3.10... to test it with it.

[psemdel]

It seems to work also with Python 3.10

[darshan-talati]

Migration worked for me after the fix, but the bot has been throwing the following error -

{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"py-trading-bot\" in pod \"py-trading-bot-         │
│6879c9b769-tjqwd\" is waiting to start: CreateContainerConfigError","reason":"BadRequest","code":400}

[psemdel]

That's the log?

[darshan-talati]

Yes.

[psemdel]

I need to investigate more in details... Probably this weekend. I suppose, I still have to work on this docker image.

[psemdel]

So sorry, that it took me so long. I am quite busy right now. So I did it all step by step (each yaml file applied one after the others, checking each step). I confirm it works. To be even more sure, I will use from now the Docker version on daily basis, normally I just run it locally.

Concerning your error, it is often caused by a missing configuration map or secret. Did you do the: kubectl apply -f py-trading-bot/kubernetes/configmap.yml kubectl apply -f py-trading-bot/kubernetes/secret.yml

?

Moreover, I would advise to start again from scratch. I had also problems, because I thought that I was working with a clean DB, but now, it was the old one. So did you remove the PV and PVC before retrying. I would sugger to delete your cluster (minikube delete). And try again. With DB half migrated and systems half in their old state and half in their new state, it can quickly become messy.