Docker

[lucali2014]

Appreciate your effort for this project. I tried to deploy this project on my local MacBook but failed. Is it possible to deploy it on Docker? It'll be very convenient for the users to deploy, thanks.

[psemdel]

I can try. The installation of TA-lib is the tricky part.

[psemdel]

So it won't be a quick shot.

1. I need to create a branch with vertorbt instead of vectorbtpro. As you may not have vectorbtpro and I suppose you want first to see if something works before investing more money.
2. I will create a Dockerfile based on the vectorbt one (as it is the main lib for my bot). It itself bases on jupyter. Jupyter image is 2.5 Gb heavy (alone), vectorbt pro is 4.7 Gb, be ready!
3. Then I have to create the manifests for the bot. As it uses Django, a pod for the DB, a pod for Redis and one for the worker must run. So it needs some structure. So it won't be a light container / light solution. But I will definitely try. In parallel, I will write some clear installation instruction. Here I suppose too many things.

[lucali2014]

Appreciate it. You are very helpful. As I am a vectorbtpro user, you can just focus on step2 and stpe3 for now which may save some of your time. Thanks again.

[psemdel]

So It seems that I made it work. I will test it again from a to z, and check everything again. I am unfortunately away for the next days. I will finish on Sunday.

[psemdel]

I let you try.

[lucali2014]

Thanks for your effort. I tried to deploy it. But I found some pods is not ready, what should I do to solve this porblem? thanks. NAME READY STATUS RESTARTS AGE django-migrations-8zdjb 0/1 ErrImageNeverPull 0 14m py-trading-bot-d79b7866b-7vbxx 0/1 ErrImageNeverPull 0 14m py-trading-bot-postgres-c899fc6bd-mqmz8 1/1 Running 1 (7m ago) 15m py-trading-bot-redis-557dc7d54c-w5tkh 1/1 Running 1 (5m12s ago) 15m py-trading-bot-worker-688df49dcc-qdnrx 0/1 ErrImageNeverPull 0 14m

[psemdel]

It seems you did not create the image py-trading-bot or at least your minikube cannot find it. When you type "docker images" from minikube, do you see the py-trading-bot?

[lucali2014]

yes. I have the py-trading-bot on docker.

Docker info as below: kicbase/stable | IN USE | v0.0.34 | 9e28dc02239f | about 2 months ago | 1.06 GB | RUN py-trading-bot latest 8b15e4552aa0 7 minutes ago 4.67 GB

Installation info as below => => transferring context: 312.85MB 6.3s => [ 2/11] WORKDIR /tmp 0.6s => [ 3/11] RUN apt-get update && apt-get install -yq --no-install-recommends cmake && apt-get clean && rm 13.4s => [ 4/11] RUN wget https://netcologne.dl.sourceforge.net/project/ta-lib/ta-lib/0.4.0/ta-lib-0.4.0-src.tar.gz 46.4s => [ 5/11] RUN rm -R ta-lib ta-lib-0.4.0-src.tar.gz 0.2s => [ 6/11] RUN pip install --quiet --no-cache-dir 'jupyter-dash' 'plotly>=5.0.0' && fix-permissio 97.1s => [ 7/11] RUN pip install --quiet --no-cache-dir 'numpy==1.21' 'numba==0.55.1' 'schedule' ' 214.2s => [ 8/11] RUN pip install --quiet --no-cache-dir django==4.1.2 'asyncio'==3.4.3 'celery[redis]'==5.2 18.7s => [ 9/11] RUN pip install -U "vectorbtpro[base] @ git+https://@githu 31.7s => [10/11] COPY . /home/jovyan/ 0.8s => [11/11] WORKDIR /home/jovyan/py-trading-bot/ 0.0s => exporting to image 9.1s => => exporting layers 9.1s => => writing image sha256:8b15e4552aa05fcd4bbd2b00514bb387a31441f1f1ead68093d27f541aeafb23 0.0s => => naming to docker.io/library/py-trading-bot 0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them (base) luca@LucadeMBP-2 py-trading-bot-main % ./py-trading-bot/kubernetes/sequence_start.sh configmap/py-trading-bot-configmap unchanged secret/py-trading-bot-secret unchanged deployment.apps/py-trading-bot-redis unchanged service/py-trading-bot-redis-service unchanged deployment.apps/py-trading-bot-postgres unchanged service/py-trading-bot-postgres-service unchanged persistentvolume/py-trading-bot-postgres-pv unchanged persistentvolumeclaim/py-trading-bot-postgres-pvc unchanged deployment.apps/py-trading-bot unchanged service/py-trading-bot-service unchanged job.batch/django-migrations unchanged deployment.apps/py-trading-bot-worker unchanged (base) luca@LucadeMBP-2 py-trading-bot-main % kubectl get pods NAME READY STATUS RESTARTS AGE django-migrations-8zdjb 0/1 ErrImageNeverPull 0 9h py-trading-bot-d79b7866b-7vbxx 0/1 ErrImageNeverPull 0 9h py-trading-bot-postgres-c899fc6bd-mqmz8 1/1 Running 3 (5h17m ago) 9h py-trading-bot-redis-557dc7d54c-w5tkh 1/1 Running 3 (5h17m ago) 9h py-trading-bot-worker-688df49dcc-qdnrx 0/1 ErrImageNeverPull 0 9h (base) luca@LucadeMBP-2 py-trading-bot-main %

[psemdel]

Hum strange. The ErrImageNeverPull means that it does not find it locally (as I set imagePullPolicy: Never, it won't look in Docker hub for the image). I can only share my experience with this kind of topic (which was not gathered with this image ;)).

When I start a terminal without minikube or anything and do "docker images" I have get a list of images. When I then start minikube with "minikube start" and then do "eval $(minikube docker-env)" I will see a different list of images with docker images. The discrepancy between both lists is a gotcha. The image needs to be, at least in my experience, in the second list.

Normally the problem does not happen, if you build the image from minikube and after the eval. If for some reason, the image is only in the first list but not in the second, you can do "minikube image load py-trading-bot". Alternatively, build it directly in minikube with "minikube image build -t py-trading-bot --build-arg GH_TOKEN=<>".

[lucali2014]

Tried several times, I got another error now. Not quite sure what happened. Appreciate your help~

(base) luca@LucadeMacBook-Pro-2 py-trading-bot-main % kubectl get pods NAME READY STATUS RESTARTS AGE django-migrations-59hk7 0/1 Error 0 2m58s django-migrations-gjkpt 0/1 Error 0 2m43s django-migrations-jbj4b 0/1 Error 0 3m18s py-trading-bot-d79b7866b-www2q 1/1 Running 0 3m18s py-trading-bot-postgres-c899fc6bd-p2bcj 1/1 Running 0 3m58s py-trading-bot-redis-557dc7d54c-f8p86 1/1 Running 0 3m58s py-trading-bot-worker-688df49dcc-jtbds 1/1 Running 0 3m17s

[psemdel]

Ok that's much better. It means the pods found the images, we see it is all running. The django-migrations is just a job. To know what went wrong do "kubectl logs django-migrations-jbj4b", it will display the logs. I tested it though, but this was the last part of my implementation...

[lucali2014]

I got the erros just now, it seems a high version libpq is needed. Do you know how to upgrade it? thanks.

/opt/conda/lib/python3.9/site-packages/django/core/management/commands/makemigrations.py:143: RuntimeWarning: Got an error checking a consistent migration history performed for database connection 'default': SCRAM authentication requires libpq version 10 or above

[psemdel]

First sorry for the delay, I was (again) on my way. I feared that I made my tests wrong, and repeated them. On my side everything is fine. A look in google told me that your error related to (Apple)-M1-CPU See https://stackoverflow.com/questions/62807717/how-can-i-solve-postgresql-scram-authentication-problem

[psemdel]

They recommand using psycopg2 instead of psycopg2-binary. However, it is not easy, I always got an error/warning if I try to install psycopg directly.

[lucali2014]

Not at all, I am appreciated for your huge effort on this. I deployed it to another laptop (None M1), the good news is it works now. But there is still some little problem, it seems the data is not imported to the database properly, I see so many exceptions on the website. Do you have any idea about how to fix this problem? Thanks a lot.

django-migrations logs: Traceback (most recent call last): File "/opt/conda/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "auth_user_username_key" DETAIL: Key (username)=(admin) already exists. The above exception was the direct cause of the following exception: Traceback (most recent call last):

exceptions on the website. relation "reporting_report" does not exist LINE 1: SELECT COUNT(*) AS "__count" FROM "reporting_report"

[psemdel]

I updated the documentation for that. The first error is because you already have the superuser created in the DB (probably last time you tried and it crashed) The second one is because the migration did not work, which is logic as it is the step after the creation of the superuser in the script and it crashed at that point.

[lucali2014]

Sorry to bother you again. I deleted all the pods and started to deploy from a new environment with your lastest document, but I still encountered some problems when importing the data to the database. The errors are as below.

Applying auth.0009_alter_user_last_name_max_length... OK Applying auth.0010_alter_group_name_max_length... OK Applying auth.0011_update_proxy_permissions... OK Applying auth.0012_alter_user_first_name_max_length... OK Applying sessions.0001_initial... OK

Traceback (most recent call last): File "/opt/conda/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "django_content_type_app_label_model_76bd3d3b_uniq" DETAIL: Key (app_label, model)=(orders, action) already exists.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/jovyan/py-trading-bot/manage.py", line 22, in main() File "/home/jovyan/py-trading-bot/manage.py", line 18, in main execute_from_command_line(sys.argv) File "/opt/conda/lib/python3.9/site-packages/django/core/management/init.py", line 446, in execute_from_command_line utility.execute() File "/opt/conda/lib/python3.9/site-packages/django/core/management/init.py", line 440, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/opt/conda/lib/python3.9/site-packages/django/core/management/base.py", line 402, in run_from_argv self.execute(*args, *cmd_options) File "/opt/conda/lib/python3.9/site-packages/django/core/management/base.py", line 448, in execute output = self.handle(args, options) File "/opt/conda/lib/python3.9/site-packages/django/core/management/commands/loaddata.py", line 102, in handle self.loaddata(fixture_labels) File "/opt/conda/lib/python3.9/site-packages/django/core/management/commands/loaddata.py", line 163, in loaddata self.load_label(fixture_label) File "/opt/conda/lib/python3.9/site-packages/django/core/management/commands/loaddata.py", line 253, in load_label if self.save_obj(obj): File "/opt/conda/lib/python3.9/site-packages/django/core/management/commands/loaddata.py", line 209, in save_obj obj.save(using=self.using) File "/opt/conda/lib/python3.9/site-packages/django/core/serializers/base.py", line 288, in save models.Model.save_base(self.object, using=using, raw=True, kwargs) File "/opt/conda/lib/python3.9/site-packages/django/db/models/base.py", line 863, in save_base updated = self._save_table( File "/opt/conda/lib/python3.9/site-packages/django/db/models/base.py", line 976, in _save_table updated = self._do_update( File "/opt/conda/lib/python3.9/site-packages/django/db/models/base.py", line 1040, in _do_update return filtered._update(values) > 0 File "/opt/conda/lib/python3.9/site-packages/django/db/models/query.py", line 1215, in _update return query.get_compiler(self.db).execute_sql(CURSOR) File "/opt/conda/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql cursor = super().execute_sql(result_type) File "/opt/conda/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1398, in execute_sql cursor.execute(sql, params) File "/opt/conda/lib/python3.9/site-packages/django/db/backends/utils.py", line 103, in execute return super().execute(sql, params) File "/opt/conda/lib/python3.9/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers( File "/opt/conda/lib/python3.9/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers return executor(sql, params, many, context) File "/opt/conda/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute return self.cursor.execute(sql, params) File "/opt/conda/lib/python3.9/site-packages/django/db/utils.py", line 91, in exit raise dj_exc_value.with_traceback(traceback) from exc_value File "/opt/conda/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute return self.cursor.execute(sql, params) django.db.utils.IntegrityError: Problem installing fixture '/home/jovyan/py-trading-bot/dump.json': Could not load contenttypes.ContentType(pk=11): duplicate key value violates unique constraint "django_content_type_app_label_model_76bd3d3b_uniq" DETAIL: Key (app_label, model)=(orders, action) already exists.

(base) luca@MBP ~ %

[psemdel]

Sad sad sad. I made some tests, I am not able to reproduce your issue, but it seems that the dump contains my superuser. It does not throw an error, but it's a fact. So there is indeed something to improve by the migration and dump. I am a bit out of patience for today, I will try again tomorrow.

[psemdel]

Oh, it was a long fight. First, I understood why I was not seeing the same as you: somehow it kept on reloading my first db (hostPath, see readme) so my tests were completely wrong. Then I noticed that the init.py are required (I had already an exception in my gitignore but for some reasons, did not had effect). Then I had a big issue with permission on those init. I set the User as root... I have to find a better way. Eventually, I had to fix the migration like in https://www.youtube.com/watch?v=N4gjiJumTZg . There is still my users in the dump, but when I try to remove it, I have a bug with foreign key. So in short, it is not perfect, but I was able to test it on my side.

[lucali2014]

Got it. Relax a little bit, you can try to fix it when you are available and have enough time. Thanks for your effort and patience.

[psemdel]

No thank you, properly tested softwares don't require so much beta testing... The problems you mentioned are real, they need to be fixed.