Start bot

[ben1628]

I use TIME_ZONE = 'US/Eastern' in settings.py

and also update NYSE and Nasdaq to use timezone: US/Eastern and set the start time to 4 am and end time to 8 pm

Is there anything else I have to do? all I am getting IB gateway is the currency, I do get my accounts (I have a adviser account which may complicate things, lot)

[psemdel]

To be honest, the TIME_ZONE is setting change the time of the database, I would not change it. For the behavior of start and end time, I changed the behavior (yesterday) there won't be need for change on your side anymore.

[ben1628]

ic, I revert back the changes in settings!

In Nasdaq and NYSE, I like to start getting data from 4 am to 8 pm (including PM and AH). Hence, I set the opening time to 04:00:00 and closing to 20:00:00 and then set the Timezone to US/Eastern. Is that correct?

I update my codes to reflect your changes yesterday, thanks!

[psemdel]

Yes, it is correct. My code changes often.

There is a big bug in my dev branch concerning the reverse order. Please be careful and set Perform_order to false. I have corrected but need to see from my side if it is now working as expected.

[ben1628]

not a problem, I will be using paper account for quite a while.

What am I supposed to see once I click start_bot? It doesn't seem to be talking to IB to retrieve market data (1 minute or whatever), and I have no indication it is using AAPL and TSLA.

Have to wait until Monday to try this again.

[ben1628]

some more info:

On trade.log, i see this:

[2023-02-25 13:27:19] Starting automatic reverse order execution, symbol: AAPL [2023-02-25 13:27:19] Starting automatic reverse order execution, symbol: TSLA [2023-02-25 13:27:55] Starting automatic reverse order execution, symbol: LULU [2023-02-25 13:27:58] Starting automatic reverse order execution, symbol: PTON

and in my ib gateway,

10:32:56:658 -> ---373-1-5-DU278406--GrossPositionValue-S-59076.96-USD- 10:32:56:658 -> ---,73-1-5-DU278406--InitMarginReq-32492.33-USD- 10:32:56:658 -> ---.73-1-5-DU278406--InitMarginReq-S-32492.33-USD- 10:32:56:658 -> ---873-1-5-DU278406--LookAheadAvailableFunds-1087504.69-USD- 10:32:56:658 -> ---:73-1-5-DU278406--LookAheadAvailableFunds-S-1087504.69-USD- 10:32:56:658 -> ---973-1-5-DU278406--LookAheadExcessLiquidity-1090458.54-USD- 10:32:56:658 -> ---;73-1-5-DU278406--LookAheadExcessLiquidity-S-1090458.54-USD- 10:32:56:658 -> ---573-1-5-DU278406--LookAheadInitMarginReq-32492.33-USD- 10:32:56:658 -> ---773-1-5-DU278406--LookAheadInitMarginReq-S-32492.33-USD- 10:32:56:658 -> ---673-1-5-DU278406--LookAheadMaintMarginReq-29538.48-USD- 10:32:56:658 -> ---873-1-5-DU278406--LookAheadMaintMarginReq-S-29538.48-USD- 10:32:56:659 -> ----73-1-5-DU278406--MaintMarginReq-29538.48-USD- 10:32:56:660 -> ---/73-1-5-DU278406--MaintMarginReq-S-29538.48-USD- 10:32:56:660 -> ---/73-1-5-DU278406--NetLiquidation-1119997.02-USD- 10:32:56:660 -> ---173-1-5-DU278406--NetLiquidation-S-1119997.02-USD-

every 3 to 4 minutes, but I don't see it getting any data from IB.

Nasdaq/NYSE markets are open

[psemdel]

The alerting system will check the stocks that you have in IB or in one pf. Do you have something somewhere?

If i connected, it will write IB connected on port xy...

In the worker you should see this kind of thing

 0%|          | 0/1 [00:00<?, ?it/s]kPoolWorker-2] 
 AAPL:   0%|          | 0/1 [00:00<?, ?it/s]ker-2] 
 AAPL: 100%|##########| 1/1 [00:00<00:00,  8.02it/s]
 AAPL: 100%|##########| 1/1 [00:00<00:00,  8.01it/s]

[ben1628]

I don't see anything like what you have.

Here is what my terminal shows, I add some text to aid in my debugging, I don't see it doing any Check_PF. I must be missing something simple.

Do you spot anything unusual compared to your terminal?

5322:C 27 Feb 2023 16:30:16.480 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
5322:C 27 Feb 2023 16:30:16.480 # Redis version=6.0.16, bits=64, commit=00000000, modified=0, pid=5322, just started
5322:C 27 Feb 2023 16:30:16.480 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
5322:M 27 Feb 2023 16:30:16.481 * Increased maximum number of open files to 10032 (it was originally set to 1024).
5322:M 27 Feb 2023 16:30:16.481 # Could not create server TCP listening socket *:6379: bind: Address already in use
getting nameserver!
vectorbt  172.26.176.1
getting nameserver!
vectorbt  172.26.176.1
Watching for file changes with StatReloader
Performing system checks...

Telegram else
TelegramBot configured telegram_sub.py
System check identified no issues (0 silenced).
February 27, 2023 - 21:30:22
Django version 4.1.6, using settings 'trading_bot.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
[1]-  Exit 1                  redis-server
bot started, task id e35e52xxx
bot_start ok
[27/Feb/2023 21:30:37] "GET /start_bot/ HTTP/1.1" 302 0
[27/Feb/2023 21:30:37] "GET /reports/ HTTP/1.1" 200 3581
Not Found: /favicon.ico
[27/Feb/2023 21:30:37] "GET /favicon.ico HTTP/1.1" 404 5538
getting nameserver!
vectorbt  172.26.176.1
Telegram else
TelegramBot configured telegram_sub.py

 -------------- celery@MainPC v5.2.7 (dawn-chorus)
--- ***** -----
-- ******* ---- Linux-5.15.79.1-microsoft-standard-WSL2-x86_64-with-glibc2.35 2023-02-27 21:30:40
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app:         trading_bot:0x7fb8e4ccd990
- ** ---------- .> transport:   redis://localhost:6379/0
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> trading-bot-worker exchange=trading-bot-worker(direct) key=trading-bot-worker

[tasks]
  . reporting.telegram.async_sched
  . trading_bot.celery.debug_task

[2023-02-27 21:30:42,320: INFO/MainProcess] Connected to redis://localhost:6379/0
[2023-02-27 21:30:42,323: INFO/MainProcess] mingle: searching for neighbors
[2023-02-27 21:30:43,333: INFO/MainProcess] mingle: all alone
[2023-02-27 21:30:43,371: WARNING/MainProcess] /home/usr/.vbt/lib/python3.10/site-packages/celery/fixups/django.py:203: UserWarning: Using settings.DEBUG leads to a memory
            leak, never use this setting in production environments!
  warnings.warn('''Using settings.DEBUG leads to a memory

[2023-02-27 21:30:43,371: INFO/MainProcess] celery@MainPC ready.
[2023-02-27 21:30:43,377: INFO/MainProcess] Task reporting.telegram.async_sched[e35e52xxx] received
[2023-02-27 21:30:43,379: WARNING/ForkPoolWorker-7] inside async_sched
[2023-02-27 21:30:43,380: WARNING/ForkPoolWorker-7] init
[2023-02-27 21:30:43,381: WARNING/ForkPoolWorker-7] Config(
    api_key=None,
    weirdness=5
)
[2023-02-27 21:30:43,381: WARNING/ForkPoolWorker-7] telegram:
[2023-02-27 21:30:43,382: WARNING/ForkPoolWorker-7] {'token': '55240xxxx', 'use_context': True, 'persistence': True, 'defaults': {}, 'drop_pending_updates': True}
[2023-02-27 21:30:43,568: WARNING/ForkPoolWorker-7] register handlers
[2023-02-27 21:30:43,568: WARNING/ForkPoolWorker-7] prepare arguments
[2023-02-27 21:30:44,171: INFO/ForkPoolWorker-7] Application started
[2023-02-27 21:30:44,173: WARNING/ForkPoolWorker-7] try to send
[2023-02-27 21:30:44,174: WARNING/ForkPoolWorker-7] -724xxxx
[2023-02-27 21:30:44,412: INFO/ForkPoolWorker-7] -7244xxxx - Bot: "I'm back to life!"

Thanks in advance.

[psemdel]

I looks ok (like always, I am completing the doc with your comments). What must disturb you is that if you want to use IB to get the data, you need to set "USE_IB_FOR_DATA" to true in the settings. Personally, I don't use it.

The 0%| | 0/1 [00:00<?, ?it/s]kPoolWorker-2] AAPL: 0%| | 0/1 [00:00<?, ?it/s]ker-2]

comes afterward, after a few minutes.

[ben1628]

So you just use Yahoo for data?

I see that you just update your codes with some more info, that's nice.

I'm still trying to find out what's wrong... :)

[ben1628]

I don't see this message, that may be the reason!

[2023-02-28 07:27:38,781: INFO/ForkPoolWorker-2] **Scheduler started**

I only see this

[2023-02-28 07:27:38,984: INFO/ForkPoolWorker-2] 5402841128 - Bot: "I'm back online!" and I certainly don't see this either:

[2023-02-28 07:27:38,987: INFO/ForkPoolWorker-2] Starting schedule manager with jobs [Every 10 minutes do check_pf() (last run: [never], next run: 2023-02-28 07:37:38), Every 1 week at 08:03:00 do check_pf(opening='9h') (last run: [never], next run: 2023-03-06 08:03:00), Every 1 week at 08:03:00 do check_pf(opening='9h') (last run: [never], next run: 2023-02-28 08:03:00), Every 1 week at 08:03:00 do check_pf(opening='9h') (last run: [never], next run: 2023-03-01 08:03:00), Every 1 week at 08:03:00 do chec

[psemdel]

After starting it what happens if you put http://localhost:8000/start_bot in your browser?

Yes, I use Yahoo for data. It is not as reliable, I do agree, but it is faster. Maybe I should differentiate between alert (accuracy more important) and report (it should not take more than 10 min in total).

[ben1628]

nothing happened, it's just redirect to /report

[ben1628]

Compared to what your output

[2023-02-28 07:27:38,576: INFO/ForkPoolWorker-2] Initializing bot
[2023-02-28 07:27:38,577: INFO/ForkPoolWorker-2] Loaded chat ids [51232456]
[2023-02-28 07:27:38,781: INFO/ForkPoolWorker-2] Running bot My_telegram_bot
[2023-02-28 07:27:38,781: INFO/ForkPoolWorker-2] Scheduler started

I have different output

[2023-02-28 19:24:43,181: INFO/MainProcess] celery@MainPC ready.
reporting.telegram.async_sched[359ad928-ae3a-40ad-b94a-51c793eb3a3b] received
[2023-02-28 19:25:44,684: INFO/ForkPoolWorker-7] inside async_sched
[2023-02-28 19:25:44,684: WARNING/ForkPoolWorker-7] init
[2023-02-28 19:25:44,685: WARNING/ForkPoolWorker-7] Config(
    api_key=None,
    weirdness=5
)
[2023-02-28 19:25:44,685: WARNING/ForkPoolWorker-7] telegram:
[2023-02-28 19:25:44,685: WARNING/ForkPoolWorker-7] {'token': '5524090055:AAETBn-rhkx24X_eiSBmKh59X4Ze__khi14', 'use_context': True, 'persistence': True, 'defaults': {}, 'drop_pending_updates': True}
[2023-02-28 19:25:44,860: WARNING/ForkPoolWorker-7] register handlers
[2023-02-28 19:25:44,861: WARNING/ForkPoolWorker-7] prepare agruments
[2023-02-28 19:25:45,447: INFO/ForkPoolWorker-7] Application started
[2023-02-28 19:25:45,447: WARNING/ForkPoolWorker-7] try to send
[2023-02-28 19:25:45,448: WARNING/ForkPoolWorker-7] -724460572
[2023-02-28 19:25:45,745: INFO/ForkPoolWorker-7] -724460572 - Bot: "I'm back to life!"

I have Application started while you have

Initializing bot
Running bot My_telegram_bot

so my codes are calling

async_sched.delay(TELEGRAM_TOKEN)

but I don't think it is calling

bot.start(in_background=True)
  MyScheduler(bot)

[ben1628]

I think there is a difference in how celery is interacting with Django 4.1 compared to earlier version. That may be why I am seeing 'Application started', it doesn't seem to call start() afterwards.

I'm using Django 4.1!

[psemdel]

I will update and try again. Basically, there is no need for all command lines to be started in one file. You can perfectly open one terminal start redis there, open another terminal start celery there, and a third for Django. Then when everything is running open the browser with the url. I cannot identify anything wrong, with what is displayed on your side.

[ben1628]

I know, have to do some more investigation. I will probably run some simple tests to make sure they're talking to each others

[ben1628]

I update to latest redis 7.0.9 (I think you're using 6.5, mine was 6.0).

Something is getting interesting now. I got an error 'Make sure that only one bot instance is running', however it looks like the program is running periodically which means the bot is alive!

  File "/home/usr/.vbt/lib/python3.10/site-packages/telegram/ext/_updater.py", line 607, in _network_loop_retry
    if not await action_cb():
  File "/home/usr/.vbt/lib/python3.10/site-packages/telegram/ext/_updater.py", line 335, in polling_action_cb
    raise exc
  File "/home/usr/.vbt/lib/python3.10/site-packages/telegram/ext/_updater.py", line 320, in polling_action_cb
    updates = await self.bot.get_updates(
  File "/home/usr/.vbt/lib/python3.10/site-packages/telegram/ext/_extbot.py", line 524, in get_updates
    updates = await super().get_updates(
  File "/home/usr/.vbt/lib/python3.10/site-packages/telegram/_bot.py", line 331, in decorator
    result = await func(*args, **kwargs)  # skipcq: PYL-E1102
  File "/home/usr/.vbt/lib/python3.10/site-packages/telegram/_bot.py", line 3510, in get_updates
    await self._post(
  File "/home/usr/.vbt/lib/python3.10/site-packages/telegram/_bot.py", line 419, in _post
    return await self._do_post(
  File "/home/usr/.vbt/lib/python3.10/site-packages/telegram/ext/_extbot.py", line 306, in _do_post
    return await super()._do_post(
  File "/home/usr/.vbt/lib/python3.10/site-packages/telegram/_bot.py", line 450, in _do_post
    return await request.post(
  File "/home/usr/.vbt/lib/python3.10/site-packages/telegram/request/_baserequest.py", line 165, in post
    result = await self._request_wrapper(
  File "/home/usr/.vbt/lib/python3.10/site-packages/telegram/request/_baserequest.py", line 330, in _request_wrapper
    raise Conflict(message)
telegram.error.Conflict: Conflict: terminated by other getUpdates request; make sure that only one bot instance is running

So have to figure out why more than one bot is running.

[psemdel]

I checked, I use Django 4.1.2. Celery 5.2.7 and redis 4.3.4 in pip. Let me update redis to...4.5.1. When it starts it shows Redis: 6.2.6 (no idea, why there are discrepancies in the numbers). The main difference is our Python version. I am still on 3.8.

Concerning the two bots running, close your terminal and python editor. If it is still there, restart your computer.

[ben1628]

To help me with my debugging, is it possible that you can put in some more text into info.txt so I can make sure that my codes are calling those functions. I have no ideas whether the program is doing anything or just sitting idle.

I know what the code is supposed to do next, but I don't see it happening even after I try to put some text into mine.

I'm on python 3.10.6

Again, thanks for your help...

[psemdel]

Is tests/telegram_minimal.ipynb working on your side. It would prove at least that the Telegram part is ok. Then put "HEARTBEAT":True in the settings, so you will get often some messages (every 10 seconds). In the def heartbeat_f(self) in telegram.py you can also add a print to see if something happens. Otherwise there is a tutorial for the scheduler in vectorbt https://nbviewer.org/github/polakowo/vectorbt/blob/master/examples/TelegramSignals.ipynb There was also a minimum example for the scheduler, but I cannot find it right now.

[psemdel]

You could try something like

You can try

import vectorbtpro as vbt

class MyScheduler():
    def __init__(self, **kwargs):
        self.manager = vbt.ScheduleManager()
        self.manager.every(10, 'seconds').do(self.test)

    def test(self): 
        print("test ok")

if __name__ == '__main__':
    s=MyScheduler()
    s.manager.start_in_background()

[ben1628]

Have been fighting this issue for a while. Here is what I have so far.

1/ I use telegram_minimal.ipynb in tests,

telegram_bot = vbt.TelegramBot(token="bot_token")
telegram_bot.start(in_background=True)

will always return runtime error: This event loop is already running

Don't know why, but I can still continue

#if nothing happens, think of making /start in telegram directly
await telegram_bot.send_message(-7244xxxx (Group chat id),"test whoever")
await telegram_bot.send_message_to_all("test second another whoever")

Have to add Await, which is fine.

I need to send an message to that group first, and then I can send_message_to_all afterwards.

If I just send an message to the group from my desktop or mobile and then just send message to all here, it won't work.

So I must be doing something wrong.

P.S. I have also disable privacy in bot as suggested by others, then remove the bot from the group and re-add it back. I also change bot permission to Admin.

Still working on this :(

[ben1628]

[2023-03-03 14:36:01,610: WARNING/ForkPoolWorker-7] /home/usr/.vbt/lib/python3.10/site-packages/celery/app/trace.py:285: RuntimeWarning: coroutine 'Updater.start_polling' was never awaited

There were async def before in the codes, but was commented out

[ben1628]

This may be my problem since I'm using python 3.10

https://blog.teclado.com/changes-to-async-event-loops-in-python-3-10/

[ben1628]

Finally, I have some success...

The problem is due to python-telegram-bot version.

I was using latest version which is 20, as it's evident from the message 'Telegram Else'. I don't see the same message in your Appendix which lead me to think you were using a lower version.

Version 20 behaves quite different from previous version, I test it with a simple program and is working as expected. So I downgrade it to 13.15, and run the telegram-min.ipynb from vectorbtpro (Funny, it did have a snippet to test version and warned it won't be compatible). It's working well now.

I will have to wait until tomorrow to test it when the market is open.

I create another virtual environment using python 3.8.6, but I don't think that's the problem.

[ben1628]

I'm quite happy with the outcome, It's up to you when you want to fix the problem with version 20, though it's not urgent

everything seems to be working! Moving onto next task

Thanks for your help.

[psemdel]

Ah s***, I have a déjà vu with this topic. I am quite sure I ran into the same issue at one point when upgrading the packages. I actually run 13.14. I will add the same warning.