custom exchange drivers does not appear and error [You are missing the "plugins.py" file]

[uttamsingh]

Describe the bug Creating custom exchange drivers

To Reproduce Steps to reproduce the behavior:

1. Create a plugins.py from drivers.AngleOne import AngleOneSpot

import_candles_drivers = { 'AngleOne Spot': AngleOneSpot }

3. AngleOne Spot will not appear as it is not present in exchanges env (config.py)

'exchanges': { 'AngleOne Spot': { 'fee': 0, 'type': 'spot',

accepted values are: 'cross' and 'isolated'

            'futures_leverage_mode': 'isolated',
            # 1x, 2x, 10x, 50x, etc. Enter as integers
            'futures_leverage': 1,
            'balance': 10_000,
        },

Expected behavior New exchange to appear in UI

Screenshots If applicable, add screenshots to help explain your problem.

Enviroment (please complete the following information):

• OS: [Docker]
• Version [use pip show jesse]

Additional context This result in below for some reason even though plugins.py is added it is not able to find

jesse | Traceback (most recent call last): jesse | File "/jesse-docker/jesse/modes/import_candles_mode/init.py", line 77, in run jesse | driver: CandleExchange = drivers[exchange]() jesse | TypeError: 'module' object is not callable jesse | jesse | During handling of the above exception, another exception occurred: jesse | jesse | Traceback (most recent call last): jesse | File "/jesse-docker/jesse/services/multiprocessing.py", line 20, in run jesse | mp.Process.run(self) jesse | File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run jesse | self._target(*self._args, **self._kwargs) jesse | File "/jesse-docker/jesse/modes/import_candles_mode/init.py", line 81, in run jesse | raise FileNotFoundError('You are missing the "plugins.py" file') jesse | FileNotFoundError: You are missing the "plugins.py" file jesse |

[uttamsingh]

Few more logs .. this time around i changed forceload=true for driver load

jesse       | INFO:     ('172.24.0.1', 61620) - "WebSocket /ws" [accepted]
jesse       | INFO:     172.24.0.1:61614 - "POST /general-info HTTP/1.1" 200 OK
jesse       |  *************** 
jesse       | {'AngleOne Spot': <module 'drivers.AngleOne.AngleOneSpot' from '/home/drivers/AngleOne/AngleOneSpot.py'>}
jesse       |  *************** 
jesse       | {'AngleOne Spot': <module 'drivers.AngleOne.AngleOneSpot' from '/home/drivers/AngleOne/AngleOneSpot.py'>}
jesse       | INFO:     172.24.0.1:61614 - "POST /get-config HTTP/1.1" 200 OK
jesse       | INFO:     172.24.0.1:61614 - "POST /update-config HTTP/1.1" 200 OK
jesse       | INFO:     172.24.0.1:61632 - "POST /import-candles HTTP/1.1" 202 Accepted
jesse       |  *************** 
jesse       | {'AngleOne Spot': <module 'drivers.AngleOne.AngleOneSpot' from '/home/drivers/AngleOne/AngleOneSpot.py'>}
jesse       |  *************** 
jesse       | {'AngleOne Spot': <module 'drivers.AngleOne.AngleOneSpot' from '/home/drivers/AngleOne/AngleOneSpot.py'>}
jesse       | [2022-08-28 17:42:26,151] [timeloop] [INFO] Starting Timeloop..
jesse       | [2022-08-28 17:42:26,156] [timeloop] [INFO] Registered job <function run.<locals>.handle_time at 0x4026d9d700>
jesse       | [2022-08-28 17:42:26,157] [timeloop] [INFO] Timeloop now started. Jobs will run based on the interval set
jesse       | Unhandled exception in the process:
jesse       | Traceback (most recent call last):
jesse       |   File "/jesse-docker/jesse/modes/import_candles_mode/__init__.py", line 77, in run
jesse       |     driver: CandleExchange = drivers[exchange]()
jesse       | TypeError: 'module' object is not callable
jesse       | 
jesse       | During handling of the above exception, another exception occurred:
jesse       | 
jesse       | Traceback (most recent call last):
jesse       |   File "/jesse-docker/jesse/services/multiprocessing.py", line 20, in run
jesse       |     mp.Process.run(self)
jesse       |   File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
jesse       |     self._target(*self._args, **self._kwargs)
jesse       |   File "/jesse-docker/jesse/modes/import_candles_mode/__init__.py", line 81, in run
jesse       |     raise FileNotFoundError('You are missing the "plugins.py" file')
jesse       | FileNotFoundError: You are missing the "plugins.py" file
jesse       | 
postgres    | 2022-08-28 17:42:26.861 UTC [27] WARNING:  could not open statistics file "pg_stat_tmp/global.stat": Operation not permitted

and this is the code snippet which I modified in container -

_local_drivers = locate('plugins.import_candles_drivers', forceload=True)
print(' *************** ')
print(_local_drivers)
# drivers must be a dict which is merged of _builtin_drivers and _local_drivers
drivers = {}
drivers.update(_builtin_drivers)
if _local_drivers is not None:
    drivers.update(_local_drivers)

[saleh-mir]

Thank you for reporting. I'm a little busy at the moment but I will check this as soon as I can.

[saleh-mir]

I tested it with my custom_drivers/test_exchange.py being:

import requests

import jesse.helpers as jh
from jesse import exceptions
from jesse.modes.import_candles_mode.drivers.interface import CandleExchange

class TestExchange(CandleExchange):
    def __init__(self) -> None:
        # import here instead of the top of the file to prevent the possible circular imports issue
        from jesse.modes.import_candles_mode.drivers.Binance.BinanceSpot import BinanceSpot

        super().__init__(
            name='Test Exchange',
            count=1440,
            rate_limit_per_second=6,
            backup_exchange_class=BinanceSpot
        )

    def get_starting_time(self, symbol):
        formatted_symbol = symbol.replace('USDT', 'PERP')

        end_timestamp = jh.now()
        start_timestamp = end_timestamp - (86400_000 * 365 * 8)

        payload = {
            'resolution': 86400,
            'start_time': start_timestamp / 1000,
            'end_time': end_timestamp / 1000,
        }

        response = requests.get(
            f'https://ftx.com/api/markets/{formatted_symbol}/candles',
            params=payload
        )

        self._handle_errors(response)

        data = response.json()['result']

        # since the first timestamp doesn't include all the 1m
        # candles, let's start since the second day then
        first_timestamp = int(data[0]['time'])
        second_timestamp = first_timestamp + 60_000 * 1440

        return second_timestamp

    def fetch(self, symbol, start_timestamp):
        end_timestamp = start_timestamp + (self.count - 1) * 60000

        payload = {
            'resolution': 60,
            'start_time': start_timestamp / 1000,
            'end_time': end_timestamp / 1000,
        }

        formatted_symbol = symbol.replace('USDT', 'PERP')

        response = requests.get(
            f'https://ftx.com/api/markets/{formatted_symbol}/candles',
            params=payload
        )

        self._handle_errors(response)

        data = response.json()['result']
        candles = []

        for d in data:
            candles.append({
                'id': jh.generate_unique_id(),
                'symbol': symbol,
                'exchange': self.name,
                'timestamp': int(d['time']),
                'open': float(d['open']),
                'close': float(d['close']),
                'high': float(d['high']),
                'low': float(d['low']),
                'volume': float(d['volume'])
            })

        return candles

    def _handle_errors(self, response):
        # Exchange In Maintenance
        if response.status_code == 502:
            raise exceptions.ExchangeInMaintenance('ERROR: 502 Bad Gateway. Please try again later')

        if response.status_code != 200:
            raise Exception(response.json()['error'])

And my plugins.py file being:

from custom_drivers.test_exchange import TestExchange

import_candles_drivers = {
    # this is for example only. Otherwise we now have built-in support for "FTX Futures"
    'Custom exchange': TestExchange
}

It loads the driver now but it's not on the list because since a few versions ago list of supported exchanges is loaded from the config dict (as you said). I need to refactor it to auto load in the config.

This might take a while as I'm currently focused on another task.

[uttamsingh]

thanks @saleh-mir for responding .. I am not sure why are you not getting but in docker env actual stack trace is -

[INFO] Timeloop now started. Jobs will run based on the interval set jesse | <module 'drivers.AngleOne.AngleOneSpot' from '/home/drivers/AngleOne/AngleOneSpot.py'> jesse | Traceback (most recent call last): jesse | File "/jesse-docker/jesse/modes/import_candles_mode/init.py", line 80, in run jesse | driver: CandleExchange = drivers[exchange]() jesse | TypeError: 'module' object is not callable jesse | Unhandled exception in the process: jesse | Traceback (most recent call last): jesse | File "/jesse-docker/jesse/modes/import_candles_mode/init.py", line 80, in run jesse | driver: CandleExchange = drivers[exchange]() jesse | TypeError: 'module' object is not callable jesse | jesse | During handling of the above exception, another exception occurred: jesse | jesse | Traceback (most recent call last): jesse | File "/jesse-docker/jesse/services/multiprocessing.py", line 20, in run jesse | mp.Process.run(self) jesse | File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run jesse | self._target(*self._args, **self._kwargs) jesse | File "/jesse-docker/jesse/modes/import_candles_mode/init.py", line 85, in run jesse | raise FileNotFoundError('You are missing the "plugins.py" file') jesse | FileNotFoundError: You are missing the "plugins.py" file

instead for jesse | <class 'drivers.AngleOne.AngleOneSpot' from '/home/drivers/AngleOne/AngleOneSpot.py'> we have jesse | <module 'drivers.AngleOne.AngleOneSpot' from '/home/drivers/AngleOne/AngleOneSpot.py'> which is what causing the issue i guess

[saleh-mir]

The plugin system was broken and not needed anymore hence I removed it. I'm sorry for the trouble you went through.