Asyncssh clashing with IB TWS API python script

[anarchy89]

I have been using your package for months now with no issues.

I run multiple scripts on multiple servers with no issues except 1 has been plaguing me recently and I can't seem to figure it out.

I run IBKR tws api https://interactivebrokers.github.io/tws-api/introduction.html in the cloud using your package to execute the commands remotely.

I run a simple script like this using your tool,

commands = [f'/home/{REMOTEUSERNAME}/scripts/trade.sh'] 
asyncio.run(run_multiple_clients(hosts=instance_ips, commands=commands, username=REMOTEUSERNAME, client_keys=[SSH_KEY_PATH], known_hosts=None))

So if I connect to my server directly and run the script located above, it works fine, the trade go through.

However, if I run it using the command above, the script runs though but no trades are placed. I believe it's because both programs are asynchronous in nature and they are clashing with each other. Do you have any ideas what I could do to resolve this issue?

[ronf]

If I understand right, trade.sh and any commands inside it would all be running on the remote machine, while AsyncSSH is running on the local machine. So, even if both make asynchronous calls, I don't think that'll be an issue. That's generally only an issue if you try to use multiple async libraries in a single Python interpreter, meaning the TWS APIs would have to be getting called on the local machine.

Without more information about what's in trade.sh, there's not much I can suggest here. However, one difference between running the command by hand vs. what you're doing here is that you'd have a TTY allocated to your process when trade.sh is run by hand, while AsyncSSH (and OpenSSH) would not allocate a TTY by default when exec'ing a command like this. You can try adding term_type='ansi' to the arguments to asyncio.run() to see if that makes any difference. Another thought is that there could be a difference in your PATH or other environment variables on the remote system, perhaps causing the TWS API to fail.

Here are some other general thoughts:

• If you use ssh to connect to the remote machine and exec trade.sh, does it work? That'd be closer to what you're doing here with AsyncSSH.
• When you run with AsyncSSH, do you get an exit code in the response to asyncio.run()? Do you get any output on stdout or stderr?
• Have you tried enabling debug logging to see if anything unusual shows up? You can do that by adding the following before your call to asyncio.run():

    logging.basicConfig()
    asyncssh.set_log_level('DEBUG')
    asyncssh.set_debug_level(2)

[anarchy89]

Hey @ronf so, I have also tried doing this where I run a script called trade.sh using asyncssh. Within trade.sh, it looks like this, screen -dmS "trade" "python3 trade.py", so I am assigning it a screen when it runs. It still does not work as intended?

Does this mean that I am creating a TTY for it or no?

I think it could be a path problem as well, any advice for this?

I will try you ansi and debugging advice now and see what happens.

I tried, term_type but i got this error,

asyncio.run(run_multiple_clients(hosts=instance_ips, commands=commands, username=REMOTEUSERNAME, client_keys=[SSH_KEY_PATH], known_hosts=None), term_type='ansi')
TypeError: run() got an unexpected keyword argument 'term_type'

This is the current code I am using,

async def run_client(host, commands, **kwargs):
    output = ''

    for retries in range(60):
        try:
            conn = await asyncssh.connect(host, **kwargs)
            break
        except (OSError, asyncssh.Error):
            await asyncio.sleep(10)
    else:
        return f'Connect to {host} failed\n'

    async with conn:
        for i, command in enumerate(commands):
            result = await conn.run(command)

            if result.exit_status != 0:
                output += f'=== Command {i+1}: {command} failed ===\n' + result.stderr
            else:
                output += f'=== Command {i+1}: {command} succeeded ===\n' + result.stdout

        return output

async def run_multiple_clients(hosts, commands, **kwargs):
    tasks = (run_client(host, commands, **kwargs) for host in hosts)
    results = await asyncio.gather(*tasks, return_exceptions=True)

    for i, result in enumerate(results):
        if isinstance(result, Exception):
            print(f'Connection {i+1} to host {hosts[i]} failed: ' + str(result))
        else:
            print(f'Connection {i+1} to host {hosts[i]}:\n' + result, end='')

        print(75*'-')

Also, here is my import code for trade.py

TWS_SOURCE = f'/home/{USERNAME}/Documents/{TRADE_DIR}/scripts/trade/twsapi_macunix/IBJts/source/pythonclient'
TWS_SAMPLES = f'/home/{USERNAME}/Documents/{TRADE_DIR}/scripts/trade/twsapi_macunix/IBJts/samples/Python'
import sys
import os
try:
    script_dir = os.path.dirname(os.path.abspath(__file__))
    parent_dir = os.path.abspath(os.path.join(script_dir,'..'))
    insert_path = os.path.join(parent_dir, 'misc')
    sys.path.insert(0, insert_path)

except NameError: # for jupyter
    parent_dir = os.path.abspath('..')
    insert_path = os.path.join(parent_dir, 'misc')
    sys.path.insert(0, insert_path)

from global_variables import (TWS_SOURCE, TWS_SAMPLES)

sys.path.insert(0, TWS_SOURCE)
sys.path.insert(0, TWS_SAMPLES)

[ronf]

Regarding the terminal type argument, I misspoke -- that needs to be an argument on conn.run(), not asyncio.run(). More specifically, this would go in run_client(), either directly or by passing in **kwargs into the conn.run() call there, so you could pass whatever arguments you like from run_multiple_clients.

As for the use of 'screen', that's not something I have done with AsyncSSH. Normally, screen requires a tty to work properly, but maybe with "-dm" that won't be the case, since you'd be starting a new screen session in detached mode. Either way, screen itself should create a TTY for the trade.py script, so if that need a TTY you'll probably end up with one even if SSH doesn't allocate one.

I'm not sure what requirements in terms of path or environment variables that the TWS APIs require, but you could try printing out the environment at the start of the script and comparing that output between when you run with AsyncSSH and when you run with OpenSSH or from an interactive shell directly on the remote machine.

I still think your best bet is to check the output in stdout and stderr to see if anything shows up there, and then to try adding more output in the script at key points to see whether you get to those points or not. If you aren't getting any output, trying the debug logging would tell you if you actually succeeded in authenticating or not, and whether you are starting any remote sessions. It'll also confirm the command you are asking the remote SSH server to run.

[anarchy89]

@ronf im gonna start debugging the code using print statements etc. I have a feeling it's the import paths. Any idea how those are done? Should I hardcode absolute paths?

Any idea how I can print the environment?

[ronf]

In trade.sh, you could print the environment by just running the "env" command. In trade.py, you could do something like:

    import os
    print(os.environ)

The output isn't as pretty here as it all comes out as one long line, but you can fix that with:

    import os
    print('\n'.join(f'{k}: {v}' for k, v in os.environ.items()))

In terms of paths, hardcoding them in shell scripts so you don't rely on the environment is a good practice. For Python imports, though, I'm not sure you can use absolute paths (at least not with the regular "import"), but setting the right absolute paths into sys.path should work.

[ronf]

Closing due to inactivity. Feel free to reopen this or open a new issue if you need anything else.