Bug using Kathara.get_instance().exec() with Kubernetes and stream=True with Kathara 3.7.3

[BMG-DYNAMIT]

Operating System

Ubuntu 22.04

Kathará Version

3.7.3

Bug Description

Hi,

I am using the Python-API to create a Network Scenario. I am using Kathara with Kubernetes and Megalos with Kathara version 3.7.3

I am using Flannel CNI.

If I call Kathara.get_instance().exec(machine_name=machine_name, command=command, lab_hash=lab.hash, stream=True) on a Machine it just returns a Generator[None, None, None], so print(list(output)) ist just contains None.

If I use stream=False I get a correct Tuple as return.

With Kathara 3.7.1 it worked correctly and returned: Generator[Tuple[bytes, bytes], None, None]

Steps To Reproduce

Execute a command with Kathara.get_instance().exec(), stream=True, using Kubernetes and Kathara version 3.7.3

Expected Behavior

Kathara.get_instance().exec() with stream=True should return: Generator[Tuple[bytes, bytes], None, None]

Check Command Output

No response

[BMG-DYNAMIT]

The command I used to reproduce this was: "ping X -c 4"

[Skazza94]

Hi @BMG-DYNAMIT, thanks for the issue.

We indeed changed the exec API in 3.7.3 (to both support stream/non-stream execs) and we might broke something. Could you please provide a code snippet to reproduce the problem?

Thanks, Mariano.

[BMG-DYNAMIT]

HI,

This is a possible code snippet. Notice that you need to set default manager type to kubernetes first with kathara settings

import logging
import os.path

from Kathara.manager.Kathara import Kathara
from Kathara.model.Lab import Lab
from Kathara.setting.Setting import Setting

# Get manager-type kubernetes
Setting.get_instance().load_from_disk()

logger = logging.getLogger("Kathara")
logger.setLevel(logging.INFO)

logger.info("Creating Lab BGP Announcement...")
lab = Lab("BGP Announcement")

logger.info("Creating router1...")
# Create router1 with image "kathara/frr"
router1 = lab.new_machine("router1", **{"image": "kathara/frr"})

# Create and connect router1 interfaces
lab.connect_machine_to_link(router1.name, "A")
lab.connect_machine_to_link(router1.name, "B")

logger.info("Creating router2...")
# Create router2 with image "kathara/frr"
router2 = lab.new_machine("router2", **{"image": "kathara/frr"})

# Create and connect router1 interfaces
lab.connect_machine_to_link(router2.name, "A")
lab.connect_machine_to_link(router2.name, "C")

logger.info("Deploying BGP Announcement lab...")
Kathara.get_instance().deploy_lab(lab)

command="echo Hello"

print("\n")
print("This is not working:\n")
#Not Working, Returns a list with many None. Should the Nones not be ""?
output = Kathara.get_instance().exec(machine_name=router1.name,lab_hash=lab.hash, command=command, stream=True)
print(list(output))

print("\n")
print("This is not working:\n")
# Not Working, Returns Type None if there is no stdout.  Should return "" instead
output = Kathara.get_instance().exec(machine_name=router1.name,lab_hash=lab.hash, command=command, stream=True)

for stdout, stderr in output:
    print(stdout)

#Working
print("\n")
print("This is working:\n")
outputTuple = Kathara.get_instance().exec(machine_name=router1.name,lab_hash=lab.hash, command=command, stream=False)
print(outputTuple[0].decode())

logger.info("Undeploying BGP Announcement lab...")
Kathara.get_instance().undeploy_lab(lab_name=lab.name)

The Generator Prints NoneType instead "" to stdout bytes if there is no output

[Skazza94]

Hi @BMG-DYNAMIT, I tried your code and I think you are not using the generator returned by the stream properly. If you unroll it as a list, you would end up with a lot of (None, None) tuples (that are correct) since the Pod is not generating any output. But if you search inside the list, you will find the actual output:

The correct way to use the generator is to add an infinite loop with a try-except on the StopIteration exception. Inside the try body, you ask for the next element of the stream and print/store the byte-stream in another variable.

So, this is an example of how to implement this:

command="echo Hello"
output = Kathara.get_instance().exec(machine_name=router1.name,lab_hash=lab.hash, command=command, stream=True)
all_stdout = b''
all_stderr = b''
while True:
    try:
        (stdout, stderr) = next(output)
        if stdout:
            all_stdout += stdout
        if stderr:
            all_stderr += stderr
    except StopIteration:
        break

print(all_stdout.decode('utf-8'), all_stderr.decode('utf-8'))

And now you can see the proper output:

Indeed, this is similar to the stream=False behaviour, since you are accumulating the output in the all_stdout variable. To have a "live" output, for example during the ping:

# At the beginning
import sys

# ...

command="ping -c 4 127.0.0.1"
output = Kathara.get_instance().exec(machine_name=router1.name,lab_hash=lab.hash, command=command, stream=True)
while True:
    try:
        (stdout, stderr) = next(output)
        if stdout:
            sys.stdout.write(stdout.decode('utf-8'))
        if stderr:
            sys.stderr.write(stderr.decode('utf-8'))
    except StopIteration:
        break

Result:

As another quick note, always wrap the main file code with if __name__ == '__main__':, Kathará is multithreaded so it requires to know if a module is the main or not, otherwise you could encounter some undesired exceptions:

import logging
import os.path
import time
import sys

from Kathara.manager.Kathara import Kathara
from Kathara.model.Lab import Lab
from Kathara.setting.Setting import Setting

if __name__ == '__main__':
    # Get manager-type kubernetes
    Setting.get_instance().load_from_dict({'manager_type': 'kubernetes', "debug_level": "DEBUG"})

   # ... Rest of the code ...

Cheers, Mariano.

[BMG-DYNAMIT]

Hi,

thank you very much for your help and your time!

You have helped me a lot. I think this can be closed now.

Cheers, Moritz.