asyncio support

[JonasKs]

Hi!

I was porting another ICMP package over to be async when I found this. I would love to create a async def ping which is not blocking and is awaitable, using the current codebase but running the sockets in executors, and using a randint instead of PID. Let me know if you are open to accept PRs.

The alternative is for me to create my own package that basically only utilizes yours, but has it's own async def ping function.

[ValentinBELYN]

Hi @JonasKs 👋

Thanks for the suggestion. It is an excellent idea! I never had the time to discover in detail the mechanism offered by asyncio, so it will be a good opportunity.

However, icmplib 2.0 is currently nearing the end of its development and is already introducing major software architecture changes to provide, in particular, an execution mode for non-privileged users. So, this will be certainly for the next major version.

In the meantime, as you suggested, you are welcome to create your own package with icmplib as a dependency. This could even be used later to integrate your work directly in this library.

Pay attention to the identifier which must not be generated randomly. There is a probability that the same number will be drawn at random.

[JonasKs]

Hi :)

However, icmplib 2.0 is currently nearing the end of its development and is already introducing major software architecture changes to provide, in particular, an execution mode for non-privileged users. So, this will be certainly for the next major version.

No problem, I understand that. :) Library is very clean and structured and provides beautiful outputs, so I'm happy I found it.

In the meantime, as you suggested, you are welcome to create your own package with icmplib as a dependency. This could even be used later to integrate your work directly in this library.

Perfect. I'll ping you and write in this issue when it's done, if you would like to go back and have a look or look into integrating it into this package (or want me to do so for you).

Pay attention to the identifier which must not be generated randomly. There is a probability that the same number will be drawn at random.

Noted. I'll use the asyncio's task_id() or similar. Do you know if it has to be an int, or if I can use uuid.uuid4().hex?

[ValentinBELYN]

No problem, I understand that. :) Library is very clean and structured and provides beautiful outputs, so I'm happy I found it.

Thank you so much! I strive to create a stable library compatible on many systems and taking into account the remarks and suggestions. It's a lot of work but it's worth it, especially when you have great feedback like yours 😄

Perfect. I'll ping you and write in this issue when it's done, if you would like to go back and have a look or look into integrating it into this package (or want me to do so for you).

👍

Noted. I'll use the asyncio's task_id() or similar. Do you know if it has to be an int, or if I can use uuid.uuid4().hex

The identifier corresponds to the ID field of the ICMP header which is an unsigned short (16-bit encoded). Therefore, you can use an integer between 0 and 65535. If you exceed the limit, the identifier is automatically reset by the library.

[JonasKs]

Thanks! I’ll see what I can do to build some logic around that.

Have a good week!

[ValentinBELYN]

Thank you! Have a good week too!

[JonasKs]

In your upcoming release, would you mind renaming your variables a bit? socket, type and similar variables shadows builtin functions, which is a bad practice and makes most editors complain. Other good names to use could be sock, socket_type and so on.

It's not so easy to check which type the type variable has, for instance. Try print(type(type)) ;)

[ValentinBELYN]

Hi @JonasKs,

I saw your message but I didn't have time to answer it.

I agree with you on this subject. Regarding the socket variable name, there was no variable shadowing. However, I made the modification to avoid confusion with the module of this name. Concerning the id and type variables, it is a choice I made initially so as not to lengthen the name of the variables. It is also more natural for developers who use the library.

request.id is more readable and understandable than request.request_id or request.icmp_id given that the identifier refers directly to the request. Obviously, everyone will have their opinion on it. I think the functions and methods are small enough not to be a problem when debugging.

Moreover, for the next major version, I want the transition to require relatively few modifications in the programs that use this library. I will think about it anyway in the future :)

[ValentinBELYN]

Hi! I reopen this issue to keep this idea in mind 😃

[JonasKs]

Hi!

I did start on writing an async implementation(and iirc it works as it should) but I haven’t had to finish it yet. https://github.com/JonasKs/aioicmp

[bdraco]

❤️ This would be amazing for Home Assistant as the executor can get overloaded when the user has a large number of ping binary sensors

[ValentinBELYN]

@bdraco The next major version of icmplib is almost ready. I still have to push the various changes made to the library.

And yes the next version of icmplib will be async and easier to use (you will no longer need to worry about the ID of ICMP requests)!

I hope to finalize this version by the end of this month.

[JonasKs]

Awesome! Do you have a branch going? I’d love to see the implementation.

[ValentinBELYN]

Hmmm... it's a secret! I can just say that a preview will be available tomorrow 😜

[JonasKs]

Hehe, alright. Curious to whether the implementation of asyncio is similar to mine or not. I can most definitely wait until tomorrow though 😉

[bdraco]

Exciting stuff 🥳

[ValentinBELYN]

Hi 👋

The preview of icmplib 3.0 is online! Feel free to report bugs if you encounter one. Your suggestions are also welcome 😉

To try it:

# Uninstall icmplib
pip3 uninstall icmplib

# Download and extract this repository
wget -qO- https://github.com/ValentinBELYN/icmplib/archive/main.tar.gz | tar -xzf -
cd icmplib-main

# Install the version under development
python3 setup.py install

There are quite a few changes. I invite you to look at the docstrings for the parameters to pass to the different functions.

The README will be updated soon and a changelog will be made available at the same time as the update. Finally, Python 3.7+ is now required.

Hope you like this new version!

[ValentinBELYN]

Since the documentation is not yet up to date on GitHub, here are the new parameters and exceptions for the built-in functions:

ping and async_ping

def ping(address, count=4, interval=1, timeout=2, id=None, source=None,
        family=None, privileged=True, **kwargs):

    Send ICMP Echo Request packets to a network host.

    :type address: str
    :param address: The IP address, hostname or FQDN of the host to
        which messages should be sent. For deterministic behavior,
        prefer to use an IP address.

    :type count: int, optional
    :param count: The number of ping to perform. Default to 4.

    :type interval: int or float, optional
    :param interval: The interval in seconds between sending each
        packet. Default to 1.

    :type timeout: int or float, optional
    :param timeout: The maximum waiting time for receiving a reply in
        seconds. Default to 2.

    :type id: int, optional
    :param id: The identifier of ICMP requests. Used to match the
        responses with requests. In practice, a unique identifier should
        be used for every ping process. On Linux, this identifier is
        ignored when the `privileged` parameter is disabled. The library
        handles this identifier itself by default.

    :type source: str, optional
    :param source: The IP address from which you want to send packets.
        By default, the interface is automatically chosen according to
        the specified destination.

    :type family: int, optional
    :param family: The address family if a hostname or FQDN is specified.
        Can be set to `4` for IPv4 or `6` for IPv6 addresses. By default,
        this function searches for IPv4 addresses first before searching
        for IPv6 addresses.

    :type privileged: bool, optional
    :param privileged: When this option is enabled, this library fully
        manages the exchanges and the structure of ICMP packets.
        Disable this option if you want to use this function without
        root privileges and let the kernel handle ICMP headers.
        Default to True.
        Only available on Unix systems. Ignored on Windows.

    Advanced (**kwags):

    :type payload: bytes, optional
    :param payload: The payload content in bytes. A random payload is
        used by default.

    :type payload_size: int, optional
    :param payload_size: The payload size. Ignored when the `payload`
        parameter is set. Default to 56.

    :type traffic_class: int, optional
    :param traffic_class: The traffic class of ICMP packets.
        Provides a defined level of service to packets by setting the
        DS Field (formerly TOS) or the Traffic Class field of IP
        headers. Packets are delivered with the minimum priority by
        default (Best-effort delivery).
        Intermediate routers must be able to support this feature.
        Only available on Unix systems. Ignored on Windows.

    :rtype: Host
    :returns: A `Host` object containing statistics about the desired
        destination.

    :raises NameLookupError: If you pass a hostname or FQDN in
        parameters and it does not exist or cannot be resolved.
    :raises SocketPermissionError: If the privileges are insufficient to
        create the socket.
    :raises SocketAddressError: If the source address cannot be assigned
        to the socket.
    :raises ICMPSocketError: If another error occurs. See the
        `ICMPv4Socket` or `ICMPv6Socket` class for details.

multiping and async_multiping

def multiping(addresses, count=2, interval=0.5, timeout=2,
        concurrent_tasks=50, source=None, family=None, privileged=True,
        **kwargs):

    Send ICMP Echo Request packets to several network hosts.

    :type addresses: list[str]
    :param addresses: The IP addresses of the hosts to which messages
        should be sent. Hostnames and FQDNs are allowed but not
        recommended. You can easily retrieve their IP address by calling
        the built-in `resolve` function.

    :type count: int, optional
    :param count: The number of ping to perform per address.
        Default to 2.

    :type interval: int or float, optional
    :param interval: The interval in seconds between sending each
        packet. Default to 0.5.

    :type timeout: int or float, optional
    :param timeout: The maximum waiting time for receiving a reply in
        seconds. Default to 2.

    :type concurrent_tasks: int, optional
    :param concurrent_tasks: The maximum number of concurrent tasks to
        speed up processing. This value cannot exceed the maximum number
        of file descriptors configured on the operating system.
        Default to 50.

    :type source: str, optional
    :param source: The IP address from which you want to send packets.
        By default, the interface is automatically chosen according to
        the specified destinations. This parameter should not be used if
        you are passing both IPv4 and IPv6 addresses to this function.

    :type family: int, optional
    :param family: The address family if a hostname or FQDN is specified.
        Can be set to `4` for IPv4 or `6` for IPv6 addresses. By default,
        this function searches for IPv4 addresses first before searching
        for IPv6 addresses.

    :type privileged: bool, optional
    :param privileged: When this option is enabled, this library fully
        manages the exchanges and the structure of ICMP packets.
        Disable this option if you want to use this function without
        root privileges and let the kernel handle ICMP headers.
        Default to True.
        Only available on Unix systems. Ignored on Windows.

    Advanced (**kwags):

    :type payload: bytes, optional
    :param payload: The payload content in bytes. A random payload is
        used by default.

    :type payload_size: int, optional
    :param payload_size: The payload size. Ignored when the `payload`
        parameter is set. Default to 56.

    :type traffic_class: int, optional
    :param traffic_class: The traffic class of ICMP packets.
        Provides a defined level of service to packets by setting the
        DS Field (formerly TOS) or the Traffic Class field of IP
        headers. Packets are delivered with the minimum priority by
        default (Best-effort delivery).

        Intermediate routers must be able to support this feature.
        Only available on Unix systems. Ignored on Windows.

    :rtype: list[Host]
    :returns: A list of `Host` objects containing statistics about the
        desired destinations. The list is sorted in the same order as
        the addresses passed in parameters.

    :raises NameLookupError: If you pass a hostname or FQDN in
        parameters and it does not exist or cannot be resolved.
    :raises SocketPermissionError: If the privileges are insufficient to
        create the socket.
    :raises SocketAddressError: If the source address cannot be assigned
        to the socket.
    :raises ICMPSocketError: If another error occurs. See the
        `ICMPv4Socket` or `ICMPv6Socket` class for details.

traceroute

def traceroute(address, count=2, interval=0.05, timeout=2, first_hop=1,
        max_hops=30, fast=False, id=None, source=None, family=None,
        **kwargs):

    Determine the route to a destination host.

    The Internet is a large and complex aggregation of network hardware,
    connected together by gateways. Tracking the route one's packets
    follow can be difficult. This function uses the IP protocol time to
    live field and attempts to elicit an ICMP Time Exceeded response
    from each gateway along the path to some host.

    This function requires root privileges to run.

    :type address: str
    :param address: The IP address, hostname or FQDN of the host to
        reach. For deterministic behavior, prefer to use an IP address.

    :type count: int, optional
    :param count: The number of ping to perform per hop. Default to 2.

    :type interval: int or float, optional
    :param interval: The interval in seconds between sending each
        packet. Default to 0.05.

    :type timeout: int or float, optional
    :param timeout: The maximum waiting time for receiving a reply in
        seconds. Default to 2.

    :type first_hop: int, optional
    :param first_hop: The initial time to live value used in outgoing
        probe packets. Default to 1.

    :type max_hops: int, optional
    :param max_hops: The maximum time to live (max number of hops) used
        in outgoing probe packets. Default to 30.

    :type fast: bool, optional
    :param fast: When this option is enabled and an intermediate router
        has been reached, skip to the next hop rather than perform
        additional requests. The `count` parameter then becomes the
        maximum number of requests in the event of no response.
        Default to False.

    :type id: int, optional
    :param id: The identifier of ICMP requests. Used to match the
        responses with requests. In practice, a unique identifier should
        be used for every traceroute process. The library handles this
        identifier itself by default.

    :type source: str, optional
    :param source: The IP address from which you want to send packets.
        By default, the interface is automatically chosen according to
        the specified destination.

    :type family: int, optional
    :param family: The address family if a hostname or FQDN is specified.
        Can be set to `4` for IPv4 or `6` for IPv6 addresses. By default,
        this function searches for IPv4 addresses first before searching
        for IPv6 addresses.

    Advanced (**kwags):

    :type payload: bytes, optional
    :param payload: The payload content in bytes. A random payload is
        used by default.

    :type payload_size: int, optional
    :param payload_size: The payload size. Ignored when the `payload`
        parameter is set. Default to 56.

    :type traffic_class: int, optional
    :param traffic_class: The traffic class of ICMP packets.
        Provides a defined level of service to packets by setting the
        DS Field (formerly TOS) or the Traffic Class field of IP
        headers. Packets are delivered with the minimum priority by
        default (Best-effort delivery).
        Intermediate routers must be able to support this feature.
        Only available on Unix systems. Ignored on Windows.

    :rtype: list[Hop]
    :returns: A list of `Hop` objects representing the route to the
        desired destination. The list is sorted in ascending order
        according to the distance, in terms of hops, that separates the
        remote host from the current machine. Gateways that do not
        respond to requests are not added to this list.

    :raises NameLookupError: If you pass a hostname or FQDN in
        parameters and it does not exist or cannot be resolved.
    :raises SocketPermissionError: If the privileges are insufficient to
        create the socket.
    :raises SocketAddressError: If the source address cannot be assigned
        to the socket.
    :raises ICMPSocketError: If another error occurs. See the
        `ICMPv4Socket` or `ICMPv6Socket` class for details.

resolve and async_resolve

def resolve(name, family=None):

    Resolve a hostname or FQDN to an IP address. Depending on the name
    specified in parameters, several IP addresses may be returned.
    This function relies on the DNS name server configured on your
    operating system.

    :type name: str
    :param name: A hostname or a Fully Qualified Domain Name (FQDN).

    :type family: int, optional
    :param family: The address family. Can be set to `4` for IPv4 or `6`
        for IPv6 addresses. By default, this function searches for IPv4
        addresses first for compatibility reasons (A DNS lookup) before
        searching for IPv6 addresses (AAAA DNS lookup).

    :rtype: list[str]
    :returns: A list of IP addresses associated with the name passed as
        a parameter.

    :raises NameLookupError: If the requested name does not exist or
        cannot be resolved.

[ValentinBELYN]

The Host and Hop classes have also changed. You can now retrieve all the round-trip times (rtts property) and calculate the jitter (jitter property)!

[bdraco]

Thanks. Great work.

When do you think you will be ready to publish?

[JonasKs]

Looks awesome! I’ll stress test it this week.

[ValentinBELYN]

Thanks 😄

@bdraco I hope to release it next week. This will give me the time to do my tests, finalize various things and update the documentation.

@JonasKs With pleasure! Thank you for your participation 👍

[bdraco]

On the surface everything seems to be working as expected in https://github.com/home-assistant/core/pull/50808

[ValentinBELYN]

I added some comments! home-assistant/core#50808

[bdraco]

Thanks. I haven't run into any issues in testing.

[ValentinBELYN]

Thanks for your feedback! I found some bugs under macOS (bug also present in icmplib 2.1.1) and Windows. It is now fixed.

Normally, I publish the final version this weekend, or at the latest at the beginning of next week.

[bdraco]

Thanks for the update. Looks like we will miss the cutoff for Home Assistant release (Wed morning), but there is always next month.

[ValentinBELYN]

Sorry, I prefer to have everything ready before publishing this new version 😉

[bdraco]

No worries. Better to get it right the first time 👍

[ValentinBELYN]

Hello everyone!

The release date of icmplib 3 has been defined! 🎉 It will be released on June 1st.

The new documentation is now online: https://github.com/ValentinBELYN/icmplib/tree/main/docs

[ValentinBELYN]

icmplib 3.0 is available for download! :rocket:

Thank you very much for your advice, help and feedback!

@JonasKs I owe you a lot for this version of icmplib. Your implementation and your comments have greatly simplified the integration of asyncio within the library. A huge thank you!

@bdraco Can't wait to see icmplib 3.0 in Home Assistant!

[JonasKs]

Congratulations! It's a very clean implementation. Happy to be a part of it😊

[kk71]

congrats on the release of new version!

I quickly overviewed the doc, it seems ping and multi-ping is asyncio-ready, so when will traceroute to be turned into asyncio?

thx!

[ValentinBELYN]

Hi @kk71,

I would love it but I have the impression that there is no alternative to recvfrom at the socket level in the asyncio library.

The sock_recv method currently used for the asynchronous implementation does not return the source IP address of ICMP responses.

As much as for the ping and multiping functions this is not a problem, but for the traceroute where we need to retrieve the IP addresses of the intermediate gateways, this is problematic.

[JonasKs]

An alternative is a less beautiful method, using run_in_executor as proposed in my initial design.

https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor

[ValentinBELYN]

Thank you for your reply. However, I prefer to avoid this solution which relies on the use of threads.

Moreover, it does not require any particular implementation. It is therefore not necessary to integrate it.

A user who wants to use the traceroute function asynchronously can add in its code:

import asyncio
from functools import partial
from icmplib import traceroute

async def async_traceroute(*args, **kwargs):
    loop = asyncio.get_running_loop()
    return await loop.run_in_executor(None, traceroute, partial(*args, **kwargs))

hops = asyncio.run(async_traceroute('1.1.1.1'))

for hop in hops:
    print(hop)

I haven't tested this code but it should work for those who want an asynchronous traceroute function.

[JonasKs]

I think we can close this issue? Everything works as expected for me. 🎉

[ValentinBELYN]

Yes, we can close this issue. Thanks!