search

mosquito / aio-pika

AMQP 0.9 client designed for asyncio and humans.
https://aio-pika.readthedocs.org/
Apache License 2.0
1.26k stars 190 forks source link
aiormq amqp amqp-connection async-python asyncio mypy python-3 python-3-6 python3-10 python3-11 python3-7 python3-8 python3-9 rabbitmq typehints

readme

.. _documentation: https://aio-pika.readthedocs.org/ .. _adopted official RabbitMQ tutorial: https://aio-pika.readthedocs.io/en/latest/rabbitmq-tutorial/1-introduction.html

aio-pika

.. image:: https://readthedocs.org/projects/aio-pika/badge/?version=latest :target: https://aio-pika.readthedocs.org/ :alt: ReadTheDocs

.. image:: https://coveralls.io/repos/github/mosquito/aio-pika/badge.svg?branch=master :target: https://coveralls.io/github/mosquito/aio-pika :alt: Coveralls

.. image:: https://github.com/mosquito/aio-pika/workflows/tests/badge.svg :target: https://github.com/mosquito/aio-pika/actions?query=workflow%3Atests :alt: Github Actions

.. image:: https://img.shields.io/pypi/v/aio-pika.svg :target: https://pypi.python.org/pypi/aio-pika/ :alt: Latest Version

.. image:: https://img.shields.io/pypi/wheel/aio-pika.svg :target: https://pypi.python.org/pypi/aio-pika/

.. image:: https://img.shields.io/pypi/pyversions/aio-pika.svg :target: https://pypi.python.org/pypi/aio-pika/

.. image:: https://img.shields.io/pypi/l/aio-pika.svg :target: https://pypi.python.org/pypi/aio-pika/

A wrapper around aiormq_ for asyncio and humans.

Check out the examples and the tutorial in the documentation_.

If you are a newcomer to RabbitMQ, please start with the adopted official RabbitMQ tutorial_.

.. _aiormq: http://github.com/mosquito/aiormq/

.. note:: Since version 5.0.0 this library doesn't use pika as AMQP connector. Versions below 5.0.0 contains or requires pika's source code.

.. note:: The version 7.0.0 has breaking API changes, see CHANGELOG.md for migration hints.

Features

.. _Transactions: https://www.rabbitmq.com/semantics.html .. _publisher confirms: https://www.rabbitmq.com/confirms.html

Installation

.. code-block:: shell

pip install aio-pika

Usage example

Simple consumer:

.. code-block:: python

import asyncio
import aio_pika
import aio_pika.abc

async def main(loop):
    # Connecting with the given parameters is also possible.
    # aio_pika.connect_robust(host="host", login="login", password="password")
    # You can only choose one option to create a connection, url or kw-based params.
    connection = await aio_pika.connect_robust(
        "amqp://guest:guest@127.0.0.1/", loop=loop
    )

    async with connection:
        queue_name = "test_queue"

        # Creating channel
        channel: aio_pika.abc.AbstractChannel = await connection.channel()

        # Declaring queue
        queue: aio_pika.abc.AbstractQueue = await channel.declare_queue(
            queue_name,
            auto_delete=True
        )

        async with queue.iterator() as queue_iter:
            # Cancel consuming after __aexit__
            async for message in queue_iter:
                async with message.process():
                    print(message.body)

                    if queue.name in message.body.decode():
                        break

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main(loop))
    loop.close()

Simple publisher:

.. code-block:: python

import asyncio
import aio_pika
import aio_pika.abc

async def main(loop):
    # Explicit type annotation
    connection: aio_pika.RobustConnection = await aio_pika.connect_robust(
        "amqp://guest:guest@127.0.0.1/", loop=loop
    )

    routing_key = "test_queue"

    channel: aio_pika.abc.AbstractChannel = await connection.channel()

    await channel.default_exchange.publish(
        aio_pika.Message(
            body='Hello {}'.format(routing_key).encode()
        ),
        routing_key=routing_key
    )

    await connection.close()

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main(loop))
    loop.close()

Get single message example:

.. code-block:: python

import asyncio
from aio_pika import connect_robust, Message

async def main(loop):
    connection = await connect_robust(
        "amqp://guest:guest@127.0.0.1/",
        loop=loop
    )

    queue_name = "test_queue"
    routing_key = "test_queue"

    # Creating channel
    channel = await connection.channel()

    # Declaring exchange
    exchange = await channel.declare_exchange('direct', auto_delete=True)

    # Declaring queue
    queue = await channel.declare_queue(queue_name, auto_delete=True)

    # Binding queue
    await queue.bind(exchange, routing_key)

    await exchange.publish(
        Message(
            bytes('Hello', 'utf-8'),
            content_type='text/plain',
            headers={'foo': 'bar'}
        ),
        routing_key
    )

    # Receiving message
    incoming_message = await queue.get(timeout=5)

    # Confirm message
    await incoming_message.ack()

    await queue.unbind(exchange, routing_key)
    await queue.delete()
    await connection.close()

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main(loop))

There are more examples and the RabbitMQ tutorial in the documentation_.

See also

aiormq_

aiormq is a pure python AMQP client library. It is under the hood of aio-pika and might to be used when you really loving works with the protocol low level. Following examples demonstrates the user API.

Simple consumer:

.. code-block:: python

import asyncio
import aiormq

async def on_message(message):
    """
    on_message doesn't necessarily have to be defined as async.
    Here it is to show that it's possible.
    """
    print(f" [x] Received message {message!r}")
    print(f"Message body is: {message.body!r}")
    print("Before sleep!")
    await asyncio.sleep(5)   # Represents async I/O operations
    print("After sleep!")

async def main():
    # Perform connection
    connection = await aiormq.connect("amqp://guest:guest@localhost/")

    # Creating a channel
    channel = await connection.channel()

    # Declaring queue
    declare_ok = await channel.queue_declare('helo')
    consume_ok = await channel.basic_consume(
        declare_ok.queue, on_message, no_ack=True
    )

loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.run_forever()

Simple publisher:

.. code-block:: python

import asyncio
from typing import Optional

import aiormq
from aiormq.abc import DeliveredMessage

MESSAGE: Optional[DeliveredMessage] = None

async def main():
    global MESSAGE
    body = b'Hello World!'

    # Perform connection
    connection = await aiormq.connect("amqp://guest:guest@localhost//")

    # Creating a channel
    channel = await connection.channel()
    declare_ok = await channel.queue_declare("hello", auto_delete=True)

    # Sending the message
    await channel.basic_publish(body, routing_key='hello')
    print(f" [x] Sent {body}")

    MESSAGE = await channel.basic_get(declare_ok.queue)
    print(f" [x] Received message from {declare_ok.queue!r}")

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

assert MESSAGE is not None
assert MESSAGE.routing_key == "hello"
assert MESSAGE.body == b'Hello World!'

The patio and the patio-rabbitmq

PATIO is an acronym for Python Asynchronous Tasks for AsyncIO - an easily extensible library, for distributed task execution, like celery, only targeting asyncio as the main design approach.

patio-rabbitmq provides you with the ability to use RPC over RabbitMQ services with extremely simple implementation:

.. code-block:: python

from patio import Registry, ThreadPoolExecutor from patio_rabbitmq import RabbitMQBroker

rpc = Registry(project="patio-rabbitmq", auto_naming=False)

@rpc("sum") def sum(*args): return sum(args)

async def main(): async with ThreadPoolExecutor(rpc, max_workers=16) as executor: async with RabbitMQBroker( executor, amqp_url="amqp://guest:guest@localhost/", ) as broker: await broker.join()

And the caller side might be written like this:

.. code-block:: python

import asyncio
from patio import NullExecutor, Registry
from patio_rabbitmq import RabbitMQBroker

async def main():
    async with NullExecutor(Registry(project="patio-rabbitmq")) as executor:
        async with RabbitMQBroker(
            executor, amqp_url="amqp://guest:guest@localhost/",
        ) as broker:
            print(await asyncio.gather(
                *[
                    broker.call("mul", i, i, timeout=1) for i in range(10)
                 ]
            ))

FastStream_

FastStream is a powerful and easy-to-use Python library for building asynchronous services that interact with event streams..

If you need no deep dive into RabbitMQ details, you can use more high-level FastStream interfaces:

.. code-block:: python

from faststream import FastStream from faststream.rabbit import RabbitBroker

broker = RabbitBroker("amqp://guest:guest@localhost:5672/") app = FastStream(broker)

@broker.subscriber("user") async def user_created(user_id: int): assert isinstance(user_id, int) return f"user-{user_id}: created"

@app.after_startup async def pub_smth(): assert ( await broker.publish(1, "user", rpc=True) ) == "user-1: created"

Also, FastStream validates messages by pydantic, generates your project AsyncAPI spec, supports In-Memory testing, RPC calls, and more.

In fact, it is a high-level wrapper on top of aio-pika, so you can use both of these libraries' advantages at the same time.

python-socketio_

Socket.IO_ is a transport protocol that enables real-time bidirectional event-based communication between clients (typically, though not always, web browsers) and a server. This package provides Python implementations of both, each with standard and asyncio variants.

Also this package is suitable for building messaging services over RabbitMQ via aio-pika adapter:

.. code-block:: python

import socketio from aiohttp import web

sio = socketio.AsyncServer(client_manager=socketio.AsyncAioPikaManager()) app = web.Application() sio.attach(app)

@sio.event async def chat_message(sid, data): print("message ", data)

if name == 'main': web.run_app(app)

And a client is able to call chat_message the following way:

.. code-block:: python

import asyncio import socketio

sio = socketio.AsyncClient()

async def main(): await sio.connect('http://localhost:8080') await sio.emit('chat_message', {'response': 'my response'})

if name == 'main': asyncio.run(main())

The taskiq and the taskiq-aio-pika

Taskiq is an asynchronous distributed task queue for python. The project takes inspiration from big projects such as Celery and Dramatiq. But taskiq can send and run both the sync and async functions.

The library provides you with aio-pika broker for running tasks too.

.. code-block:: python

from taskiq_aio_pika import AioPikaBroker

broker = AioPikaBroker()

@broker.task async def test() -> None: print("nothing")

async def main(): await broker.startup() await test.kiq()

Rasa_

With over 25 million downloads, Rasa Open Source is the most popular open source framework for building chat and voice-based AI assistants.

With Rasa, you can build contextual assistants on:

Your own custom conversational channels or voice assistants as:

Rasa helps you build contextual assistants capable of having layered conversations with lots of back-and-forth. In order for a human to have a meaningful exchange with a contextual assistant, the assistant needs to be able to use context to build on things that were previously discussed – Rasa enables you to build assistants that can do this in a scalable way.

And it also uses aio-pika to interact with RabbitMQ deep inside!

Versioning

This software follows Semantic Versioning_

For contributors

Setting up development environment

Clone the project:

.. code-block:: shell

git clone https://github.com/mosquito/aio-pika.git
cd aio-pika

Create a new virtualenv for aio-pika_:

.. code-block:: shell

python3 -m venv env
source env/bin/activate

Install all requirements for aio-pika_:

.. code-block:: shell

pip install -e '.[develop]'

Running Tests

NOTE: In order to run the tests locally you need to run a RabbitMQ instance with default user/password (guest/guest) and port (5672).

The Makefile provides a command to run an appropriate RabbitMQ Docker image:

.. code-block:: bash

make rabbitmq

To test just run:

.. code-block:: bash

make test

Editing Documentation

To iterate quickly on the documentation live in your browser, try:

.. code-block:: bash

nox -s docs -- serve

Creating Pull Requests

Please feel free to create pull requests, but you should describe your use cases and add some examples.

Changes should follow a few simple rules:

.. _"thank's to" section: https://github.com/mosquito/aio-pika/blob/master/docs/source/index.rst#thanks-for-contributing .. _Semantic Versioning: http://semver.org/ .. _aio-pika: https://github.com/mosquito/aio-pika/ .. _faststream: https://github.com/airtai/faststream .. _patio: https://github.com/patio-python/patio .. _patio-rabbitmq: https://github.com/patio-python/patio-rabbitmq .. _Socket.IO: https://socket.io/ .. _python-socketio: https://python-socketio.readthedocs.io/en/latest/intro.html .. _taskiq: https://github.com/taskiq-python/taskiq .. _taskiq-aio-pika: https://github.com/taskiq-python/taskiq-aio-pika .. _Rasa: https://rasa.com/docs/rasa/