Bot API 4.0

[koyuawsmbrtn]

The last commit is two months ago. Any plans making it compatible with the 4.0 version of the Bot API or are the changes so minor, that the library doesn't need to be updated?

[nickoala]

I've been thinking long and hard about this for a few days. My feeling is that I'd rather move on to other projects than spending more time on telepot. The original intent of telepot was purely educational. Over time, it has grown to greatly exceed classroom needs. I appreciate all the supports, both from the Telegram platform and from the user community. Not talented enough to pursue new interests while maintaining old ones at the same time, I think I have to declare the end of enhancements on telepot. (A notice will be put up on the front page shortly).

For one last time, thanks everyone for using telepot :blush:

[codingsett]

@nickoala please let others maintain the project if you are not able to do it since it the library helps alot of users.Thanks!

[uherting]

Dear @nickoala,

I do like telepot, use it with my Raspberry Pis and it would be sad if the project couldn't be carried on. I would like to ask you kindly to consider handing over the project to other ppl and maybe describe roughly what you did when the Bot API changed. Thank you!

In case this helps I would apply for the job as project owner till the community decides otherwise. To give you an idea: I am a senior software developer with a degree in Computer Science and among my main subjects are Linux/Unix, Perl, PHP, Python, SQL, C, C++, Raspberry Pi, ESP8266.

Regards, Uwe Herting (Germany)

[nickoala]

Thanks to @uherting. I am happy to hand off to anyone who is willing.

For the Github repo, I guess you can just fork it.

For PYPI (you have to be able to upload telepot there for people to download), I need some investigation.

For readthedoc (telepot documentations), I also need some investigation.

I can give you some pointers on how to update telepot for the latest Bot API 4.0. I can also brief you on how to use all the test scripts (those that I run before every release to make sure everything is ok).

I don't have time right now to give all those info at once. I will give them bit by bit, day by day, using this space. Stay tune.

Thanks again :blush:

[nickoala]

Let me brief on anyone who wants to make changes to accommodate the latest Bot API (4.0). I start with more trivial ones, then move on to harder ones.

---

Telegram says:

Added the field thumb to the Audio object to contain the thumbnail of the album cover to which the music file belongs.

Key phrase is "Added the field thumb to the Audio object". In telepot, the habit is to represent API objects using Python dictionaries, so a new field in an object should not affect us. However, to facilitate accessing a field using . notation, there is a way to convert a dictionary into a namedtuple. As a result, every API object has a corresponding namedtuple. When a field is added to an object, we need to make a corresponding change to that namedtuple. The file is namedtuple.py. The original definition of Audio is:

# incoming
Audio = _create_class('Audio', [
            'file_id',
            'duration',
            'performer',
            'title',
            'mime_type',
            'file_size'
        ])

With the addition of a thumb field of the type PhotoSize, it should be modified as:

# incoming
Audio = _create_class('Audio', [
            'file_id',
            'duration',
            'performer',
            'title',
            'mime_type',
            'file_size',
            _Field('thumb', constructor=PhotoSize),
        ])

The thumb field looks more complicated than others because its type is not primitive (int, float, string, etc) and we need to tell it to interpret the dictionary in that place into another namedtuple. The "constructor" in this case is just the name of the target namedtuple, and can be considered a parsing hint.

Also note the comment above the namedtuple definition. They can be:

• incoming: meaning we only receive the object from Telegram servers, but never send it out
• outgoing: meaning we only send it out, but never receive it
• or both (rarely)

This distinction is important because:

• for incoming namedtuples, we must make sure all non-primitive fields be given a "constructor" like above. Otherwise, that field would remain a dictionary, which defeats the purpose of using namedtuples. This is not needed for outgoing namedtuples because fields are supplied by user, so no parsing hint is required,

• for outgoing namedtuples, some fields have default values. For example, InlineQueryResultArticle's type field is default to article, as required by Bot API. In contrast, no field of any incoming namedtuples has default values.

---

Telegram says:

Added the field animation to the Message object. For backward compatibility, when this field is set, the document field will be also set.

Key phrase: Added the field animation to the Message object. Points to consider:

1. Message is an incoming namedtuple
2. the field animation is of the type Animation

Changes should be similar. I leave that to the reader as an exercise.

---

Telegram says:

Added support for Foursquare venues: added the new field foursquare_type to the objects Venue, InlineQueryResultVenue and InputVenueMessageContent, and the parameter foursquare_type to the sendVenue method.

Points to consider:

1. The field foursquare_type is of the type String. No parsing hint (if incoming) is needed.
2. Venue is incoming
3. InlineQueryResultVenue and InputVenueMessageContent are outgoing

I also leave the changes as an exercise.

As for the method sendVenue, I will delay the discussion until later, lumping it together with other method changes.

---

Telegram says:

Added vCard support when sharing contacts: added the field vcard to the objects Contact, InlineQueryResultContact, InputContactMessageContent and the method sendContact.

1. The field vcard is of the type String.
2. Contact is incoming
3. InlineQueryResultContact and InputContactMessageContent are outgoing

I will delay the discussion of the method sendContact similarly.

---

Telegram says:

Added support for editing the media content of messages: added the method editMessageMedia and new types InputMediaAnimation, InputMediaAudio, and InputMediaDocument.

Finally, we have to create new types of namedtuple here.

1. InputMediaAnimation, InputMediaAudio, and InputMediaDocument are all outgoing

Luckily, there are two InputMedia* siblings in existence already: InputMediaPhoto and InputMediaVideo. Find them, and use them as templates for the new members.

I will delay the discussion of the method editMessageMedia similarly.

---

That's it for today. Good luck.

[nickoala]

For @uherting, and for anyone interested to take up the maintenance of telepot, I need to know your username at:

• pypi.org
• readthedocs.org

in order to give you the ability to upload new releases and generate docs, and possibly the project ownership, in the future. So, @uherting, please let me know. Thank you.

[uherting]

Hello @nickoala, thanks a lot for the first hints! My user name for both sites is uherting like here on github.

[nickoala]

Tonight, let's see how to change methods.

---

Telegram says:

Added support for Foursquare venues: added ... the parameter foursquare_type to the sendVenue method.

Added vCard support when sharing contacts: added the field vcard to ... the method sendContact.

The two methods, sendVenue and sendContact, are adjacent in the file __init__.py, within the Bot class. Let's look at them together. Here are the original:

def sendVenue(self, chat_id, latitude, longitude, title, address,
              foursquare_id=None,
              disable_notification=None,
              reply_to_message_id=None,
              reply_markup=None):
    """ See: https://core.telegram.org/bots/api#sendvenue """
    p = _strip(locals())
    return self._api_request('sendVenue', _rectify(p))

def sendContact(self, chat_id, phone_number, first_name,
                last_name=None,
                disable_notification=None,
                reply_to_message_id=None,
                reply_markup=None):
    """ See: https://core.telegram.org/bots/api#sendcontact """
    p = _strip(locals())
    return self._api_request('sendContact', _rectify(p))

Changes are straight-forward:

def sendVenue(self, chat_id, latitude, longitude, title, address,
              foursquare_id=None,
              foursquare_type=None,
              disable_notification=None,
              reply_to_message_id=None,
              reply_markup=None):
    """ See: https://core.telegram.org/bots/api#sendvenue """
    p = _strip(locals())
    return self._api_request('sendVenue', _rectify(p))

def sendContact(self, chat_id, phone_number, first_name,
                last_name=None,
                vcard=None,
                disable_notification=None,
                reply_to_message_id=None,
                reply_markup=None):
    """ See: https://core.telegram.org/bots/api#sendcontact """
    p = _strip(locals())
    return self._api_request('sendContact', _rectify(p))

Both are optional parameters, so default to None. There's nothing to change in the method body. _strip(locals()) puts all method parameters other than self into a dict. _rectify(p) removes all None values before passing them to self._api_request().

---

Telegram says:

Added the method sendAnimation, which can be used instead of sendDocument to send animations, specifying their duration, width and height.

Luckily, we have a lot of send* methods to copy from, e.g. sendDocument, sendVideo, sendVoice, etc. I use sendVideo as an example:

def sendVideo(self, chat_id, video,
              duration=None,
              width=None,
              height=None,
              caption=None,
              parse_mode=None,
              supports_streaming=None,
              disable_notification=None,
              reply_to_message_id=None,
              reply_markup=None):
    """
    See: https://core.telegram.org/bots/api#sendvideo

    :param video: Same as ``photo`` in :meth:`telepot.Bot.sendPhoto`
    """
    p = _strip(locals(), more=['video'])
    return self._api_request_with_file('sendVideo', _rectify(p), 'video', video)

Making sendAnimation is just a matter of fixing names and matching method parameters with Telegram docs:

def sendAnimation(self, chat_id, animation,
                  duration=None,
                  width=None,
                  height=None,
                  caption=None,
                  parse_mode=None,
                  disable_notification=None,
                  reply_to_message_id=None,
                  reply_markup=None):
    """
    See: https://core.telegram.org/bots/api#sendanimation

    :param video: Same as ``photo`` in :meth:`telepot.Bot.sendPhoto`
    """
    p = _strip(locals(), more=['animation'])
    return self._api_request_with_file('sendAnimation', _rectify(p), 'animation', animation)

Sending files takes some special handling. That's why I have to _strip the animation and pass it to self._api_request_with_file() outside of the rectified dict.

Unfortunately, I wasn't aware of the "thumb" parameter, which must have been added at some earlier date. To take care of it, we have to modify _api_request_with_file() to handle one more file to be uploaded.

---

My time is up tonight. I will continue tomorrow, or the day after tomorrow.

[nickoala]

@uherting, I have added you as a maintainer at pypi.org and readthedocs.org. We'll see how to go from here.

[nickoala]

Let's continue ...

Last night was the first time I am aware of the thumb parameter. Its addition requires me to cram one more file to the call to _api_request_with_file(), whose original version is this:

def _api_request_with_file(self, method, params, file_key, file_value, **kwargs):
    if _isstring(file_value):
        params[file_key] = file_value
        return self._api_request(method, _rectify(params), **kwargs)
    else:
        files = {file_key: file_value}
        return self._api_request(method, _rectify(params), files, **kwargs)

Bot API allows file_value to be either a string (serving as a file id which refers to an existing file on Telegram servers) or a local file to be uploaded. That's why I have to distinguish between file_value being a string or not above. If it is a string, merge it to params. If not a string, I assume it's a file and pass it separately. Note that variable files is a dict.

Now, I want _api_request_with_file() to be able to handle multiple files. I would squeeze file_key and file_value into one parameter of dict:

def _api_request_with_file(self, method, params, files, **kwargs):
    params.update({
        k:v for k,v in files.items() if _isstring(v) })

    files = {
        k:v for k,v in files.items() if v is not None and not _isstring(v) }

    return self._api_request(method, _rectify(params), files, **kwargs)

I hate sprinkling trivial comments among code, so I explain here:

• params.update({ k:v for k,v in files.items() if _isstring(v) }) basically extracts all string values from files and merges them with params
• files = { k:v for k,v in files.items() if v is not None and not _isstring(v) } basically extracts all non-null non-string values from files and reassigns them to the variable files

With this change to _api_request_with_file() and the addition of the thumb parameter, sendAnimation should looks like this:

def sendAnimation(self, chat_id, animation,
                  duration=None,
                  width=None,
                  height=None,
                  thumb=None,
                  caption=None,
                  parse_mode=None,
                  disable_notification=None,
                  reply_to_message_id=None,
                  reply_markup=None):
    """
    See: https://core.telegram.org/bots/api#sendanimation

    :param video: Same as ``photo`` in :meth:`telepot.Bot.sendPhoto`
    """
    p = _strip(locals(), more=['animation', 'thumb'])
    return self._api_request_with_file('sendAnimation',
                                       _rectify(p),
                                       {'animation': animation, 'thumb': thumb})

If you don't mind being a bit venturesome, let me suggest one more change regarding the function _strip(). Instead of simply removing certain parameters from locals(), I want it to package them into a separate dict, so I can pass it directly.

Originally:

def _strip(params, more=[]):
    return {key: value for key,value in params.items() if key not in ['self']+more}

I would change it to:

def _strip(params, files=[]):
    return (
        { k:v for k,v in params.items() if k not in ['self']+files },
        { k:v for k,v in params.items() if k in files })

Then, sendAnimation becomes:

def sendAnimation(self, chat_id, animation,
                  duration=None,
                  width=None,
                  height=None,
                  thumb=None,
                  caption=None,
                  parse_mode=None,
                  disable_notification=None,
                  reply_to_message_id=None,
                  reply_markup=None):
    """
    See: https://core.telegram.org/bots/api#sendanimation

    :param video: Same as ``photo`` in :meth:`telepot.Bot.sendPhoto`
    """
    p,f = _strip(locals(), files=['animation', 'thumb'])
    return self._api_request_with_file('sendAnimation', _rectify(p), _rectify(f))

Remember to add thumb parameter to all other relevant methods. For example, sendVideo should look like:

def sendVideo(self, chat_id, video,
              duration=None,
              width=None,
              height=None,
              thumb=None,
              caption=None,
              parse_mode=None,
              supports_streaming=None,
              disable_notification=None,
              reply_to_message_id=None,
              reply_markup=None):
    """
    See: https://core.telegram.org/bots/api#sendvideo

    :param video: Same as ``photo`` in :meth:`telepot.Bot.sendPhoto`
    """
    p,f = _strip(locals(), files=['video', 'thumb'])
    return self._api_request_with_file('sendVideo', _rectify(p), _rectify(f))

Because of the change to _strip's return value, all calls to it should be re-examined and changed to:

p,f = _strip(locals())

Please also note that none of the above has been tested, although I am confident that they should not be far off.

There are more changes to be made. Let's continue some time later. Good luck.

---

UPDATE: Some bugs and shortcomings in this post's code are later found. Fixes and improvements can be seen below.

[nickoala]

After one night of sleeping, I realize some bugs and shortcomings in last night's changes. I am going to fix them today.

---

The function _rectify(dict) serves two purposes:

1. If any values is a list, dict, or tuple, _rectify flattens it into a JSON-encoded string.

2. It filters out null values.

I always use _rectify to clean/normalize a dict before passing it to _api_request_with_file(). However, in last night's changes, I made the mistake of doing _rectify(f), using _rectify to clean the files dict.

The values in f normally are either strings (file id on Telegram servers) or file objects (local files to be uploaded). But they could also be tuples, to include the filename in addition to the file object. When _rectify sees a tuple, it flattens it into a JSON-encoded string, which is wrong in this case. I only want it to filter out nulls.

Those changes I proposed yesterday now become this:

def _split(params, files=[]):
    return (
        { k:v for k,v in params.items() if k not in ['self']+files },
        { k:v for k,v in params.items() if k in files })

def _nonull(params):
    return { k:v for k,v in params.items() if v is not None }

def _rectify(params):
    #
    # no change
    #

class Bot(_BotBase):
    def _api_request(self, method, params=None, files=None, **kwargs):
        #
        # no change
        #

    def _api_request_with_file(self, method, params, files, **kwargs):
        params.update({
            k:v for k,v in files.items() if _isstring(v) })

        files = {
            k:v for k,v in files.items() if not _isstring(v) }

        return self._api_request(method, params, files, **kwargs)

    def sendMessage( ... ):
        p,f = _split(locals())
        return self._api_request('sendMessage', _rectify(p))

    def sendAnimation( ... ):
        p,f = _split(locals(), files=['animation', 'thumb'])
        return self._api_request_with_file('sendAnimation', _rectify(p), _nonull(f))

1. _strip is renamed to _split, meaning to split parameters into regular ones and file ones.

2. Add a function _nonull() whose only job is to remove null values from a dict.

3. _api_request_with_file() assumes the supplied dicts are always cleaned and normalized. No need to worry about null values inside.

4. sendMessage() demonstrates how to implement a method with no file attached.

5. sendAnimation() demonstrates how to implement a method with files attached potentially.

With the repeated applications of _rectify() and _nonull(), you may prefer to hide them in one more level of function call, or even hide them in _api_request() and _api_request_with_file(). I think it's just a matter of taste and style. I prefer the transparency, to write and see them explicitly, to remind myself that parameters should be cleaned and normalized before use.

---

Ok. That's it for today. We still haven't finished the job. I will continue some time later.

[nickoala]

Let me keep going ...

---

Now that I am satisfied with sendAnimation, there is one related place yet to modify. In the file helper.py, there is a class named Sender. The class basically wraps around a bot's send methods, with a fixed chat_id, so chat_id can be omitted when sending things. It works by creating a partial function for every send method, with a fixed chat_id of course.

We have to add sendAnimation to this class:

class Sender(object):
    def __init__(self, bot, chat_id):
        for method in ['sendMessage',
                       'forwardMessage',
                       'sendPhoto',
                       'sendAudio',
                       'sendDocument',
                       'sendSticker',
                       'sendVideo',
                       'sendAnimation',
                       'sendVoice',
                       'sendVideoNote',
                       'sendMediaGroup',
                       'sendLocation',
                       'sendVenue',
                       'sendContact',
                       'sendGame',
                       'sendChatAction',]:
            setattr(self, method, partial(getattr(bot, method), chat_id))
            # Essentially doing:
            #   self.sendMessage = partial(bot.sendMessage, chat_id)

Don't forget to update the docstring as well.

---

So far, we have only been touching the traditional version (which supports Python 2.7 and Python 3.5+). We still have an async version (Python 3.5+) to take care of.

All methods we have touched so far (sendVenue, sendContact, sendAnimation), you should make the same mirroring changes to async version.

Sender is shared between traditional and async version. Same for namedtuples. No mirroring changes are needed for them.

---

One more method to go: editMessageMedia

Luckily, there are a few editMessage to copy from. Unluckily, there is InputMedia* to deal with. The media field of InputMedia is a string, but can mean several things; it can be a file id, a URL, or "attach://" to upload a local file. That means we have to deal with potentially uploading files here.

Fortunately, the method sendMediaGroup gives an example of how to deal with InputMedia. Please read that method and its documentation.

Furthermore, I modify _split() once again. Here you go:

def _split(params, files=[], discard=[]):
    return (
        { k:v for k,v in params.items() if k not in ['self']+files+discard },
        { k:v for k,v in params.items() if k in files })

class Bot(_BotBase):
    def editMessageMedia(self, msg_identifier, media,
                         reply_markup=None):
        """
        See: https://core.telegram.org/bots/api#editmessagemedia

        :param msg_identifier: Same as ``msg_identifier`` in :meth:`telepot.Bot.editMessageText`

        :param media:
            Same as ``media`` in :meth:`telepot.Bot.sendMediaGroup`, except that here is a single
            `InputMedia <https://core.telegram.org/bots/api#inputmedia>`_ object, not an array.
        """
        p,f = _split(locals(), discard=['msg_identifier', 'media'])
        p.update(_dismantle_message_identifier(msg_identifier))

        legal_media, files_to_attach = _split_input_media_array([media])
        p['media'] = legal_media[0]

        return self._api_request('editMessageMedia', _rectify(p), files_to_attach)

1. One of three things can happen to the values passed to _split():
   • it can stay as a regular parameter, as part of the first return value
   • it is to be treated as a file parameter, as part of the second return value
   • it is to be discarded and not returned, usually because it needs special processing. After specially processed, it's usually patched back onto the regular parameters, as was the case above.

---

I think that's about it ....

There are more things in Bot API 4.0. I haven't thought through everything thoroughly, but I think I have covered the major things. @uherting, take your time. I've covered quite a bit of ground. It may take some time to digest them all. Good luck :blush:

Update on 2018-09-19: Thanks to this suggestion, I fixed the line involving legal_media above.

[nickoala]

@uherting, when you are ready, I can tell you about how to use the test scripts to make sure everything is in order.

[uherting]

@nickoala , after careful consideration I have to step back from taking over the maintance of telepot. I am sorry to say this. Unforseen tasks popped up out of the blue and as my time is limited I had to set priorities, In my pull request I added files which are a cut&paste from your description here. I also changed namedtuple.py and documented the status in the README files.

[nickoala]

@uherting, it's ok. Take it easy. You are always welcome back when you have time. I will leave telepot as it is, because I don't like to introduce untested changes. Whenever you decide to come back, we can do it again.