dividend calender: aditional filter request parameter

[AureliusMarcusHu]

Dear Chavithra,

I have made a dividend calender dictionary for current year, started six months in the past and ended six monts in te future. The dictionary contains 6179 items. deGiro send maximum a list of 100 items at once. As you can see on the console output, there are several days the daily dividend list cross these 100 items. This means you have not all the daily dividends on the dates showing on the console output. On this way it is impossible to find the dividend where you looking for.

Therefore, an additional parameter as a filter would be highly desirable when sending the dividend request. In my opinion, sending the request with the currency parameter as additional filter parameter, is a good option. I have tried this but currently the module does not recognize this option.

Perhaps there is an existing option to filter the dividend request. But how to do?

Console output

Dividend dictionary dvd_calender.json

[Chavithra]

Hello @AureliusMarcusHu,

PAGINATION You can retrieve more than 100 items.

For that you need to use pagination.

There were similar questions about ProductSearch, maybe that can help: https://github.com/Chavithra/degiro-connector/issues/27#issuecomment-965817294 https://github.com/Chavithra/degiro-connector/issues/121#issuecomment-1873441676

CURRENCY Which method and parameters are you using?

If it is get_agenda, extra parameters are allowed in the AgendaRequest at least in the latest versions.

This should be allowed:

AgendaRequest(
        calendar_type=CalendarType.DIVIDEND_CALENDAR,
        currencies="USD",
        ...
)

Thanks

[AureliusMarcusHu]

Dear Chavithra,

Thanks for the fast reply. Sorry I tried request filtering by currency with parameter "currency" instead of "currencies". With currencies as parameter it works fine. Can currencies parameter be a tuple or a list ?

But I don't understand pagination concept. Below the code snippet of the request with my parameters:

div_lst = trapi.get_agenda(agenda_request=AgendaRequest(calendar_type=CalendarType.DIVIDEND_CALENDAR,
                        start_date=stdts,
                        end_date=endts,
                        currencies=cur,
                        offset=0,
                        limit=250,
                    ),
                    raw=True,
                ).get('items', [])

With this snippet I get only a maximum of 100 items, unless limit=250 parameter. Look at the console output. Do you have a real example of pagination ?

Console output

Dictionary output divd_calender.json

[Chavithra]

PAGINATION SUMMARY Just looking for pagination python you will find countless resources already explaining it in details.

So I will keep the explanations short here.

In a nutshell to retrieve all the batches, you make the same request multiple time just updating the offset.

Example:

agenda_request = AgendaRequest(...)

# BATCH 0
agenda_request.offset = 0
api.get_agenda(agenda_request=agenda_request)

# BATCH 1
agenda_request.offset = agenda_request.limit * 1
api.get_agenda(agenda_request=agenda_request)

...

# BATCH X
agenda_request.offset = agenda_request.limit * X
api.get_agenda(agenda_request=agenda_request)

Off course you need to stop fetching batches before offset > total.

CURRENCY Looks like these are comma separated values like:

currencies="USD,EUR"

[AureliusMarcusHu]

Dear Chavithra,

Thanks for the fast reply and explanation. The functions below made a offline dividend history calender dictionary with combined index keys. I will update every week the offline dictionary by calling the function "div_lookup()". On this way searching in the dividend history is very easy. Iterate over the dictionary keys list, split the key (',') and you can search on year, date, isin, name and currency.

It works as a charme. Thank you.

def div_request(stdts, endts, curs, pag, offs):
    global trapi
    from degiro_connector.trading.models.agenda import AgendaRequest, CalendarType
    div_lst = []
    stat = True
    try:
        div_lst = trapi.get_agenda(
            agenda_request=AgendaRequest(
                calendar_type=CalendarType.DIVIDEND_CALENDAR,
                start_date=stdts,
                end_date=endts,
                currencies=curs,
                offset=offs,
                limit=100,
            ),
            raw=True,
        ).get('items', [])
    except:
        logNotes([chr(10), 'Connection page ; ' + pag + ' ; failed'], logloc='dvd_lookup')
        sleep(0.99)
        lgn()
        stat = False
    return div_lst, stat

def div_lookup(show=None):
    global trapi
    from deps.file_tools import doesFileExist, pathFile, load_json
    from datetime import datetime as dt
    from dateutil.relativedelta import relativedelta
    show = False if show is None else show
    curs = 'EUR,USD'
    flnm = 'dvd_hist_cald.json'
    map = 'data'
    pf = pathFile(fnm=flnm, mp=map, use_rd=False)
    if doesFileExist(pf, noted=False):
        divd = load_json(fn_or_pf=pf, log=False)
        mntsb = 1
    else:
        divd = {}
        mntsb = 6
    stdivdlng = len(divd)
    logNotes(['dictionary ; existent items ; ' + str(stdivdlng)], logloc='dvd_lookup')

    nu = dt.now()
    stdt = (nu - relativedelta(months=mntsb)).strftime('%Y-%m-%d') + ' 00:00:00'
    endt = (nu + relativedelta(months=6)).strftime('%Y-%m-%d') + ' 00:00:00'
    offs = 0
    lng_lst = 0
    while (offs == 0) or (lng_lst) >= 100:
        pag = str(int(offs / 100 + 1))
        div_lst, stat = div_request(stdt, endt, curs, pag, offs=offs)
        lng_lst = len(div_lst)
        if stat:
            offs += 100
            if lng_lst > 0:
                for itm in div_lst:
                    div_dat = itm.get('dateTime', '1900-01-01')[:10]
                    idxk = div_dat + ',' + itm.get('isin', '') + ',' + itm.get('organizationName', '').upper()\
                           + ',' + itm.get('currency', '') + ',' + str(itm.get('eventId', ''))
                    chck = divd.get(idxk, False)
                    if not chck:
                        divd[idxk] = itm
                lnlst_note = 'page ; ' + pag + ' ; ' + str(lng_lst)
                logNotes([lnlst_note], logloc='dvd_lookup')
        else:
            lng_lst = 100
            #continue
        sleep(0.69)
    from deps.handle_tickers import sort_dict
    divd = sort_dict(divd, rev=True)
    endivdlng = len(divd)
    logNotes([chr(10), 'items ; added ; ' + str(endivdlng - stdivdlng)], logloc='dvd_lookup')
    dct_recs = 'dictionary ; total items ; ' + str(endivdlng)
    logNotes([dct_recs], logloc='dvd_lookup')
    save_json(divd, fnm=flnm, mp=map, show=show)
    return

[Chavithra]

Nice, will close the issue then.

Note: there is an example on how to turn this Series into a DataFrame. Doing these operations on a DataFrame might be easier.