Updated ECB exchange rates

[emetor]

A feature I really would appreciate is to have an option for updated ECB exchange rates.

Whenever you request an exchange rate, check if the current ECB exchange rate file is already available in the app, if not download it from the ECB. Downloading the latest exchange rates from the ECB is possible as follows:

import urllib2
from zipfile import ZipFile
from StringIO import StringIO
zip = ZipFile(StringIO(urllib2.urlopen('http://www.ecb.europa.eu/stats/eurofxref/eurofxref.zip').read()))
zip.extractall()

What I'm not sure about is where to store files, how to keep them as backups in case that a download is not possible, exact timing of when to download a new file (16:00 CET?), etc. I would also appreciate if a solution could be integrated into currencyconverter for better maintainability. What do you think?

[alexprengere]

A few thoughts on that matter of using the latest rates available.

What is currently possible

• I commit in the sources every week the embedded CSV file containing the ECB rates. So if you update the package regularly using the git remote in your requirements.txt, you will get rates that are about a week old. Additionally, ECB itself can publish the rates a few days late over the weekends. I understand that this delay can be a problem.
• If you download yourself every day the ECB rates in a file.csv at some location, you may use this: c = CurrencyConverter('./path/to/file.csv'). Every time the currency converter is initialized, you will get the latest rates (and no re-installation of the project needed this time).

What we could do

I am not in favor of automatic rates downloading at "request time". I want everything that happens after initialization to be fast and predictable, because I usually load the app once and then do billions of conversions. But it may be a good idea to be able to check for new rates when initializing the CurrencyConverter object. For example, one way to do this is to support urls as input locations:

c = CurrencyConverter('http://www.ecb.int/stats/eurofxref/eurofxref-hist.zip')

This way, the latest rates are downloaded at each initialization.

Any thoughts?

[emetor]

Thank you very much for your thoughts Alex!

I use CurrencyConverter in a web application and generally do just a single conversion after every initialization. I'll probably opt for an option where I download myself the ECB rates as soon as the latest file is older than a day. Something like:

c = MyCurrencyConverter('./path/to/file.csv', expiry = '24', download_dir = 'http://www.ecb.europa.eu/stats/eurofxref/eurofxref.zip'

I can post my code here once I have implemented my solution in case it is useful for anybody else.

[alexprengere]

I hope you just initialize once at the start of your web application :smile:, because during init the file has to be loaded and the missing rates have to be filled, that's why it is slow. Then all conversions are fast.

Anyway I could support something like this:

c = CurrencyConverter('http://www.ecb.int/stats/eurofxref/eurofxref-hist.zip')

And then each time you restart your webapp you would get the latest rates, would this is be OK?

[emetor]

The webapp is generally not restarted and I initialize CurrencyConverter on every request for certain pages. The single day file is just 436 bytes so loading should go rather fast. However, there is a problem with the date that is given in the format '13 October 2016' - no idea why the ECB chooses a different format in the single day file? This makes the parse_date function brake.

This is my approach so far:

def get_exchange_rates():
    fname = os.path.join(os.path.dirname(__file__), '../../userfiles/exchange_rates/' + 'eurofxref' + date.today().strftime('%Y%m%d') + '.csv')
    # Check if the exchange rate file has already been downloaded today?
    if not os.path.isfile(fname):
        try:
            # Download latest file from ECB
            zip = ZipFile(StringIO(urllib2.urlopen('http://www.ecb.europa.eu/stats/eurofxref/eurofxref.zip').read()))
            zip.extractall(os.path.dirname(fname))
            fname_dwnld = os.path.join(os.path.dirname(__file__), '../../userfiles/exchange_rates/eurofxref.csv')
            os.rename(fname_dwnld, fname)
        except:
            # Download failed
            pass
    # Check for most recent file
    for days in range(0, 30):
        try:
            datum = date.today()-timedelta(days)
            fname = os.path.join(os.path.dirname(__file__), '../../userfiles/exchange_rates/' + 'eurofxref' + datum.strftime('%Y%m%d') + '.csv')
            CC = CurrencyConverter(fname)
            str_error = None
        except Exception as str_error:
            pass
        if not str_error:
            break
    return {'EUR': 1.0, 'USD': CC.convert(1, 'EUR', 'USD'), 'CHF': CC.convert(1, 'EUR', 'CHF')}

[alexprengere]

I committed a few things on the master branch, related to this issue, I hope this can help:

• the single day file from http://www.ecb.europa.eu/stats/eurofxref/eurofxref.zip should now work, I modified the date parser to accept both date formats. Also the currencies had trailing whitespaces in that file so I had to .strip() them to avoid bugs,
• zip files are now supported, if the input file name ends with a .zip extension, the currency converter will automatically decompress it in memory and load it. Additionally, the new default data source is now the embedded raw zipped file from http://www.ecb.int/stats/eurofxref/eurofxref-hist.zip, not the uncompressed version,
• Urls as input should now work, meaning you can now do:

c = CurrencyConverter('http://www.ecb.int/stats/eurofxref/eurofxref-hist.zip')
c = CurrencyConverter('http://www.ecb.europa.eu/stats/eurofxref/eurofxref.zip')
c = CurrencyConverter('./a/file.zip')
c = CurrencyConverter('./a/file.csv')

[emetor]

Great! Thank you for the support, exactly what I needed. Closing the issue.