search

geobricks / geobricks_qgis_plugin_faostat

QGIS Plugin to fetch FAOSTAT data and create thematic maps.
https://kalimadev.wordpress.com/2015/11/09/create-faostat-maps-with-qgis/
GNU General Public License v2.0
5 stars 3 forks source link

Error in QGIS 2.12-Lyon Windows 64bit #6

Open mbacou opened 8 years ago

mbacou commented 8 years ago

Tried to download cassava harvested area.

2015-11-12T07:08:08 1   Traceback (most recent call last):
              File "C:/Users/mbacou/.qgis2/python/plugins\geobricks_qgis_plugin_faostat\geobricks_qgis_plugin_faostat.py", line 207, in download_data
                data = get_data(domain_code, element_code, item_code)
              File "C:/Users/mbacou/.qgis2/python/plugins\geobricks_qgis_plugin_faostat\geobricks_faostat_connector.py", line 84, in get_data
                'List1Codes': create_countries_parameter(domain_code, lang),
              File "C:/Users/mbacou/.qgis2/python/plugins\geobricks_qgis_plugin_faostat\geobricks_faostat_connector.py", line 121, in create_countries_parameter
                print countries
            IOError: [Errno 9] Bad file descriptor
Kalimaha commented 8 years ago

I have tested your use case and it works on Mac, so I am trying to figure out whether this is a Windows-related problem. Moreover, the error is located in a piece of code that should be removed. I will then update the plugin to version 0.2. In the meanwhile, you can try to open (with textpad, or similar) the file:

C:/Users/mbacou/.qgis2/python/plugins\geobricks_qgis_plugin_faostat\geobricks_faostat_connector.py

and remove the line 121:

print countries

After that you should restart QGIS and check whether the problem is still there.

mbacou commented 8 years ago

I commented Line 121, but other problems remain. Looks like a Windows issue.

2015-11-12T10:03:56 1   Traceback (most recent call last):
              File "C:/Users/mbacou/.qgis2/python/plugins\geobricks_qgis_plugin_faostat\geobricks_qgis_plugin_faostat.py", line 207, in download_data
                data = get_data(domain_code, element_code, item_code)
              File "C:/Users/mbacou/.qgis2/python/plugins\geobricks_qgis_plugin_faostat\geobricks_faostat_connector.py", line 93, in get_data
                print data
            IOError: [Errno 9] Bad file descriptor
Kalimaha commented 8 years ago

Version 0.2 is available from the plugin repository, could you please try to update your plugin?

mbacou commented 8 years ago

Just upgraded to v0.2, but still same messages:

2015-11-12T10:11:39 1   Traceback (most recent call last):
              File "C:/Users/mbacou/.qgis2/python/plugins\geobricks_qgis_plugin_faostat\geobricks_qgis_plugin_faostat.py", line 207, in download_data
                data = get_data(domain_code, element_code, item_code)
              File "C:/Users/mbacou/.qgis2/python/plugins\geobricks_qgis_plugin_faostat\geobricks_faostat_connector.py", line 93, in get_data
                response = urllib2.urlopen(req)
            IOError: [Errno 9] Bad file descriptor

Stack trace:

An error has occured while executing Python code: 

IOError: [Errno 9] Bad file descriptor 
Traceback (most recent call last):
  File "C:/Users/mbacou/.qgis2/python/plugins\geobricks_qgis_plugin_faostat\geobricks_qgis_plugin_faostat.py", line 207, in download_data
    data = get_data(domain_code, element_code, item_code)
  File "C:/Users/mbacou/.qgis2/python/plugins\geobricks_qgis_plugin_faostat\geobricks_faostat_connector.py", line 93, in get_data
    response = urllib2.urlopen(req)
IOError: [Errno 9] Bad file descriptor

Python version: 2.7.5 (default, May 15 2013, 22:44:16) [MSC v.1500 64 bit (AMD64)] 
QGIS version: 2.12.0-Lyon Lyon, cd9d645 

Python Path:
C:/Users/mbacou/.qgis2/python/plugins\processing
C:/OSGEO4~1/apps/qgis/./python
C:/Users/mbacou/.qgis2/python
C:/Users/mbacou/.qgis2/python/plugins
C:/OSGEO4~1/apps/qgis/./python/plugins
C:\OSGEO4~1\apps\Python27\lib\site-packages\matplotlib-1.3.1-py2.7-win-amd64.egg
C:\OSGEO4~1\apps\Python27\lib\site-packages\nose-1.3.3-py2.7.egg
C:\OSGEO4~1\apps\Python27\lib\site-packages\tornado-4.0.1-py2.7-win-amd64.egg
C:\OSGEO4~1\apps\Python27\lib\site-packages\backports.ssl_match_hostname-3.4.0.2-py2.7.egg
C:\OSGEO4~1\apps\Python27\lib\site-packages\certifi-14.05.14-py2.7.egg
c:\osgeo4~1\apps\python27\lib\site-packages\python_dateutil-2.1-py2.7.egg
c:\osgeo4~1\apps\python27\lib\site-packages\six-1.3.0-py2.7.egg
C:\OSGeo4W64\apps\Python27\Lib
C:\OSGEO4~1\bin\python27.zip
C:\OSGEO4~1\apps\Python27\DLLs
C:\OSGEO4~1\apps\Python27\lib
C:\OSGEO4~1\apps\Python27\lib\plat-win
C:\OSGEO4~1\apps\Python27\lib\lib-tk
C:\OSGEO4~1\bin
C:\OSGEO4~1\apps\Python27
C:\OSGEO4~1\apps\Python27\lib\site-packages
C:\OSGEO4~1\apps\Python27\lib\site-packages\PIL
C:\OSGEO4~1\apps\Python27\lib\site-packages\jinja2-2.7.2-py2.7.egg
C:\OSGEO4~1\apps\Python27\lib\site-packages\markupsafe-0.23-py2.7-win-amd64.egg
C:\OSGEO4~1\apps\Python27\lib\site-packages\pytz-2012j-py2.7.egg
C:\OSGEO4~1\apps\Python27\lib\site-packages\win32
C:\OSGEO4~1\apps\Python27\lib\site-packages\win32\lib
C:\OSGEO4~1\apps\Python27\lib\site-packages\Pythonwin
C:\OSGEO4~1\apps\Python27\lib\site-packages\Shapely-1.2.18-py2.7-win-amd64.egg
C:\OSGEO4~1\apps\Python27\lib\site-packages\wx-2.8-msw-unicode
C:\OSGEO4~1\apps\Python27\lib\site-packages\xlrd-0.9.2-py2.7.egg
C:\OSGEO4~1\apps\Python27\lib\site-packages\xlwt-0.7.5-py2.7.egg
C:/Users/mbacou/.qgis2//python
C:\Users\mbacou\.qgis2\python\plugins\mmqgis/forms
C:\Users\mbacou\.qgis2\python\plugins
C:\Users\mbacou\.qgis2\python\plugins\QuickMultiAttributeEdit/forms
C:\OSGEO4~1\apps\qgis\python\plugins\fTools\tools
Kalimaha commented 8 years ago

Can you try to run the following code in the QGIS Python console? This works on Mac, I am trying to find out whether this is a Windows-related problem or not.

import urllib
import urllib2
import json

BASE_URL = 'http://fenixapps2.fao.org/api/v1.0/'

def get_data(domain_code, element_code, item_code, lang='en'):
    out = []
    url = BASE_URL + lang + '/data/'
    values = {
        'domain_code': domain_code,
        'List1Codes': create_countries_parameter(domain_code, lang),
        'List2Codes': element_code,
        'List3Codes': item_code,
        'List4Codes': create_years_parameter(),
        'group_by': '',
        'order_by': '',
        'operator': ''
    }
    data = urllib.urlencode(values, True)
    req = urllib2.Request(url, data)
    response = urllib2.urlopen(req)
    json_data = response.read()
    rows = json.loads(json_data)['data']
    for row in rows:
        out.append({
            'code': row['Country Code'],
            'value': row['Value'],
            'year': row['Year']
        })
    return out

def create_years_parameter():
    years = []
    for y in range(1961, 2015):
        years.append(str(y))
    return years

def create_countries_parameter(domain_code, lang='en'):
    out = []
    r = BASE_URL + lang + '/codes/countries/' + domain_code
    req = urllib2.Request(r)
    response = urllib2.urlopen(req)
    json_data = response.read()
    countries = json.loads(json_data)['data']
    for country in countries:
        out.append(str(country['code']))
    return out

print get_data('QC', 2312, 125)
mbacou commented 8 years ago

OK, here is the response.

Python Console 
Use iface to access QGIS API interface or Type help(iface) for more info
import urllib
import urllib2
import json
BASE_URL = 'http://fenixapps2.fao.org/api/v1.0/'
def get_data(domain_code, element_code, item_code, lang='en'):
    out = []
    url = BASE_URL + lang + '/data/'
    values = {
        'domain_code': domain_code,
        'List1Codes': create_countries_parameter(domain_code, lang),
        'List2Codes': element_code,
        'List3Codes': item_code,
        'List4Codes': create_years_parameter(),
        'group_by': '',
        'order_by': '',
        'operator': ''
    }
    data = urllib.urlencode(values, True)
    req = urllib2.Request(url, data)
    response = urllib2.urlopen(req)
    json_data = response.read()
    rows = json.loads(json_data)['data']
    for row in rows:
        out.append({
            'code': row['Country Code'],
            'value': row['Value'],
            'year': row['Year']
        })
    return out
def create_years_parameter():
    years = []
    for y in range(1961, 2015):
        years.append(str(y))
    return years
def create_countries_parameter(domain_code, lang='en'):
    out = []
    r = BASE_URL + lang + '/codes/countries/' + domain_code
    req = urllib2.Request(r)
    response = urllib2.urlopen(req)
    json_data = response.read()
    countries = json.loads(json_data)['data']
    for country in countries:
        out.append(str(country['code']))
    return out
print get_data('QC', 2312, 125)
[{'code': 5, 'value': 3.0, 'year': u'1961'}, {'code': 5, 'value': 3.0, 'year': u'1962'}, {'code': 5, 'value': 3.0, 'year': u'1963'}, {'code': 5, 'value': 6.0, 'year': u'1964'}, {'code': 5, 'value': 2.0, 'year': u'1965'}, {'code': 5, 'value': 2.0, 'year': u'1966'}, {'code': 5, 'value': 6.0, 'year': u'1967'}, {'code': 5, 'value': 2.0, 'year': u'1968'}, {'code': 5, 'value': 2.0, 'year': u'1969'}, {'code': 5, 'value': 2.0, 'year': u'1970'}, {'code': 5, 'value': 2.0, 'year': u'1971'}, {'code': 5, 'value': 3.0, 'year': u'1972'}, {'code': 5, 'value': 3.0, 'year': u'1973'}, {'code': 5, 'value': 3.0, 'year': u'1974'}, {'code': 5, 'value': 3.0, 'year': u'1975'}, {'code': 5, 'value': 3.0, 'year': u'1976'}, {'code': 5, 'value': 3.0, 'year': u'1977'}, {'code': 5, 'value': 3.0, 'year': u'1978'}, {'code': 5, 'value': 3.0, 'year': u'1979'}, {'code': 5, 'value': 4.0, 'year': u'1980'}, {'code': 5, 'value': 4.0, 'year': u'1981'}, {'code': 5, 'value': 4.0, 'year': u'1982'}, {'code': 5, 'value': 4.0, 'year': u'1983'}, {'code': 5, 'value': 4.0, 'year': u'1984'}, {'code': 5, 'value': 4.0, 'year': u'1985'}, {'code': 5, 'value': 4.0, 'year': u'1986'}, {'code': 5, 'value': 4.0, 'year': u'1987'}, {'code': 5, 'value': 4.0, 'year': u'1988'}, {'code': 5, 'value': 3.0, 'year': u'1989'}, {'code': 5, 'value': 7.0, 'year': u'1990'}, {'code': 5, 'value': 3.0, 'year': u'1991'}, {'code': 5, 'value': 3.0, 'year': u'1992'}, {'code': 5, 'value': 52.0, 'year': u'1993'}, {'code': 5, 'value': 48.0, 'year': u'1994'}, {'code': 5, 'value': 49.0, 'year': u'1995'}, {'code': 5, 'value': 52.0, 'year': u'1996'}, {'code': 5, 'value': 58.0, 'year': u'1997'}, {'code': 5, 'value': 50.0, 'year': u'1998'}, {'code': 5, 'value': 15.0, 'year': u'1999'}, {'code': 5, 'value': 15.0, 'year': u'2000'}, {'code': 5, 'value': 15.0, 'year': u'2001'}, {'code': 5, 'value': 15.0, 'year': u'2002'}, {'code': 5, 'value': 34.0, 'year': u'2003'}, {'code': 5, 'value': 15.0, 'year': u'2004'}, {'code': 5, 'value': 15.0, 'year': u'2005'}, {'code': 5, 'value': 15.0, 'year': u'2006'}, {'code': 5, 'value': 14.0, 'year': u'2007'}, {'code': 5, 'value': 17.0, 'year': u'2008'}, {'code': 5, 'value': 13.0, 'year': u'2009'}, {'code': 5, 'value': 15.0, 'year': u'2010'}, {'code': 5, 'value': 13.0, 'year': u'2011'}, {'code': 5, 'value': 14.0, 'year': u'2012'}, {'code': 5, 'value': 15.0, 'year': u'2013'}, {'code': 7, 'value': 350000.0, 'year': u'1961'}, {'code': 7, 'value': 400000.0, 'year': u'1962'}, {'code': 7, 'value': 400000.0, 'year': u'1963'}, {'code': 7, 'value': 400000.0, 'year': u'1964'}, {'code': 7, 'value': 420000.0, 'year': u'1965'}, {'code': 7, 'value': 450000.0, 'year': u'1966'}, {'code': 7, 'value': 
[truncated]
Kalimaha commented 8 years ago

It seems that the communication with the DB is interrupted before all the required data is transferred. I have updated slightly the code, could you please try the following one? If this works I'll make version 0.3 :smile: 

import urllib
import urllib2
import json

BASE_URL = 'http://fenixapps2.fao.org/api/v1.0/'

def get_data(domain_code, element_code, item_code, lang='en'):
    out = []
    url = BASE_URL + lang + '/data/'
    values = {
        'domain_code': domain_code,
        'List1Codes': create_countries_parameter(domain_code, lang),
        'List2Codes': element_code,
        'List3Codes': item_code,
        'List4Codes': create_years_parameter(),
        'group_by': '',
        'order_by': '',
        'operator': ''
    }
    data = urllib.urlencode(values, True)
    req = urllib2.Request(url, data)
    fp = urllib2.urlopen(req)
    response = ''
    while 1:
        data = fp.read()
        if not data:
            break
        response += data
    json_data = response
    rows = json.loads(json_data)['data']
    for row in rows:
        out.append({
            'code': row['Country Code'],
            'value': row['Value'],
            'year': row['Year']
        })
    return out

def create_years_parameter():
    years = []
    for y in range(1961, 2015):
        years.append(str(y))
    return years

def create_countries_parameter(domain_code, lang='en'):
    out = []
    r = BASE_URL + lang + '/codes/countries/' + domain_code
    req = urllib2.Request(r)
    response = urllib2.urlopen(req)
    json_data = response.read()
    countries = json.loads(json_data)['data']
    for country in countries:
        out.append(str(country['code']))
    return out

print get_data('QC', 2312, 125)
mbacou commented 8 years ago

Ha no sorry I added [truncated] because the response was too long for Github! The response was a complete json.

Kalimaha commented 8 years ago

Ah ok! Well, if the code works in the Python console it should work in the plugin as well, considering it is the same code! I will release version 0.3 anyways, it may solve your problem.