Loading a default schema fails on Windows 10 with Python 3.7

[akmiller01]

Here's my full error trace.

Traceback (most recent call last):
  File "utils.py", line 534, in <module>
    xml_to_csv("test_data/new_and_updated.xml")
  File "utils.py", line 414, in xml_to_csv
    v203_schema = iati.default.activity_schema('2.03')
  File "C:\Program Files\Python37\lib\site-packages\iati\version.py", line 365, in wrap_decimalise_integer
    return input_func(version, *args[1:], **kwargs)
  File "C:\Program Files\Python37\lib\site-packages\iati\version.py", line 386, in wrap_normalise_decimals
    return input_func(version, *args[1:], **kwargs)
  File "C:\Program Files\Python37\lib\site-packages\iati\version.py", line 303, in wrap_allow_known_version
    return input_func(*args, **kwargs)
  File "C:\Program Files\Python37\lib\site-packages\iati\default.py", line 311, in activity_schema
    return _schema(iati.resources.get_activity_schema_paths, iati.ActivitySchema, version, populate)
  File "C:\Program Files\Python37\lib\site-packages\iati\default.py", line 288, in _schema
    schema = _populate_schema(schema, version)
  File "C:\Program Files\Python37\lib\site-packages\iati\default.py", line 255, in _populate_schema
    codelists_to_add = codelists(version)
  File "C:\Program Files\Python37\lib\site-packages\iati\default.py", line 127, in codelists
    return _codelists(version)
  File "C:\Program Files\Python37\lib\site-packages\iati\version.py", line 365, in wrap_decimalise_integer
    return input_func(version, *args[1:], **kwargs)
  File "C:\Program Files\Python37\lib\site-packages\iati\version.py", line 386, in wrap_normalise_decimals
    return input_func(version, *args[1:], **kwargs)
  File "C:\Program Files\Python37\lib\site-packages\iati\version.py", line 271, in wrap_allow_fully_supported_version
    return input_func(*args, **kwargs)
  File "C:\Program Files\Python37\lib\site-packages\iati\default.py", line 101, in _codelists
    paths = iati.resources.get_codelist_paths(version)
  File "C:\Program Files\Python37\lib\site-packages\iati\version.py", line 365, in wrap_decimalise_integer
    return input_func(version, *args[1:], **kwargs)
  File "C:\Program Files\Python37\lib\site-packages\iati\version.py", line 336, in wrap_allow_possible_version
    return input_func(*args, **kwargs)
  File "C:\Program Files\Python37\lib\site-packages\iati\resources.py", line 99, in get_codelist_paths
    files = pkg_resources.resource_listdir(PACKAGE, folder_path[len(resource_filesystem_path('')):])
  File "C:\Program Files\Python37\lib\site-packages\pkg_resources\__init__.py", line 1155, in resource_listdir
    resource_name
  File "C:\Program Files\Python37\lib\site-packages\pkg_resources\__init__.py", line 1417, in resource_listdir
    return self._listdir(self._fn(self.module_path, resource_name))
  File "C:\Program Files\Python37\lib\site-packages\pkg_resources\__init__.py", line 1510, in _listdir
    return os.listdir(path)
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\resources\\standard\\2-03\\codelists'

Should be reproducible with just the code in the README:

import iati.default

schema = iati.default.activity_schema('2.03')

Seems to be an issue with absolute filepaths?

[andylolz]

I think you are right that this is due to filepaths. Specifically, I think it’s because resources are all loaded via relative filepaths:

https://github.com/IATI/pyIATI/blob/269a2c1cb73479875a534ae3dfcb6ef9c03e1766/iati/resources.py#L28-L31

I think best practice in packages is to use absolute filepaths everywhere. I guess that’s the problem here.

[akmiller01]

Oddly enough it works just fine on my Ubuntu partition and doesn't try and access /resources/ at my system root. Just a Windows specific bug for the moment.

[andylolz]

Yes, same. I think using relative paths in this way is dodgy, because it’s not always obvious what it is relative to. That’s why package developers usually do something like:

basepath = os.path.dirname(os.path.abspath(__file__))

…and then prepend that to all relative paths.

[andylolz]

Aside: contrary to what the usage section in the README says, this library is not currently under active development.

See this announcement on IATI discuss: https://discuss.iatistandard.org/t/state-of-pyiati/1360/7

[akmiller01]

Found a fix, in case this project is ever rebooted.

On line 99 here, https://github.com/IATI/pyIATI/blob/269a2c1cb73479875a534ae3dfcb6ef9c03e1766/iati/resources.py#L99

folder_path[len(resource_filesystem_path('')):] comes out as "\\resources\\standard\\2-03\\codelists" on Windows, while it comes out as "/resources/standard/2-03/codelists" on Unix based operating systems.

For some reason, the default Python package resource manager pkg_resources interprets that as relative to the package for Unix, but absolute to the C: drive for Windows. Can be fixed by importing os, splitting the path, and re-joining it before searching for the resources.

import os
files = pkg_resources.resource_listdir(PACKAGE, os.path.join(*folder_path[len(resource_filesystem_path('')):].split(os.sep)))

[dalepotter]

I've added a PR to fix this in #329.

I've also separately added python 3.7 builds to Travis in #330 - ideally you would have a seperate Travis builds running in both Unix and Windows to confirm cross-platform support, but I haven't looked into if/how that is possible.

[andylolz]

ideally you would have a seperate Travis builds running in both Unix and Windows to confirm cross-platform support, but I haven't looked into if/how that is possible.

It’s not currently. Travis says:

The language 'python' is currently unsupported on the Windows Build Environment.