Installation Issue with lxml

[DA-Mike]

I currently have lxml 3.8.0 on my machine, but the pyiso install wants to roll it back to 3.6.1 and fails every time. Has anyone else encountered this issue? Any solution to this?

[r24mille]

Looking at the history of requirements.txt the exact version requirement was added in 17a135c5d5b7ac1946e6f6959e35c4e711d0007e when it was changed from lxml to lxml==3.6.1 with the commit message, "upgrade lxml." Recently, during development of pyiso 3.20 we bumped it to 3.6.4 in 5c23f6fe29bb3ca6a3b5b8f725c068b97af92c3f to fix a bug.

I think it's a good idea to be less sensitive to versions in our requirements; certainly we should allow users to have a more recent version of a library. I do know off the top of my head that 3.6.4 is a minimum. If you'd like to change requirements.txt and setup.py to lxml>=3.6.4 and run the unit tests and integration tests you could see if that breaks anything. If everything passes, open a pull request.

All that said, since you're mentioning lxml 3.6.1 and not lxml 3.6.4, I assume that means you're installing pyiso 3.19 package from PyPI and installing using pip or something similar. Typically, when working on a project, it is useful to use virtual environments to isolate one project's dependency version requirements from another. See https://docs.python.org/3/tutorial/venv.html for details. Are you using virtual environments in this way, and we're just not playing nice with some other dependency in your project which requires a more recent lxml version?

[DA-Mike]

Thanks for the thorough response Reid. I am indeed installing from pip and am not familiar with unit and integration tests - I'm definitely noob/script kiddy level in terms of project development. That being said, I'd love to give this a shot and contribute to this project as it applies directly to me and my line of work. It looks like the link you provided is a good starting point, any other tips would be appreciated.

[r24mille]

No problem, and sorry this response is so late. I'd love to help you out with your contribution.

I am [...] not familiar with unit and integration tests.

I spent some time looking around for good articles, and to be honest this Stack Overflow answer about the different types of tests sums things up nicely.

Within PyISO we define unit tests and integration tests as roughly:

• Unit Test: They test individual functions or client methods. Notably:
  1. They have no external dependencies (i.e. they make no network calls). Given that our project makes so many network requests to ISO and balancing authority web sites for electricity data, that means we frequently make use of the mock library to mock out expected responses from network calls rather than making the network calls. The expected responses are stored in a fixtures directory in the tests folder.
  2. They are run by the continuous integration server (Travis CI) when a pull request is opened. You'll see the status reflected in your pull request. Generally, all tests must pass before we merge pull requests into the master branch.
  3. They're expected to be non-random in nature.
• Integration Test: They check integration between PyISO modules and with the servers of ISOs and balancing authorities we use as data providers. Notably:
  1. These tests make network calls and work with real data.
  2. Some tests require that environment variables on the host machine are set. Fore example, ISONE requires a username/password to retrieve data, which we set as environment variables.
  3. Dates are not fixed. They should test current or recent time ranges. This helps us catch problems if our external data providers change something about their data.

Directories containing unit tests and integration tests, structured as follows:

pyiso
  + --- tests
  |       + --- unit
  |       |         | test_base.py
  |       |         | test_client_one.py
  |       |         | test_client_two.py
  |       |
  |       + --- integration 
  |                 | integration_test_client_one.py
  |                 | integration_test_client_two.py
  |       
  + --- pyiso
          | base.py
          | client_one.py
          | client_two.py

Are you using virtual environments in this way?

I ran the following code to set up the latest PyISO in a Python 3 virtual environment on my Ubuntu 16.04 machine. Are you able to get it set up on its own in a virtual environment?

r24mille@hostname:~$ sudo apt-get install python3-venv
r24mille@hostname:~$ sudo apt-get build-dep -y lxml
r24mille@hostname:~$ python3 -m venv ~/example_venv
r24mille@hostname:~$ cd ~/example_venv/
r24mille@hostname:~/example_venv$ source ./bin/activate
(example_venv) r24mille@hostname:~/example_venv$ pip install pyiso
(example_venv) r24mille@hostname:~/example_venv$ python
Python 3.5.2 (default, Sep 14 2017, 22:51:06) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyiso import client_factory
>>> miso = client_factory('MISO')
>>> print(miso.get_generation(latest=True))
[{'timestamp': Timestamp('2017-11-02 02:40:00+0000', tz='UTC'), 'freq': '5m', 'gen_MW': 38822, 'fuel_name': 'coal', 'market': 'RT5M', 'ba_name': 'MISO'}, {'timestamp': Timestamp('2017-11-02 02:40:00+0000', tz='UTC'), 'freq': '5m', 'gen_MW': 11892, 'fuel_name': 'natgas', 'market': 'RT5M', 'ba_name': 'MISO'}, {'timestamp': Timestamp('2017-11-02 02:40:00+0000', tz='UTC'), 'freq': '5m', 'gen_MW': 11300, 'fuel_name': 'nuclear', 'market': 'RT5M', 'ba_name': 'MISO'}, {'timestamp': Timestamp('2017-11-02 02:40:00+0000', tz='UTC'), 'freq': '5m', 'gen_MW': 541, 'fuel_name': 'other', 'market': 'RT5M', 'ba_name': 'MISO'}, {'timestamp': Timestamp('2017-11-02 02:40:00+0000', tz='UTC'), 'freq': '5m', 'gen_MW': 4439, 'fuel_name': 'wind', 'market': 'RT5M', 'ba_name': 'MISO'}]

I had to run the apt-get build-dep step install the dependencies required for building lxml. Without it, neither 'pip install pyiso' nor pip install lxml==3.6.1 worked on my system.

Let me know how things are going and I can help with specific questions you may have. Cheers!