Update dependencies

[sajith]

Trying a solution to #140. The updated requirements.txt is produced by running pipreqs, like so:

pipreqs `pwd` --debug --force --mode compat

[sajith]

Tests pass in my local testing (macOS, Linux, Docker, Podman). One challenge that I've come across is in testing: when starting up the neo4j container, the mapping between host user and container user neo4j is not exactly deterministic. Sometimes the /imports volume is readable, sometimes it is not. I'm yet to figure out why.

Neo4j's docker-entrypoint.sh seems to perform some tricks to fix permissions on the /data volume, so it is less of a problem.

[sajith]

Also note that this PR uses ~= (or compatible release) operator, rather than pinning to specific versions. This seems to be a better idea for libraries.

[sajith]

Upon further thought (and some dependency-wrangling in fablib), it seems to me that there's a better way to handle dependencies.

1. Some dependencies are required at runtime. They are specified using install_requires.
2. Some are required by the setup script. They are specified using setup_requires.
3. Some are required by the test suite. They are specified using tests_require. (Although tests_require is deprecated as of setuptools 41.5.0 and will be removed in the future.)

Since Python can't have two different versions of the same library, the set of dependencies install_requires should be "open", meaning we can't pin them at specific versions. Reason being that packages that depend on fim are going to have trouble with version conflicts when the other packages they depend and fim don't agree.

(For example, when working on fablib, I can't install flake8 in the same environment because fim has pinned pycodestyle==2.6.0 and pycodestyle is a dependency of flake8 but there's no version of flake8 that can use that specific version of pycodestyle.)

A better way to specify them would be with this in requirements.txt:

matplotlib >= 3.6.0,
neo4j >= 4.1.0,
networkx >= 2.8.7,
networkx_query >= 1.0.1,
PyYAML >= 6.0,
recordclass >= 0.17.5,
requests >= 2.28.1

We'll also need to change setup_requires to this, I think:

setup_requires = [
    "setuptools >= 65.4.1",
    "wheel >=0.37.1".
]

And then ask people to install pytest "manually" because tests_require will no longer be a thing.

I will update the PR. And I think we ought to have a way to test this PR against multiple Python versions and on different OSes. :-)

[sajith]

Came across this, here:

<img src="https://user-images.githubusercontent.com/23618/195380096-6791f58f-f899-465a-a1a2-6cfdedfc6e11.png" alt="opinions about dependencies" width="75%" />

[sajith]

Came across another fun thing when testing this PR locally (Fedora 36, Python 3.9.14):

$ python3.9 -m venv venv
$ source venv/bin/activate
$ pip install -e .
Obtaining file:///home/sajith/projects/fabric-testbed/InformationModel
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [53 lines of output]
      /home/sajith/projects/fabric-testbed/InformationModel/venv/lib64/python3.9/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
        warnings.warn(
      running egg_info
      creating /tmp/pip-pip-egg-info-zigui0vx/fabric_fim.egg-info
      writing /tmp/pip-pip-egg-info-zigui0vx/fabric_fim.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-pip-egg-info-zigui0vx/fabric_fim.egg-info/dependency_links.txt
      writing requirements to /tmp/pip-pip-egg-info-zigui0vx/fabric_fim.egg-info/requires.txt
      writing top-level names to /tmp/pip-pip-egg-info-zigui0vx/fabric_fim.egg-info/top_level.txt
      writing manifest file '/tmp/pip-pip-egg-info-zigui0vx/fabric_fim.egg-info/SOURCES.txt'
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/home/sajith/projects/fabric-testbed/InformationModel/setup.py", line 10, in <module>
          setuptools.setup(
        File "/home/sajith/projects/fabric-testbed/InformationModel/venv/lib64/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/usr/lib64/python3.9/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/usr/lib64/python3.9/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/usr/lib64/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/home/sajith/projects/fabric-testbed/InformationModel/venv/lib64/python3.9/site-packages/setuptools/command/egg_info.py", line 299, in run
          self.find_sources()
        File "/home/sajith/projects/fabric-testbed/InformationModel/venv/lib64/python3.9/site-packages/setuptools/command/egg_info.py", line 306, in find_sources
          mm.run()
        File "/home/sajith/projects/fabric-testbed/InformationModel/venv/lib64/python3.9/site-packages/setuptools/command/egg_info.py", line 541, in run
          self.add_defaults()
        File "/home/sajith/projects/fabric-testbed/InformationModel/venv/lib64/python3.9/site-packages/setuptools/command/egg_info.py", line 578, in add_defaults
          sdist.add_defaults(self)
        File "/usr/lib64/python3.9/distutils/command/sdist.py", line 226, in add_defaults
          self._add_defaults_python()
        File "/home/sajith/projects/fabric-testbed/InformationModel/venv/lib64/python3.9/site-packages/setuptools/command/sdist.py", line 111, in _add_defaults_python
          build_py = self.get_finalized_command('build_py')
        File "/usr/lib64/python3.9/distutils/cmd.py", line 299, in get_finalized_command
          cmd_obj.ensure_finalized()
        File "/usr/lib64/python3.9/distutils/cmd.py", line 107, in ensure_finalized
          self.finalize_options()
        File "/home/sajith/projects/fabric-testbed/InformationModel/venv/lib64/python3.9/site-packages/setuptools/command/build_py.py", line 29, in finalize_options
          orig.build_py.finalize_options(self)
        File "/usr/lib64/python3.9/distutils/command/build_py.py", line 43, in finalize_options
          self.set_undefined_options('build',
        File "/usr/lib64/python3.9/distutils/cmd.py", line 286, in set_undefined_options
          src_cmd_obj = self.distribution.get_command_obj(src_cmd)
        File "/usr/lib64/python3.9/distutils/dist.py", line 857, in get_command_obj
          klass = self.get_command_class(command)
        File "/home/sajith/projects/fabric-testbed/InformationModel/venv/lib64/python3.9/site-packages/setuptools/dist.py", line 893, in get_command_class
          self.cmdclass[command] = cmdclass = ep.load()
        File "/home/sajith/projects/fabric-testbed/InformationModel/venv/lib64/python3.9/site-packages/pkg_resources/__init__.py", line 2465, in load
          return self.resolve()
        File "/home/sajith/projects/fabric-testbed/InformationModel/venv/lib64/python3.9/site-packages/pkg_resources/__init__.py", line 2471, in resolve
          module = __import__(self.module_name, fromlist=['__name__'], level=0)
      ModuleNotFoundError: No module named 'setuptools.command.build'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

It turned out that the version of setuptools in my venv (created by python3.9 -m venv venv rather than mkvirtualenv -r requirements.txt infomodel) was rather old (59.6.0); the error went away when I updated setuptools (to 65.4.1). It also seems that virtualenvwrapper installs the current version of pip and setuptools; so this wasn't a problem when using virtualenvwrapper.

However I think this is an inherent problem of setup.py-based packaging, based on my reading of Why you shouldn't invoke setup.py directly. We probably should plan on updating our build system (to using pyproject.toml rather than setup.py)?

[sajith]

Anyway, minus the problem with older versions of setuptools, tests pass locally. This PR is ready for review.

[ibaldin]

For some reason I did not see emails for these (?). Anyway, will look tomorrow.

[ibaldin]

BTW I use virtualenvwrapper. So YMMV without it. Looking at this now.

[sajith]

BTW I use virtualenvwrapper. So YMMV without it. Looking at this now.

Regarding the trouble with setuptools, it seems that virtualenvwrapper will install the current version of setuptools from PyPI, whereas venv by default will install the version of setuptools (and pip) included in the Python distribution. If newer versions of setuptools and pip is needed, venv has an --upgrade-deps argument.