The 'PyMySQL==0.9.3' distribution was not found and is required by grimoire-elk

[lukaszgryglicki]

Now when you try to use p2o.py built from master branch it returns the following error:

root@sds-debug-0:/# p2o.py 
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 582, in _build_master
    ws.require(__requires__)
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 899, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 790, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (grimoire-elk 0.63.0 (/usr/local/lib/python3.5/dist-packages), Requirement.parse('grimoire-elk==0.76.0'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/p2o.py", line 4, in <module>
    __import__('pkg_resources').require('grimoire-elk==0.76.0')
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 3256, in <module>
    @_call_aside
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 3240, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 3269, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 584, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 597, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 785, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'PyMySQL==0.9.3' distribution was not found and is required by grimoire-elk

I remember that some time ago there was an idea to add test coverage for successful built of commands. This is maybe the 4th or even more case when we try to build an image with all tools (perceval, sortinghat, p2o, etc.) and at least one of them is not working.

[lukaszgryglicki]

Seems like PyMySQL 0.10.0 is installed, but elk requires exactly PyMySQL==0.9.3, changing to PyMySQL>=0.9.3 fixes the issue. But why there is still no check if all tools are usable, why specify a requirement that breaks tools and never actually checks if that tool works?

BTW: there are warning and errors while installing tools, but they're ignored and finally, tools will either work or not - you never know, for example:

WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
ERROR: grimoire-elk 0.63.0 has requirement pandas==0.22.0, but you'll have pandas 0.25.3 which is incompatible.
ERROR: grimoire-elk 0.63.0 has requirement redis<=2.10.6,>=2.10.0, but you'll have redis 3.0.0 which is incompatible.
ERROR: cereslib 0.1.14 has requirement numpy<=1.18.3, but you'll have numpy 1.18.5 which is incompatible.
ERROR: rq 1.0 has requirement redis>=3.0.0, but you'll have redis 2.10.6 which is incompatible.
ERROR: sortinghat 0.7.14 has requirement numpy<=1.18.3, but you'll have numpy 1.18.5 which is incompatible.
ERROR: kingarthur 0.1.18 has requirement redis==3.0.0, but you'll have redis 2.10.6 which is incompatible.
ERROR: grimoire-elk 0.63.0 has requirement pandas==0.22.0, but you'll have pandas 0.25.3 which is incompatible.

[valeriocos]

Hi @lukaszgryglicki ,

The PR https://github.com/chaoss/grimoirelab-elk/pull/919 pinned the PyMySQL dependency and the CI tests were successful (https://travis-ci.org/github/chaoss/grimoirelab-elk/builds/710329778, pointer here). Also the generation of the new mordred release was successful in Dockerhub.

I guess the problem is with p2o, which is deprecated since long time in favor of micro-mordred. There are no tests for p2o.

I remember that some time ago there was an idea to add test coverage for successful built of commands.

I don't remember this discussion, but the tests in Travis should do what you ask (again p2o isn't covered in the CI tests).

BTW: there are warning and errors while installing tools, but they're ignored and finally, tools will either work or not - you never know, for example ...

You are right about these warning. Their correction and the way of improving the dependencies/releases management are part of the roadmap (https://github.com/chaoss/grimoirelab/projects/1, https://github.com/chaoss/grimoirelab/issues/240). Feel free to join the discussion at https://github.com/chaoss/grimoirelab/issues/240

[lukaszgryglicki]

Ok, so p2o is not supported at all. That explains everything. Well, we're not using micro-mordred because we have our own orchestrator (SDS) that deals with a lot of other tasks we have to support. So when p2o is no longer supported there is no direct way to just get data (like Perceval does) and put it to ES (which is what p2o.py deas). Should I understand that only mordred or sir mordred is the only way to actually get data and put it to ES? I think that stack should also maintain a low-level command that can do this and be used by some other tools/deployments that want to execute that manually ...

[valeriocos]

I think that stack should also maintain a low-level command that can do this and be used by some other tools/deployments that want to execute that manually ...

Your point is totally valid, and probably it makes sense to provide a low-level library similar to p2o (or evaluate how much work is needed to give a new life to p2o). I'd suggest to open an issue to discuss the inclusion of this work to the roadmap (https://github.com/chaoss/grimoirelab/projects/1).

Should I understand that only mordred or sir mordred is the only way to actually get data and put it to ES?

Yes, sirmordred and micro-mordred are the supported ways. In particular, Micro-mordred was created to be able to execute single mordred tasks (without using its scheduler). The main reasons of deprecating p2o was that it wasn't possible to execute most of the ELK studies, while micro-mordred allowed full compatibility with ELK.

[lukaszgryglicki]

Thanks!

[lukaszgryglicki]

Created a feature request https://github.com/chaoss/grimoirelab/issues/365 cc @valeriocos If possible please consider this...