search

chaoss / grimoirelab-elk

GNU General Public License v3.0
59 stars 121 forks source link

[poetry] ELK dependencies versions mismatch #980

Closed vchrombie closed 3 years ago

vchrombie commented 3 years ago

I was trying to setup github actions with poetry and ended up with a series of version mismatch errors.

$ poetry install
Creating virtualenv grimoireelk in /home/p0tt3r/dev/sources/grimoirelab-elk/.venv
Installing dependencies from lock file
Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting outdated dependencies. Run update to update them.

  SolverProblemError

  Because grimoireelk depends on PyMySQL (0.9.3) which doesn't match any versions, version solving failed.

It seems that the lock file is not aligned with the pyproject.toml file. If we change a dependency in the pyproject.toml manually, the lock needs to be updated too along with the change by using poetry lock command. This step was missing in the PR https://github.com/chaoss/grimoirelab-elk/pull/921.

$ poetry lock
Updating dependencies
Resolving dependencies... (59.4s)

  SolverProblemError

  The current project's Python requirement (3.5 || >=3.6,<4.0) is not compatible with some of the required packages Python requirement:
    - perceval-finos requires Python ^3.6, so it will not be satisfied for Python 3.5

  Because grimoireelk depends on perceval-finos (rev master) which requires Python ^3.6, version solving failed.

We can remove 3.5 from the supported python versions as its security support life in September 2020.

$ poetry lock
Updating dependencies
Resolving dependencies... (54.6s)

  SolverProblemError

  Because perceval-finos (rev master) depends on grimoirelab-toolkit (^0.1.12)
   and grimoireelk depends on grimoirelab-toolkit (branch master), perceval-finos is forbidden.
  So, because grimoireelk depends on perceval-finos (branch master), version solving failed.

The toolkit version in perceval (& it's components) is ^0.1.12. ELK is using git source as dependency, so the version is 0.2.0 since we had a new release recently. According to the poetry documentation, ^0.1.12 means >=0.1.12 <0.2.0.

We need to align all the dependencies in one format to avoid such errors. I think we should go with the git source option, at least with the grimoirelab related dependencies as we would need the latest code for the development.

I can send the PRs once we decide on this.

cc: @sduenas @zhquan WDYT?

Note: I could run the actions with poetry after these changes. https://github.com/vchrombie/grimoirelab-elk/actions/runs/796162203

sduenas commented 3 years ago

Thanks @vchrombie for having a look on this. I think we should update the lock file as you mentioned, remove Python 3.5 support, and update ELK to use the new toolkit version. Regarding to use the "git source option" to have dependencies aligned, I think we should skip this, at least for now, at stick to version numbers. I wonder if it would possible to add these dependencies (using "git" urls) in the development dependencies.

vchrombie commented 3 years ago

Thanks for the comment @sduenas.

I think we should update the lock file as you mentioned, remove Python 3.5 support,

Okay, I will send the PR accordingly.

and update ELK to use the new toolkit version.

The ELK is using the latest version of the toolkit because it has the git source url as the dependency. The issue lies with the perceval dependencies as they are using the version numbers.

Regarding to use the "git source option" to have dependencies aligned, I think we should skip this, at least for now, at stick to version numbers.

Okay, updating the version numbers in all the perceval tools would solve the issue. But, I have a small comment. Let's say we are using perceval = "^0.17.6". As long as the version is 0.17.x, it doesn't cause any problems. If the version is updated to 0.18.x, we have to update all the dependencies again.

I wonder if it would possible to add these dependencies (using "git" urls) in the development dependencies.

Do you mean adding the git source in the development dependency along with the version number in the main dependencies?

[tool.poetry.dependencies]
perceval = "^0.17.6"

[tool.poetry.dev-dependencies]
perceval = {branch = "master", git = "https://github.com/chaoss/grimoirelab-perceval"}
sduenas commented 3 years ago

Thanks for the comment @sduenas.

I think we should update the lock file as you mentioned, remove Python 3.5 support,

Okay, I will send the PR accordingly.

and update ELK to use the new toolkit version.

The ELK is using the latest version of the toolkit because it has the git source url as the dependency. The issue lies with the perceval dependencies as they are using the version numbers.

Regarding to use the "git source option" to have dependencies aligned, I think we should skip this, at least for now, at stick to version numbers.

Okay, updating the version numbers in all the perceval tools would solve the issue. But, I have a small comment. Let's say we are using perceval = "^0.17.6". As long as the version is 0.17.x, it doesn't cause any problems. If the version is updated to 0.18.x, we have to update all the dependencies again.

Yes, we would have to do that but it shouldn't be a problem once we get 1.0 but we have plans to remove ELK at some point and also all the unnecessary dependencies like perceval. For now, it shouldn't be a problem to work with these version numbers.

I wonder if it would possible to add these dependencies (using "git" urls) in the development dependencies.

Do you mean adding the git source in the development dependency along with the version number in the main dependencies?

[tool.poetry.dependencies]
perceval = "^0.17.6"

[tool.poetry.dev-dependencies]
perceval = {branch = "master", git = "https://github.com/chaoss/grimoirelab-perceval"}

Yes, exactly that. I don't know if it'll work.

vchrombie commented 3 years ago

Hi @sduenas, sorry for the late reply.

Yes, we would have to do that but it shouldn't be a problem once we get 1.0 but we have plans to remove ELK at some point and also all the unnecessary dependencies like perceval. For now, it shouldn't be a problem to work with these version numbers.

Hmm, sounds good. I will send the PRs accordingly then.

Yes, exactly that. I don't know if it'll work.

I have tested the suggested approach.

[tool.poetry.dependencies]
perceval = "^0.17.6"

[tool.poetry.dev-dependencies]
perceval = {branch = "master", git = "https://github.com/chaoss/grimoirelab-perceval"}

Poetry picks up the dev-dependencies (for poetry install) over the other one, unless the user specifies --no-dev. During poetry publish, the normal dependency perceval = "^0.17.6" will be used.

WDYT?

sduenas commented 3 years ago

I think it makes sense because it's kind of the same behavior we have now with requirements.txt.

vchrombie commented 3 years ago

I think it makes sense because it's kind of the same behavior we have now with requirements.txt.

Yup, it would be great if you can review the PRs after I open them.