Missing `tomlkit` - Githubissues

cschwan commented 2 years ago

At least on dom on the DIS-more branch I get:

Error processing line 1 of /home/cschwan/.local/lib/python3.10/site-packages/zzz_poetry_dynamic_versioning.pth:

  Traceback (most recent call last):
    File "/usr/lib/python3.10/site.py", line 186, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "/home/cschwan/.local/lib/python3.10/site-packages/poetry_dynamic_versioning/__init__.py", line 13, in <module>
      import tomlkit
  ModuleNotFoundError: No module named 'tomlkit'

alecandido commented 2 years ago

Can you add details on how to reproduce, please? (i.e. which command did you launch)

cschwan commented 2 years ago

./rr install for instance, but anything starting with ./rr gives these warnings.

alecandido commented 2 years ago

Not reproduced locally, I'm going to try on dom

cschwan commented 2 years ago

That's strange:

cschwan@dom runcards $ which pip
/usr/bin/pip
cschwan@dom runcards $ pip show tomlkit
Error processing line 1 of /home/cschwan/.local/lib/python3.10/site-packages/zzz_poetry_dynamic_versioning.pth:

  Traceback (most recent call last):
    File "/usr/lib/python3.10/site.py", line 186, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "/home/cschwan/.local/lib/python3.10/site-packages/poetry_dynamic_versioning/__init__.py", line 13, in <module>
      import tomlkit
  ModuleNotFoundError: No module named 'tomlkit'

Remainder of file ignored
Name: tomlkit
Version: 0.9.1
Summary: Style preserving TOML library
Home-page: https://github.com/sdispater/tomlkit
Author: Sébastien Eustace
Author-email: sebastien@eustace.io
License: UNKNOWN
Location: /usr/lib/python3.10/site-packages
Requires: 
Required-by: poetry-dynamic-versioning, poetry

alecandido commented 2 years ago

I've just run:

./rr run nnpdf31_proc/HERA_CC_318GEV_EM_SIGMARED theories/theory_213.yaml

on dom, and it worked.

cschwan commented 2 years ago

Even starting python complains about it:

Error processing line 1 of /home/cschwan/.local/lib/python3.10/site-packages/zzz_poetry_dynamic_versioning.pth:

  Traceback (most recent call last):
    File "/usr/lib/python3.10/site.py", line 186, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "/home/cschwan/.local/lib/python3.10/site-packages/poetry_dynamic_versioning/__init__.py", line 13, in <module>
      import tomlkit
  ModuleNotFoundError: No module named 'tomlkit'

Remainder of file ignored
Python 3.10.2 (main, Jan 15 2022, 19:56:27) [GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()

@AleCandido could you please try to delete ~/.local and rerun ./rr install?

felixhekhorn commented 2 years ago

@cschwan this seems to me like your poetry-dynamic-versioning installation is somehow corrupted ... as far as I understood it has to be installed alongside poetry (meaning in the very same environment)

what do you get on

felix@FHe19b:~$ pip list | grep poetry
poetry                        1.1.12
poetry-core                   1.0.7
poetry-dynamic-versioning     0.13.0

and

felix@FHe19b:~$ which poetry
/home/felix/.local/bin/poetry

and tomlkit is a direct dependency https://github.com/mtkennerly/poetry-dynamic-versioning/blob/eac2f68f6e01f6e336dfb6c23a4c90911c12f013/pyproject.toml#L28 so pip has messed up, maybe?

cschwan commented 2 years ago

There you go:

cschwan@dom runcards $ pip list | grep poetry
poetry                        1.1.13
poetry-core                   1.0.7
cschwan@dom runcards $ which poetry
/usr/bin/poetry

alecandido commented 2 years ago

I tried to delete .local and I reproduced the issue.

I solved deleting again, and simply using the system available poetry. I installed manually dynamic versioning with pip.

In general, I believe the solution to be the following: do not use ./rr install to install poetry, dynamic versioning, or pre-commit. The idea to manage the developer's environment with a script is wrong on its own: each developer will have its own very personal environment, with all its quirks, and the developer itself has to be responsible for the management (it's hard to maintain something that works everywhere in any context, so better not to do).

cschwan commented 2 years ago

I understand what you're saying, but in a sense that defeats the purpose of ./rr - I don't care about poetry or any other Python package, I just want to generate processes.

felixhekhorn commented 2 years ago

Mmm but that's like saying "I just want to play that game" and then missing on the video driver, no?

alecandido commented 2 years ago

Actually, I believe this will be solved by NNPDF/runcards#127 and NNPDF/runcards#128.

The workflow I have in mind is the following:

if you're a developer, you're in charge to install poetry & co. by yourself
otherwise, you can just use pip install, both with the remote version (pip install <runcards-runner-final-name>) and with the local one (pip install .)

Once you install with pip, then rr or its replacement will be available as command, and it will take the place of ./rr. Instructions for developers are delicate and we can even spell personally (we will write something, but I'd expect it might fail in general if very detailed), instructions for everyone else will be simple, even without ./rr.

Eventually, when poetry will release 1.2.0, we will get even editable mode with pip for free.

cschwan commented 2 years ago

Mmm but that's like saying "I just want to play that game" and then missing on the video driver, no?

True :smile:, and I've worded my statement probably more strongly than I felt, because - at least subjectively - 'poetry' keeps breaking time and again for me.

I've just recently realized that what I had in mind concerning the design of our runner was the containerization of the runcards runner; this would solve all the version and installation problems reliably.

alecandido commented 2 years ago

I keep thinking about poetry, that's also the feeling of @scarlehoff for example.

But analyzing all the issues one by one, what I currently believe is that poetry failures are just the signal of us doing something bad (mostly with versions). So, in most cases it's leading to early detection of inconsistencies.

This applies to all the restrictions on the python versions, or the dependency on various packages, but it does not apply to the issues related to poetry installation and environment management. About this second category I believe the solution to be to use more scripts (like ŧest or run and so on), that we can write once and they are proved to work, plus the one written above: for actual usage (not development) rely on pip installation.

Moreover, there are the problems related to the installation of poetry itself: this is almost a non-problem, I created it myself with the installation script (I believe)...

cschwan commented 2 years ago

@AleCandido I agree with what you said; I've put 'poetry' in quotation marks in my comment above, because it's a little bit unfair to blame poetry for all problems, of course! I think I'm a bit spoiled by how well cargo and the Rust ecosystem works :smile:.

scarlehoff commented 2 years ago

To be completely honest, one of the reasons the python ecosystem is broken in dom is because we tried to be too clever using the same python environment for 4 different people... I'm actually amazed at how well it works most of the time.

we have < system>(shared) < FK>(shared) <.local>(each has their own)

in general < system> should always be consistent because is coming from the arch repositories with no pip installation and < FK> is used often enough that when it breaks someone will realise, so in dom the most likely solution most of the time is to remove all home-installed packages and then try again ...

felixhekhorn commented 2 years ago

Actually, I believe this will be solved by NNPDF/runcards#127 and NNPDF/runcards#128.

I agree, but keep in mind for that a necessary pre-requisite is settling a final name

alecandido commented 2 years ago

I agree, but keep in mind for that a necessary pre-requisite is settling a final name

I'd have said so as well, but at this point I believe we won't have a final name in less than one month and something, i.e. code meeting/collaboration meeting. At this point, I don't care any longer, I'll keep acting as if the name is final, and I'll pay the price of renaming stuffs afterwards.

This applies to all the restrictions on the python versions, or the dependency on various packages, but it does not apply to the issues related to poetry installation and environment management.

Another relevant consideration is that poetry is not mandatory. Actually, it violates the UNIX philosophy, in the sense that a single tool is performing two tasks (even if the two are quite coupled):

packaging: this we could get even with setuptools, but packaging is only needed for releases and easier distribution, so this not mandatory neither for development, nor for usage (you can always use the source, it's a valid python package even if you don't create a distribution package)
environment management: even this aspect is opt-in, since you can always decide to maintain your own virtual environment with virtualenv or whatever you prefer, simply you will have to manage dependencies on your own

At the end of the day, poetry is adding something (arguably including overhead... quite for sure overhead...), but it's removing nothing. So, if it doesn't work for you, you can always use whatever you prefer.

felixhekhorn commented 1 year ago

this is quite old - is it still an issue @cschwan ? or can we close? if I understood correctly @scarlehoff was able to use the package to reproduce everything ... cc @AleCandido

cschwan commented 1 year ago

I recently installed it somewhere and it wasn't a problem. Let's close it!

alecandido commented 1 year ago

Btw, since py3.11 (that I expect no one is using yet) tomlkit is part of Python standard library

felixhekhorn commented 1 year ago

interesting :upside_down_face:
grafik

cschwan commented 1 year ago

Race condition: github edition.

NNPDF / pinefarm

Missing `tomlkit` #9