search

makehumancommunity / makehuman

This is the main repository for the MakeHuman application as such.
http://www.makehumancommunity.org
Other
1.18k stars 244 forks source link

Standard Python packaging files #240

Closed MolassesLover closed 2 months ago

MolassesLover commented 2 months ago

Hi, there!

My aim with this pull request is to add the standard pyproject.toml and setup.py files, allowing for easier installation, and easier packaging for different platforms.

This allows Makehuman to be installed with pip, as most Python applications can be.

pip install .

Doing so adds makehuman as an executable in the user's install directory. Assuming it's in your path, just running makehuman as a command should work. Although, it does result in getpath not being found, and an error occurring as a result.

ModuleNotFoundError: No module named 'getpath'

This is presumably because Makehuman expects a specific working directory. This will be fixed when I open the pull request.

MolassesLover commented 2 months ago

This is presumably because Makehuman expects a specific working directory. This will be fixed when I open the pull request.

Relative imports are no longer an issue. Instead of importing specific attributes from makehuman.py, you can simply import it directly. Worth noting that makehuman.py was renamed to main.py to fix namespace issues.

joepal1976 commented 2 months ago

@black-punkduck : Maybe this is a better approach than the other one we discussed?

MolassesLover commented 2 months ago

@black-punkduck : Maybe this is a better approach than the other one we discussed?

Is there another installation method planned?

black-punkduck commented 2 months ago

sorry I was busy this week with normal job. Well reason Joel put me in between I have to explain.

We have this "old" makehuman in a debian ppa on launchpad.net + a windows .exe on tuxfamily. Both packets were without any bugfixes which were made in between. I wanted to change that. Now I first created a .exe no longer on tuxfamily, but on our own makehumancommunity.org server (makehuman 1.3 then, because it contains bodyshapes etc). The ppa packets I just did, but still locally on my own box. Joel will upload them when I am ready (there are no real further changes for that except that I use the start screen from elv, she did the bodyshapes).

Your work could be used for an easier installation directly from github (which I as a mainly Linux guy would prefer tbh. but of course it is not the easiest way for all users).

Well what we do now will be the last changes except bugfixes for this old makehuman.

When you check our forum, I already work on a new one. It is here on github, but still hidden. It is using pyside6, capable to use different base-meshes, no longer demanding clothes only quad/triangles, can work ortho and perspecitve mode and is using sqlite instead of handmade filecaches etc etc. I am half way through, just started with skeletons. It will be completely rewritten, especially since internally OpenGL changed .. (look since when QUADS were deprecated)

That version should also be installable from github directly and of course downloads the resources like skins or meshes from our own community servers. Also as a script but also build into application. Skins or meshes are huge and we will not place them here.

Well and pip? pip mostly works fine in a v_env. But pip itself, where people demand higher versions of some tools than the OS would handle, sometimes can trash even system processes like yum/dnf install written in python but with older libraries ... and exactly when a tool is installed in /usr/share with root permissions everything ends up in a mess (although in theory different python versions could exist on one box). There is a reason for the pip message we get, when we install with root user. But a ppa usually only handles a system-wide installation.

So I guess I need to check if sth. changes where the ppa process could go wrong. And I would put that 1.3 instead of the 1.2 into your file ;) I should be able to do this at the weekend ...

MolassesLover commented 2 months ago

sorry I was busy this week with normal job. Well reason Joel put me in between I have to explain.

Nothing to apologise for, take your time!

My motivation for getting these files in isn't just installation through pip. There are tools like Nix which take advantage of the files to make installation extremely simple.

Other than that, ABI issues can be resolved in the packaging files directly, and they're an inevitability anyways, given that the modules use dynamically linked libraries.

All in all, pip sucks, but it would make it much easier for me to create more reliable packages in the future.

Nix packaging would especially be great to see, given its reproducibility, and approach to dependencies. A user would be able to install Makehuman with Nix, and always have a guarantee it works.

black-punkduck commented 2 months ago

well the conflicts result from a change in makehuman.py I guess, but the whole thing is more complex. Took me some time, so waiting for the weekend was definitely the best way XD

We can ONLY accept the change, if we can use it with ppa and pynsist for Windows as well. Github + pip is just one way .. and here we have some major trouble, when you install an old software and demand new version with the pyproject.toml. I used a box with Ubuntu 20.4 on purpose with no new python installed but a virtual environment for the rest.

(old_makehuman) punkduck@valeria:~/build/old_makehuman/makehuman-molasses$ pip install .
Processing /data/punkduck/build/old_makehuman/makehuman-molasses
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
ERROR: Package 'makehuman-1.2.0' requires a different Python: 3.8.10 not in '==3.11.*'

New python version for a 6 year old software with openGL 1.2 which is from 1998 and outdated since OpenGL 3.1 ;)

Okay ... I changed that to >=3.6.0 (3.6.* with wildcards did not work, but anyway)

Then of course the following happens:

ERROR: Command errored out with exit status 1:
     command: /data/punkduck/build/old_makehuman/bin/python3 /tmp/tmpcx00uekq prepare_metadata_for_build_wheel /tmp/tmpkv61i0pn
         cwd: /tmp/pip-install-35yhu9r8/PyQt5
    Complete output (32 lines):
    pyproject.toml: line 7: using '[tool.sip.metadata]' to specify the project metadata is deprecated and will be removed in SIP v7.0.0, use '[project]' instead
    Traceback (most recent call last):
      File "/tmp/tmpcx00uekq", line 126, in prepare_metadata_for_build_wheel
        hook = backend.prepare_metadata_for_build_wheel
    AttributeError: module 'sipbuild.api' has no attribute 'prepare_metadata_for_build_wheel'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/tmp/tmpcx00uekq", line 280, in <module>
        main()
      File "/tmp/tmpcx00uekq", line 263, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/tmp/tmpcx00uekq", line 130, in prepare_metadata_for_build_wheel
        return _get_wheel_metadata_from_wheel(backend, metadata_directory,
      File "/tmp/tmpcx00uekq", line 159, in _get_wheel_metadata_from_wheel
        whl_basename = backend.build_wheel(metadata_directory, config_settings)
      File "/tmp/pip-build-env-1wvdv07d/overlay/lib/python3.8/site-packages/sipbuild/api.py", line 46, in build_wheel
        project = AbstractProject.bootstrap('wheel',
      File "/tmp/pip-build-env-1wvdv07d/overlay/lib/python3.8/site-packages/sipbuild/abstract_project.py", line 92, in bootstrap
        project.setup(pyproject, tool, tool_description)
      File "/tmp/pip-build-env-1wvdv07d/overlay/lib/python3.8/site-packages/sipbuild/project.py", line 587, in setup
        self.apply_user_defaults(tool)
      File "/tmp/pip-install-35yhu9r8/PyQt5/project.py", line 68, in apply_user_defaults
        super().apply_user_defaults(tool)
      File "/tmp/pip-build-env-1wvdv07d/overlay/lib/python3.8/site-packages/pyqtbuild/project.py", line 51, in apply_user_defaults
        super().apply_user_defaults(tool)
      File "/tmp/pip-build-env-1wvdv07d/overlay/lib/python3.8/site-packages/sipbuild/project.py", line 237, in apply_user_defaults
        self.builder.apply_user_defaults(tool)
      File "/tmp/pip-build-env-1wvdv07d/overlay/lib/python3.8/site-packages/pyqtbuild/builder.py", line 50, in apply_user_defaults
        raise PyProjectOptionException('qmake',
    sipbuild.pyproject.PyProjectOptionException
    ----------------------------------------
ERROR: Command errored out with exit status 1: /data/punkduck/build/old_makehuman/bin/python3 /tmp/tmpcx00uekq prepare_metadata_for_build_wheel /tmp/tmpkv61i0pn Check the logs for full command output

a typical python stackdump with "double exception" ... (I really love that because it is totally clear what the problem is, usually I then explode and shout f*** pip and python).

However I expected problems with sip, because I did a new Windows version as well and got the same problem. For windows, the result is inside a container with the complete environment, very similar to blender. It is a .exe in the end.

So I changed the following lines in your pyproject.toml according to pynsist.cfg (check in buildscripts) ... just to see if I can use a pip install '.' .

version = "1.3.0"

dependencies = [
        "sip==5.5.0",
        "numpy>=1.17.4",
        "PyQt5==5.15.1",
        "PyQt5-sip==12.9.0",
        "PyOpenGL>=3.1.0"
]

requires-python = ">=3.6.0"

Then I got this:

(old_makehuman) punkduck@valeria:~/build/old_makehuman/makehuman-molasses$ pip install .
Processing /data/punkduck/build/old_makehuman/makehuman-molasses
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting numpy>=1.17.4
  Downloading numpy-1.24.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
     |████████████████████████████████| 17.3 MB 2.3 MB/s
Collecting PyOpenGL>=3.1.0
  Downloading PyOpenGL-3.1.7-py3-none-any.whl (2.4 MB)
     |████████████████████████████████| 2.4 MB 2.4 MB/s
Collecting sip==5.5.0
  Using cached sip-5.5.0-cp38-cp38-manylinux2014_x86_64.whl (983 kB)
Collecting PyQt5==5.15.1
  Downloading PyQt5-5.15.1-5.15.1-cp35.cp36.cp37.cp38.cp39-abi3-manylinux2014_x86_64.whl (71.6 MB)
     |████████████████████████████████| 71.6 MB 35 kB/s
Collecting PyQt5-sip==12.9.0
  Downloading PyQt5_sip-12.9.0-cp38-cp38-manylinux1_x86_64.whl (332 kB)
     |████████████████████████████████| 332 kB 2.4 MB/s
Collecting toml
  Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting packaging
  Using cached packaging-24.0-py3-none-any.whl (53 kB)
Building wheels for collected packages: makehuman
  Building wheel for makehuman (PEP 517) ... done
  Created wheel for makehuman: filename=makehuman-1.3.0-py3-none-any.whl size=482705 sha256=37d097a50356b2f68333dd3b3ec2cbbbd48aade1248cee26f9778bdd40944866
  Stored in directory: /data/punkduck/.cache/pip/wheels/f3/ad/10/28713bf52020689fab443490ff26fb143abf93d523f980849a
Successfully built makehuman
Installing collected packages: numpy, PyOpenGL, toml, packaging, sip, PyQt5-sip, PyQt5, makehuman
Successfully installed PyOpenGL-3.1.7 PyQt5-5.15.1 PyQt5-sip-12.9.0 makehuman-1.3.0 numpy-1.24.4 packaging-24.0 sip-5.5.0 toml-0.10.2

As you can see, especially sip and pyqt are extremly depending on the python version. At least the mechanism itself works.

But when we update our ppa we want to force an update of the OS as well? That is something we definitely cannot do for the old version tbh. And speaking about ppa:

You renamed makehuman.py to main.py? Well not only the desktop-starter makehuman in (small bash script) points to that file.

A simple grep -r helps (output manually reduced to commands only, no redhead-packet-manager):

(old_makehuman) punkduck@valeria:~/build/old_makehuman/makehuman-molasses$ grep -r makehuman.py *
buildscripts/darwin/setup.py:APP = ['../../makehuman/makehuman.py']
buildscripts/ppa/buildPPA.py:    "usr/share/makehuman/makehuman.py",
buildscripts/ppa/extras/makehuman-community:python3 makehuman.py "$@"
buildscripts/deb/bin/makehuman:python3 makehuman.py "$@"
buildscripts/deb/buildDeb.py:    "usr/share/makehuman/makehuman.py",
buildscripts/win32/makehuman.spec:appExecutable = exportPath( 'makehuman\makehuman.py' )
buildscripts/win32/mhstartwrapper.py:subprocess.call([sys.executable,"makehuman.py"])
buildscripts/build_prepare.py:MAIN_EXECUTABLE = 'makehuman.py'
buildscripts/build_prepare.py:        if not os.path.isfile(self.sourceFile('makehuman/makehuman.py')):
buildscripts/build_prepare.py:            raise RuntimeError("The specified source folder %s does not appear to be a valid MakeHuman root folder (cannot find file %s)" % (self.sourceFile(), self.sourceFile('makehuman/makehuman.py')))
buildscripts/build_prepare.py:            f = open(self.targetFile('makehuman/makehuman.py'), 'r')
buildscripts/build_prepare.py:                    print("Replaced release declaration in makehuman/makehuman.py at line %s." % (lIdx+1))
buildscripts/build_prepare.py:                    print("Replaced version-sub declaration in makehuman/makehuman.py at line %s." % (lIdx+1))
buildscripts/build_prepare.py:            f = open(self.targetFile('makehuman/makehuman.py'), 'w')
makehuman/makehuman:python3 makehuman.py "$@"
makehuman/plugins/1_mhapi/_utility.py:            exclude = ["makehuman.py", "qtui.py", "qtgui.py", "mhmain.py"]
makehuman/plugins/1_mhapi/_utility.py:            exclude = ["makehuman.py", "qtui.py", "qtgui.py", "mhmain.py"]

Not that good honestly, the whole build mechanisms will no longer work. After that we have a pip-only version. The windows users and those without github/pip knowledge will send us 100s of complaints in the forum. Unfortunately all the buildscripts either filter or even start makehuman.py. The new version I create is different, all parameters used for building can be retrieved from a JSON file, but here it is a "historical problem"

And when I see all these compatibilty problems I often ask myself if the change from c++ to python was such a good idea ... but that was made years before I joined the team and the new version will of course also be python again.

Anyway, if you really want to invest time to create sth, which is usable for both cases, we need to send you the way how windows and ppas are created so that you will not create conflicts. Of course you need to do test installs etc. .. for the old version. I don't know if that makes sense in the end.

MolassesLover commented 2 months ago

Nevermind, I'm not willing to work on a legacy build system, or support legacy distributions. I'll probably just fork the project. Thank you for your time, nonetheless.