Compiled sucessfully in Windows 10 Python 3.11 but CreateProcess fails (call to dispread to measure)

[VgerTest]

 Traceback (most recent call last):                                           
  File "D:\src\otros\displaycal-py3.py311\DisplayCAL\worker.py", line 7082,  
 in exec_cmd                                                                  
     self.subprocess = wexpect.spawn(                                         
                       ^^^^^^^^^^^^^^                                         
   File "D:\src\otros\displaycal-py3.py311\DisplayCAL\wexpect.py", line 328,  
 in spawn                                                                     
     return spawn_windows(                                                    
            ^^^^^^^^^^^^^^                                                    
   File "D:\src\otros\displaycal-py3.py311\DisplayCAL\wexpect.py", line 1775, 
 in __init__                                                                  
     self._spawn(command, args)                                               
   File "D:\src\otros\displaycal-py3.py311\DisplayCAL\wexpect.py", line 1848, 
 in _spawn                                                                    
     self.child_fd = self.wtty.spawn(self.command, self.args, self.env)       
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^       
   File "D:\src\otros\displaycal-py3.py311\DisplayCAL\wexpect.py", line 2073, 
 in spawn                                                                     
     self.startChild(args, env)                                               
   File "D:\src\otros\displaycal-py3.py311\DisplayCAL\wexpect.py", line 2174, 
 in startChild                                                                
     self.__oproc, _, self.conpid, self.__otid = CreateProcess(               
                                                 ^^^^^^^^^^^^^^               
 TypeError: Objects of type 'bytes' can not be converted to Unicode.         

It does not like this:

        self.__oproc, _, self.conpid, self.__otid = CreateProcess(
            None, commandLine, None, None, False, CREATE_NEW_CONSOLE, env, self.cwd, si
        )

[wisdomk]

same issue here ... I am very frustrated, I've spent like 5 hours trying to make this thing work on Windows with no avail. On Ubuntu and Mac it works like a charm though.

[eoyilmaz]

okay I'll look into this shortly

[eoyilmaz]

Fixed this, but there are other problems preventing DisplayCAL to run properly... working on it.

[eoyilmaz]

I think we need to use the PyPI version of wexpect now... Our version is old and not working properly...

[eoyilmaz]

Still working on this with no avail...

The main problem is that DisplayCAL cannot read from the stdout of the child process that is spawned through DisplayCAL.wexpect module. I tried using the Wexpect module from PyPI, hopefully the interface has not been changed and DisplayCAL was able to use it to spawn processes. But it is didn't solve the problem. Then I tried using different Python versions, i.e 3.9, 3.10, 3.11, 3.12, in none of them the Wexpect is working properly. Tried using utf-8 instead cp1252 (which was the default) for the code page, not working... Tried using different ArgyllCMS versions from 3.02, 3.1.0 to 3.2.0, same error.

I believe in the past with an update called "Windows Creators Update" the behaviour of the stdout has changed, still trying to understand why it is not working...

[eoyilmaz]

As always as soon as I sent a progress message here I solve something 😄

Apparently setting the "PYTHONLEGACYWINDOWSSTDIO" environment variable to anything reverts the behaviour change and I can now start the measurement and finally was able to skip the first couple of colour patches. But unfortunatelly, it is not progresssing any further...

[eoyilmaz]

Finally, I managed to make it work...

Obviously we had an encoding issue under Windows and one another issue was the DisplayCAL.worker._safe_send() was trying to encode whatever it was trying to send to the subprocess to bytes and it was not working properlly under Windows. Now all seems to be fixed and I'm currently doing a profiling and it is working fine 👍

aaand profiling is done... but it seems that there are other issues and no profile has been created... let's fix them too.

[eoyilmaz]

Okay fixed a couple more problems and I was able to finish my first calibration and profile generation under Windows 10 🎆

[eoyilmaz]

Anyone who wants to try DisplayCAL under Windows, I updated the installation instructions here: How To Install (Windows)

Please try and report back.

[eoyilmaz]

Fixed the codepage being cp1252 vs utf-8 issue without relying on the legacy stdio trick (PYTHONLEGACYWINDOWSSTDIO env var). So, Python 3.9, 3.10 and 3.11 are all working properly. Unfortunatelly some packages are not available for Python 3.12 so we are not currently supporting it.

I want to merge the changes to develop branch. If anyone finds out any issues we can handle them in new issue tickets.

[johnloopi]

Anyone who wants to try DisplayCAL under Windows, I updated the installation instructions here: How To Install (Windows)

Please try and report back.

I'll test it on my Windows 10 and 11 pcs later this evening and report back to you. :)

[VgerTest]

"pip install displaycal" Failed due to an absolute path /Users/eoyilmaz

Manually build without venv on a clean Py3.9 as in screenshots: pip install -r requirements.txt python -m build pip install dist\DisplayCAL-3.9.12-cp39-cp39-win_amd64.whl All OK

Running with Argyll 32 (64 has some issues about non accesing id3 device, but those where in DisplayCAL py2.7) -minor issues with CCSS names on combobox (not showing some names that DisplayCAL py2.7 show) -console report uncalibrated/calibrated (dispcal -R/-r) OK -calibration OK, but -profiling crashes : DisplayCAL: Starting interaction with subprocess dispread: Error - CGATS file read error : Unable to open file 'Measurement.ti1' for reading DisplayCAL: Reached EOF (OK) dispread exitcode: 1

-measurement report crashes DisplayCAL: Starting interaction with subprocess dispread: Error - CGATS file read error : Unable to open file 'Measurement.ti1' for reading DisplayCAL: Reached EOF (OK) dispread exitcode: 1

I tried to manually select ti1 file but same result.

Also on startup a non blocking/severe error message: writing to lock file: port: 15411 No module named 'comtypes.gen' Error - could not import WMI: No module named 'wmi' Comenzando... Audio module: wx 4.2.1 Inicializando GUI...

[johnloopi]

Since the version on PyPi is still version 3.9.12, I'm met with the same error as @opopnever was here https://github.com/eoyilmaz/displaycal-py3/issues/349 and @dkflint723 was here: https://github.com/eoyilmaz/displaycal-py3/issues/334

        File "C:\Users\MyUser\AppData\Local\Temp\pip-build-env-xlh9khju\overlay\Lib\site-packages\setuptools\_distutils\util.py", line 132, in convert_path
          raise ValueError("path '%s' cannot be absolute" % pathname)
      ValueError: path '/Users/eoyilmaz/Documents/development/displaycal-py3/CHANGES.html' cannot be absolute
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

C:\Users\MyUser>

I therefore tried building it myself, which I'm not very good at it seems, by doing this: I downloaded the newest test-branch, unzipped it, cmd'd into the unzipped folder and ran the commands from Alex Hawk's attachment file here https://hub.displaycal.net/forums/topic/displaycal-is-dead-time-to-move-on-what-alternative-you-choosed/page/5/#post-141305

For some reason it gives me the files "displaycal-0.0.0.tar.gz" and "DisplayCAL-0.0.0-cp311-cp311-win_amd64.whl" instead of "displaycal-3.9.12.tar.gz" and "DisplayCAL-3.9.12-cp310-cp310-win_amd64.whl" and when I open it with command "python -m DisplayCAL" I am met with a message to update to DisplayCAL 3.8.9.3, since my version thinks it's 0.

Could it be because I'm doing this in a VM and not a "real" computer? I figured it would be easier to roll back any changes with snapshots in a VM if something went wrong or I should change something quickly.

EDIT: This was on Windows 10 22H2. Will test Windows 11 23H2 later.

[eoyilmaz]

Ah sorry, I was completely confused when I written my previous message. The install instructions are explaining what needs to be done in the future, when we upload the current version to PyPI. The current version in PyPI is expected to fail.

What you can do is to use the following tarbal and install through that: displaycal-3.9.12-202406061835.tar.gz

An run the following:

  pip install displaycal-3.9.12-202406061835.tar.gz

[johnloopi]

Ah sorry, I was completely confused when I written my previous message. The install instructions are explaining what needs to be done in the future, when we upload the current version to PyPI. The current version in PyPI is expected to fail.

What you can do is to use the following tarbal and install through that: displaycal-3.9.12-202406061835.tar.gz

An run the following:

  pip install displaycal-3.9.12-202406061835.tar.gz

I am getting some warnings like: WARNING: The script clear_comtypes_cache.exe is installed in 'C:\Users\MyUser\AppData\Roaming\Python\Python311\Scripts' which is not on PATH. and: No module named 'comtypes.gen' Error - could not import WMI: No module named 'wmi'

But it starts up, and installs ArgyllCMS 3.2.0 without any issues. Will let my meters warm up for a bit, EFI ES-2000 and X-Rite i1 Display Pro Plus, and then I will test ICC calibration and .Cube.

Anything else you want me to test?

[VgerTest]

Anything else you want me to test?

Try measurement report on current display profile (basic verification) and on your resulting custom profile after the whole process.

[eoyilmaz]

WARNING: The script clear_comtypes_cache.exe is installed in 'C:\Users\MyUser\AppData\Roaming\Python\Python311\Scripts' which is not on PATH.

I think when you installed Python 3.11 you didn't select "Add Python 3.11 to PATH", just run the Python installer again, it will show a similar option.

No module named 'comtypes.gen' Error - could not import WMI: No module named 'wmi'

These are not required, they were neded for older versions of Windows (XP and 2000), ignore them.

Anything else you want me to test?

Just calibration and profiling is enough for now. But, of course, test as many things as possible. But I would most probably prefer to handle them as individual tickets.

[johnloopi]

WARNING: The script clear_comtypes_cache.exe is installed in 'C:\Users\MyUser\AppData\Roaming\Python\Python311\Scripts' which is not on PATH.

I think when you installed Python 3.11 you didn't select "Add Python 3.11 to PATH", just run the Python installer again, it will show a similar option.

I did select "Add python.exe to PATH", but I also clicked "Customize installation" and "Install for all users" out of old habits. Uninstalling Python 3.11, reinstalling it, and just clicking "Install now" without customizing further worked to get rid of that error.

No module named 'comtypes.gen' Error - could not import WMI: No module named 'wmi'

These are not required, they were neded for older versions of Windows (XP and 2000), ignore them.

That's good to know.

Anything else you want me to test?

Just calibration and profiling is enough for now. But, of course, test as many things as possible. But I would most probably prefer to handle them as individual tickets.

Got it, if I encounter any issues I'll create a new ticket(s) for them.

[eoyilmaz]

I added a Manual Installation (Windows) section to the README, that describes how to build and install the development version under Windows. It is better to follow that instead of the "How To Install (Windows)" section.

[berryboy2012]

Still working on this with no avail...

The main problem is that DisplayCAL cannot read from the stdout of the child process that is spawned through DisplayCAL.wexpect module. I tried using the Wexpect module from PyPI, hopefully the interface has not been changed and DisplayCAL was able to use it to spawn processes. But it is didn't solve the problem. Then I tried using different Python versions, i.e 3.9, 3.10, 3.11, 3.12, in none of them the Wexpect is working properly. Tried using utf-8 instead cp1252 (which was the default) for the code page, not working... Tried using different ArgyllCMS versions from 3.02, 3.1.0 to 3.2.0, same error.

I believe in the past with an update called "Windows Creators Update" the behaviour of the stdout has changed, still trying to understand why it is not working...

Possible related issue https://github.com/raczben/wexpect/issues/26 with proposed fix at https://github.com/raczben/wexpect/commit/850eece43bb21b46c78053a078d76cb3cce92d3e .

[eoyilmaz]

@berryboy2012 ah yeah that's a good idea for fixing the venv issue 👍

otherwise everything is working fine with system Python.

[alex-mirkin]

Installed Python 3.11 using scoop. Build and installed successfully, but got this when tried to run:

ModuleNotFoundError: No module named 'win32file'

Solved with python -m pip install pywin32. Maybe pywin32 wasn't installed because of scoop?

Great work btw!

[eoyilmaz]

@alex-mirkin did you follow Manual Setup (Windows) and installed it from the correct branch? If you used any other version it will not work on Windows properly.

[alex-mirkin]

@alex-mirkin did you follow Manual Setup (Windows) and installed it from the correct branch? If you used any other version it will not work on Windows properly.

@eoyilmaz I've tested again using a fresh Windows 11 VM install. Followed the Manual Setup (Windows) and installed it from the develop branch. This time used the official python 3.11 installer instead of using scoop to install python. Still got the ModuleNotFoundError: No module named 'win32file' error. adding this to requirements.txt solves the issue for new installs:

pywin32; sys_platform=='win32'

[eoyilmaz]

Yeah don't use the develop branch it is the wrong one use 367-compiled-sucessfully-in-w10-py311-but-createprocess-fails-call-to-dispread-to-measure (that's a stupid branch name I know but it follows the issue title)

[alex-mirkin]

Ooh totally missed that part :) Thanks!